INTRODUCTION
The most common problem in Machine learning models is Overfitting. however, The Overfitted model has low accuracy and high error.
Overfitting refers to the situation where our model shows perfect results for the training dataset but is not able to predict correct results for the test dataset.
Overfitted models thus have HIGH VARIANCE and LOW BIAS.
so To avoid overfitting can increase our model’s accuracy and reduce errors.
But how do we avoid overfitting?
So there are many Regularization Techniques that help in avoiding overfitting and in this tutorial we will discuss them.
REGULARIZATION
Regularization refers to the techniques used to reduce the chance of overfitting by fitting a function on a training dataset which also helps in reducing error.
In Machine learning, regularization penalizes the coefficients.
This article however assumes that you are already familiar with the loss function and linear regression model and its error estimation.
Types of Regularization Techniques
1. L1 REGULARIZATION
L1 regularization is also known as LASSO ( Least Absolute Shrinkage and Selection Operator) REGULARIZATION. In this technique, we thus penalize the absolute value of weights to the cost function.
Cost function = Loss + ? Σ|Bj|
Here ? = Regularization Parameter.
It performs internal feature selection and thus used in models which are
Feature constrained.
2. L2 REGULARIZATION
L2 regularization is also famous as RIDGE REGULARIZATION. In this technique, we add a squared magnitude of coefficient as a penalty to the cost function.
Cost function = Loss + ? Σ|Bj|2
It performs better than L1 regularization in most cases.
It does not shrink the less important feature to zero but tries to minimize them.
L1 regularization shrinks the less important features to null which results in FEATURE SELECTION, it removes the features which do not help in increasing the prediction power of the model.
3. ELASTIC-NET REGULARIZATION
Elastic-Net regularization is the combination of both types of regularization which we had studied earlier i.e L1 and L2.
Cost function = Loss + α * Ridge Penalty + ( 1 – α ) * Lasso Penalty
Here α is the same as that of ? in L1 and L2 regularization.
Since Elastic-Net is intermediate between L1 and L2 regularization so it tries to minimize the less important feature and feature selection both at the same time.
IMPLEMENTATION
We will implement all the regularization techniques in python using the scikit-learn library on the Boston dataset which is present in the sklearn dataset.
First, we will import the necessary packages.
Now we will import the dataset which is present in the sklearn library.
Output
Now we will add our output variable to the dataset.
output
Now we will perform cross-validation.
however, we will perform regularization and also train a linear regression model on our dataset.
Now we will train our models.
Output :
Now we will print the accuracy of our models.
Output :
As we can see that in this Accuracy for RIDGE REGULARIZATION MODEL
is highest, so it is better than others in this case.
Now we will print an OLS( Ordinary Least Square) report.
Output
Now we will print estimation curves for our models
Output
Note: During Regularization only loss function varies, there is no change in the output function ( y ).
But why do we need Regularization Techniques?
Regularization reduces the Variance of the model without a subsequent increase in the Bias which helps in the Bias-Variance tradeoff.
Also if we perform too much regularization of our model then we will lose important features of our dataset, which will result in decreasing the overall accuracy of the model.
So it all depends upon choosing the optimal value for Hyperparameter (λ) if you implement your own regularization technique.
Summary
In this article we studied regularization techniques, why do we need it, its types, its implementation on the Boston Housing Price dataset, we also saw a graphical representation of model estimation with all types of regularization.
I hope you liked it.
Written By: Mohit Kumar
Reviewed By: Savya Sachi
If you are Interested In Machine Learning You Can Check Machine Learning Internship Program
Also Check Other Technical And Non Technical Internship Programs
