The Art of Regularization
What to do when your model refuses to generalise…
Underfitting and Overfitting are the most common problems while building a machine learning model. You can check out my previous blog to read more about it(https://medium.com/datadriveninvestor/bias-variance-trade-off-fb5fa4c8ab56). This blog addresses the issue of Overfitting and how to resolve it using a technique known as Regularization.
Regularization is a form of regression that constrains or shrinks the coefficient estimates towards zero. This technique discourages learning a more complex or flexible model, so as to avoid overfitting.
There are three types of Regularization :
- L1 Regularization or L1 norm:
- Also know as Lasso Regularization / Panelization performs Least Absolute Shrinkage (Minimum possible shrinkage) and Selection Operator (Performs feature selection) by setting high coefficients to 0.
- It does not work on categorical variables. To use L1 on categorical variables, you might need to use something called as a grouped lasso. But even in grouped lasso, it will only choose 1 category and ignore the rest of the categories in a grouped variable.
2. L2 Regularization or L2 norm:
- Also known as the Ridge Regularization / Penalization or Tikhnov Regularization, it minimizes the cost function without enforcing zero or doing a feature elimination
- Before applying Ridge Regression, you might need to normalize the variables by bringing them to the same scale.
3. Elastic net:
- This type of Regularization implements both a Lasso and a Ridge type of Regularization together.
- This is the best type of Regularization and can be used when in doubt as it is piece wise linear.
If you liked my article and are looking for more such posts on Data Science in Layman terms with minimal Math, please clap or follow me on medium. If you have queries you can connect with me on LinkedIn (https://www.linkedin.com/in/aishwarya-nair-21091994/). Thanks for reading all the way till here and stay tuned for more!
