Logistic Regression

In this Blog I will be writing about a widely used classification ML algorithm, that is, Logistic Regression.

Here I will cover the topics like What is Logistic Regression, Why we use it, How to get started with logistic Regression, Applications of Logistic regression, Advantages/Disadvantages also I will provide my Jupyter Notebook on implementation of Logistic regression from scratch.

Also if you want know what is regression check below link. I have briefly explained about it here.

Linear Regression

So without any further due lets get started.

What is Logistic Regression?

Logistic regression is a statistical model that in its basic form uses a logistic function to model a binary dependent variable, although many more complex extensions exist. In regression analysis, logistic regression (or logit regression) is estimating the parameters of a logistic model (a form of binary regression).

When and Why we use Logistic Regression?

1. To predict an outcome variable that is categorical from predictor variables that are continuous and/or categorical.
2. Used because having a categorical outcome variable violates the assumption of linearity in normal regression.
3. Logistic regression deals with this problem by using a logarithmic transformation on the outcome variable which allow us to model a nonlinear association in a linear way .

Linear vs Logistic regression

Before we jump into math and steps behind Logistic Regression, let’s learn about a function which is used to implement the algorithm i.e. Sigmoid Function.

What is the Sigmoid Function?

In order to map predicted values to probabilities, we use the Sigmoid function. The function maps any real value into another value between 0 and 1. In machine learning, we use sigmoid to map predictions to probabilities.

Sigmoid Function Graph

Getting Started with Logistic Regression

GIF: University of Toronto

Hypothesis Representation

When using linear regression we used a formula of the hypothesis i.e.

hΘ(x) = β₀ + β₁X

For logistic regression we are going to modify it a little bit i.e.

σ(Z) = σ(β₀ + β₁X)

We have expected that our hypothesis will give values between 0 and 1.

Z = β₀ + β₁X

hΘ(x) = sigmoid(Z)

i.e. hΘ(x) = 1/(1 + e^-(β₀ + β₁X)

The Hypothesis of logistic regression

Decision Boundary

We expect our classifier to give us a set of outputs or classes based on probability when we pass the inputs through a prediction function and returns a probability score between 0 and 1.

For Example, We have 2 classes, let’s take them like cats and dogs(1 — dog , 0 — cats). We basically decide with a threshold value above which we classify values into Class 1 and of the value goes below the threshold then we classify it in Class 2.

Example

As shown in the above graph we have chosen the threshold as 0.5, if the prediction function returned a value of 0.7 then we would classify this observation as Class 1(DOG). If our prediction returned a value of 0.2 then we would classify the observation as Class 2(CAT).

Cost Function

The cost function represents optimization objective i.e. we create a cost function and minimize it so that we can develop an accurate model with minimum error.

For logistic regression, the Cost function is defined as:

−log(hθ(x)) if y = 1

−log(1−hθ(x)) if y = 0

Cost function of Logistic Regression

Graph of logistic regression

The above two functions can be compressed into a single function i.e.

Gradient Descent

Now the question arises, how do we reduce the cost value. Well, this can be done by using Gradient Descent. The main goal of Gradient descent is to minimize the cost value. i.e. min J(θ).

Now to minimize our cost function we need to run the gradient descent function on each parameter i.e.

Objective: To minimize the cost function we have to run the gradient descent function on each parameter

Gradient Descent Simplified

Gradient descent has an analogy in which we have to imagine ourselves at the top of a mountain valley and left stranded and blindfolded, our objective is to reach the bottom of the hill. Feeling the slope of the terrain around you is what everyone would do. Well, this action is analogous to calculating the gradient descent, and taking a step is analogous to one iteration of the update to the parameters.

Gradient Descent analogy

Check below link, here’s is my Jupyter Notebook where you can find the explained implementation on Logistic Regression from Scratch. Also you can find dataset used in same directory.

Shag10/Machine-Learning

Applications of Logistic Regression

1. Credit scoring

ID Finance is a financial company that makes predictive models for credit scoring. They need their models to be easily interpretable. They can be asked by a regulator about a certain decision at any moment. Logistic regression is widely used in credit scoring and it shows remarkable results.

2. Medicine

Medical information is gathered in such a way that when a research group studies a biological molecule and its properties, they publish a paper about it. Thus, there is a huge amount of medical data about various compounds, but they are not combined into a single database.

Logistic regression is well suited for this data type when we need to predict a binary answer. Thanks to this algorithm, the accuracy of a quick blood test have been increased.

3. Text editing

As we talked about texts, it is worth mentioning that logistic regression is a popular choice in many natural language processing tasks. First, the text preprocessing is performed, then features are extracted, and finally, logistic regression is used to make some claim about a text fragment. Toxic speech detection, topic classification for questions to support, and email sorting are examples where logistic regression shows good results. Other popular algorithms for making a decision in these fields are support vector machines and random forest.

4. Hotel Booking

Booking.com has a lot of machine learning methods literally everywhere on the site. They try to predict users’ intentions and recognize entities. Where will you go, where do you prefer to stop, what are you planning to do? Some predictions are made even if the user didn’t type anything in the search line yet. But how did they start to do this? No one can build a huge and complex system with various machine learning algorithms from scratch. They have accumulated some statistics and created some simple models as the first steps.

Logistic Regression tries to predict either user will change a journey date or not. Logistic regression could well separate two classes of users. Based on this data, the company then can decide if it will change an interface for one class of users.

5. Gaming

Speed is one of the advantages of logistic regression, and it is extremely useful in the gaming industry. Speed is very important in a game. Very popular today are the games where you can use in-game purchases to improve the gaming qualities of your character, or for fancy appearance and communication with other players. In-game purchases are a good place to introduce a recommendation system.

Advantages and Disadvantages of Logistic Regression

Source: Geeks for Geeks

So this is all from my side, I tried to provide all the important information on Logistic Regression with its implementation. I hope you will find something useful here. Thank you for reading till the end.

References

Introduction to Logistic Regression

5 Real-world Examples of Logistic Regression Application | ActiveWizards: data science and…