How to Build Self-Driving-Car Using End-to-End Deep_Learning..!!

Hii folks...

Today we will build a self-driving-car system using Deep learning. I will try to keep things as simple as possible. & will explain code line by line whenever needed. So even a beginner can also pick up most things easily. Some prerequisite to follow this blog is :

1. Familiar with tools like Pandas, Numpy.. etc…
2. Should have some basic knowledge of Popular Computer vision Library “Opencv2”. (if not have then I’ll explain each line of Opencv2 code)
3. Should familiar with CNN’s.

Table of Contents- -

1.Overview Of the Problem

2. How I solved This Problem

3. Source and details of data I used for the training model.

4. Introduction With Data & Some EDA.

5. Preprocessing Of Images.

6. Creating Data Pipeline For feeding Data to model.

7 . Model Building And Training :

8. Model In Action.

1. Overview of the problem, That we are solving.

I will not discuss many things like “what is Self-Driving-Car” etc... Because if you are here reading this blog then most probably you know those things. According to a report released by the US state Department of Transportation, “Self-Driving-Cars can reduce 90% of Traffic Deaths. It is just one advantage of having Self-Driving-Cars. So now we have a sense of How important this Technology could be. A big chunk of major Automobile companies is trying to Develop Self-Driving-Cars. Some Big Players are Tesla, Waymo… Even Google is developing Self Driving Cars Which has no presence in the Automobile sector.

Now let's jump to the next part of the blog. If you want to know more about Self-Driving-Cars Click Here

2. How I solved this problem using Nvidea Research Paper.

The first thing we need to understand is the idea of self dring car is not new. It is even older than Morder computing Era. “ In General Moter’s Exhibit in 1939 Normal Bell Geddes created the first Self Driving car. Which was an electric vehicle guided by Radio Controlled Electronic Field. But After Depp learning Comes in to picture in 2011–12’s like lots of other problems Self-Driving-Car also sees a huge boost. Good models are started building and this much hype created around Self-Driving-Cars. As a result of this Researchers or engineers in Nvidea also try to build Self-Driving-Cars. & Finally in 2014, They published a Research Paper . and I used the same research paper to build my solution. So let’s see some key points of that research Paper first :

1. They proposed the whole self-driving-car as a Regression Problem.

2. They trained a model which takes picture of the car's front view and returns the angle of steering should be In that particular frame.

3. They invented a whole new 8 layer of Neural-network architecture. Which has Four Convolution Layer and Four dense layers. Bellow is the picture of his architecture.

This Model seems easy in the first chance but it is so much powerful or accurate that the team of NVidea actually deployed this system in an actual car. And surprisingly 98% of the time the car was Automated. They wrote a detailed blog about this whole journey You can read that By clicking the link below.

End-to-End Deep Learning for Self-Driving Cars | NVIDIA Developer Blog

Here I will also take the learning of his journey and will train a model by giving front camera pictures of car and corresponding angles of them. And after training my model can tell at which angle the steering should be at a particular moment. Of course, My model might not be the same robust as theirs because they trained his model on such a huge amount of data and as a student, I don’t have that many resources to train such powerful models. Also Here we are not building a state-of-the-art solution. We are just building a simple solution and trying to learn things. So now let’s jump on next part of the blog-

3.Source And Detailes of Data. That I Used for model Training:

There is a Vast range of Dataset Available for Building a Self-Driving-Car System. Some popular of them are bellow –

1. Waymo Dataset (Click_Here)
2. Udacity Self Driving Car Dataset (Click_Here)
3. Mit and Toyota Dataset (Click_Here) … ETC...

These all are really good but Very large Dataset. And since we have limitations of resources It’s not at all easy to go with the above Datasets. Hence we will use Dataset Released By Sullychen. You can go to the actual Github account by clicking here. Sullychen Published 2 datasets one in 2017 and the next in 2018. We will use the 2018’s Dataset. Now let’s see what type of Data we have :

As we can see in the above image after unzipping the original file we get 2 things: 1- data folder Which contains all images, 2- data.txt File Which Contains Information Like the Name of That image, Corresponding Angle of Each Image, Date, Time. We have total of ~47 minutes of data. and for each second we have ~23 images. So 47*60*23 ~63k Images We have in Total. It’s not a very big dataset for deep learning Tasks but reasonably good.

4. Introduction With Data & Some EDA.

Before Doing anything First Let’s just See some Random Pictures we have :

Output :

Output :

Output :

Output :

Since we have got the sense of image_data we have. Now let’s go further and Create a Dataframe so things become easy for us. This data frame of Details will have 3 columns :

1. Image Name
2. Path Of the image
3. Corresponding Steering Angle For that Image.

Output :

Let’s Plot the distplot of Steering Angles and Understand it :

Output :

Observations :

• Most of the Time steering angle is 0 or close to 0. & It makes sense Because while driving a car most of the time we go just straight. Hence Steering should be at an angle of 0.
• There are only a few points of -300 degree and +200 degree. & It also makes perfect sense Because only a few times we need to take very curvy turns.

5. Preprocessing Of Images.

In NLP problems preprocessing basically means removing stopwords, punctuations, and things which don't give information to predict class labels. The same concept is applied here in Computer vision problems. Since we have data in the form of images here so we need to remove those parts of images which doesn't give much information to predict class labels. In NLP problems we have some standard steps like removing stopwords, punctuations, etc… That we perform in most of the problems. But Unfortunately in Computer vision, there are no such steps. Because which part of the image is meaningless, it differs from problem -to problem. Hence we need to see images manually and find what preprocessing steps we can perform in this particular problem.

So the problem here is we have to predict the angle of steering should be at any moment. And we know it depends on the structure of road and traffic. Both These things are present in the lower part of the image. In the above part of the image, just a clean sky is there. which doesn’t give much information about the steering angle should be at any particular moment. Let’s see it Visually :

This is the image(500.jpg) we have seen above.

Now let’s split it into two parts :

Output :

Output :

Now see both parts of the image and think how much information above_part_of_image gives in predicting angle steering. I’ll say not much Because that has just some trees and white sky. while in the down_part_of_image whole road is present. we can tell the angle of steering should be by just looking at down_part of Image.

Since the above part of the images doesn’t give much information to us then let’s just crop all the images and only have down part of the images. So I am creating A new folder that will have all the cropped images.

Output :

In The research paper, we have seen that the model will require an image of shape (66,200,3). But after cropping our images has the shape of (150,455,3). Hence we need to resize our images in to shape of (66,200,3). So let’s resize all images and create a new folder “resized_image” which will contain all resized_images.

Output :

Since now we have our images in a different directory hence we also need to update our dataframe_of_details. which has image paths of different directories.

Output :

We will use Steering angles in Radians. But we have angles in Degrees so let’s convert angles in radians and add a new column in our dataframe angle_in_radians.

Output :

6. Creating Data Pipeline For feeding Data to model.

Now comes the Important Part which is Model Building or training. But for the training model, we need a pipeline through which we can feed the data to the model for training. Apart from creating a Data pipeline one other method to feed data to the model is to load all the training data in a variable and just give it to the model. but if Data is in reasonable Size then this way of feeding Data becomes highly inefficient. Actually, I Tried to load data in a variable just for the curiosity of what will happens next. Then after some time, my whole notebook gets crashed because of the Ram issue. Just for your information, Colab gives 12GB of ram for a session. But I could not even load half of the data in this amount of ram. Here ImageDataGenerator of Keras Comes for my rescue. ImageDataGenerator is a function of Keras which provides the flexibility of loading some amount of data in ram, give that to model and release the ram, Then again take some amount of data in ram and give to model. If you Want to Read More About ImageDataGenerator just go to the keas documentation itself by clicking the link, Bellow.

Keras documentation: Image data preprocessing

So Now let’s go and use ImageDatagenerator for our Problem :

Pipeline for Test Data :

We will use the “ flow_from_dataframe” method of ImageDataGenerator to feed data. And as the name indicates “flow_from_dataframe” needs a dataframe that should have information about that data for which we are making that DataGenerator. So will first create a test Dataframe and then initialize a DataGenerator.

I will use only 5% of the data for testing Because 5% of Data Means more than 2 minutes of data. And I think it is enough to judge a Self-Driving-Model working good or Not. Also Since we know we Didn’t have much data so I want to use maximum Data for Training purposes. But it depends on problem to problem How much data should be used for testing purposes. Since Self-Driving-Car is a special type of problem so I am okay to go with only 5% data for testing purposes. I will take (85%–90%) part of the Data as my Test Data.

OutPut :

Now let’s Initialize TestDataGenerator :

Now Lets first Create Train_dataframe:

Now Let’s Initialize TrainDataGenerator :

7 . Model Building And Training :

Now let’s start with our model defining according to the Research Paper of Nvidea :

Output :

Let's plot the Graph of the model :

OutPut :

Now Let’s Compile the Model with Optimizer = “adam & loss = “mean_absolute_error” :

I am adding a callback that will automatically save the model after every epoch if the loss gets improved from the last best model. So if in the middle of model training anything happens then we don’t need to train from scratch again.

Everything is Okey so now let’s start train the model.

The best model we get in this training has 0.0132 val_loss. which is quite good:

8. Model In Action :

Since Training is completed. It's time to see the model in action. instead of looking at the model’s prediction line by line and judging the model is simply boring and time taking. So After taking predictions from the model below is the nice visualization of pictures and rotated steering angle based on model prediction.

Output :

If you Need whole code then Please go to my GitHub Repository By clicking the link given bellow :

uttu-parashar/Self-Driving-Car

In case if you are also a AI Enthusiast just connect with me on LinkedIn by clicking on the link given bellow.

Utkarsh Parashar - Student - Applied AI Course | LinkedIn

*************Thanks For Reading. **************

*******Good-Bye Folks... Best of luck for the future.*****