Overview :
● Firstly, clone the GitHub repository given below.
● Install the necessary packages from the list provided.
➢ Tensorflow
➢ Numpy
➢ Matplotlib
➢ Open-cv
➢ Scikitlearn
➢ Pandas
➢ Random
● Make sure Correct versions of CUDA and CUDNN is installed in the system for using the Graphics Capability of the system for various tasks like Training the model and Object Classification.
Data-Set :
● The dataset that we are going to use in-order to train the model is a slight modified version of GTSRB (German Traffic Sign Recognition Benchmark). This particular data set contains 43 classes (i.e. 43 different types of Traffic Signs). It contains 39,209 training images and 12,630 test images which in total comes out to be 51,839 images. Each class has images ranging from 100 to 1750.
● The dataset is included in the GitHub URL mentioned above.
● Click For Dataset File !!!
Training the model :
● The “training.ipynb” file is a python notebook which includes every step for training the model for traffic sign recognition.
● Firstly, all the important modules are imported which are required for the project.
import numpy as np
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam
from keras.utils.np_utils import to_categorical
from keras.layers import Dropout, Flatten
from keras.layers.convolutional import Conv2D, MaxPooling2D
import cv2
from sklearn.model_selection import train_test_split
import os
import pandas as pd
import random
from keras.preprocessing.image import ImageDataGenerator
● Images as well as labels are loaded into the python file using numpy array.
● The images are reshaped to 32x32 pixels for training process.
path = "Train" labelFile = 'Train.csv'
count = 0
images = []
label = []
classes_list = os.listdir(path)
print("Total Classes Detected:",len(classes_list))
noOfClasses=len(classes_list)
print("Importing Classes.....")
for x in range (0,len(classes_list)):
imglist = os.listdir(path+"/"+str(count))
for y in imglist:
img = cv2.imread(path+"/"+str(count)+"/"+y)
img =cv2.resize(img,(32,32))
images.append(img)
label.append(count)
print(count, end =" ")
count +=1
print(" ")
images = np.array(images)
classNo = np.array(label)
data=np.array(images)
data= np.array(data).reshape(-1, 32, 32, 3)
Total Classes Detected: 43
Importing Classes.....
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
● In the next notebook cell images are split into train, test and validate folders respectively. By default, the value is set to 10 percent of images for test, 10 percent for testing and 80 percent of the images for training purpose.
X_train, X_test, y_train, y_test = train_test_split(images, classNo, test_size=0.1) Y_tests=y_test X_train, X_validation, y_train, y_validation = train_test_split(X_train, y_train, test_size=0.1)
Data Shapes
Train(31759, 32, 32, 3) (31759,)
Validation(3529, 32, 32, 3) (3529,)
Test(3921, 32, 32, 3) (3921,)
print("Data Shapes")
print("Train",end = "");print(X_train.shape,y_train.shape)
print("Validation",end = "");print(X_validation.shape,y_validation.shape)
print("Test",end = "");print(X_test.shape,y_test.shape)
● The following cell displays the number of images present in each folder i.e. number of images in train, test and validate folder.
● Now three random image is displayed from each class.
● The following notebook cell visualizes the number of each images each class has.
batch_size_val=32 steps_per_epoch_val=992 epochs_val=200
data shape (39209, 8) <class 'pandas.core.frame.DataFrame>
data=pd.read_csv(labelFile)
print("data shape ",data.shape,type(data))
num_of_samples = []
cols = 3
num_classes = noOfClasses
fig, axs = plt.subplots(nrows=num_classes, ncols=cols, figsize=(30, 300))
fig.tight_layout()
for i in range(cols):
for j,row in data.iterrows():
x_selected = X_train[y_train == j]
if len(x_selected)==0:
continue
axs[j][i].imshow(x_selected[random.randint(0, len(x_selected)-1), :, :], cmap=plt.get_cmap("gray"))
axs[j][i].axis("off")
if i == 2:
axs[j][i].set_title(str(j)+ "-"+str(row["ClassId"]))
num_of_samples.append(len(x_selected))
● In the next cell certain functions are defined which are to pre-process the image before passing it for the training phase. The pre-processing includes turning the image to grayscale and normalization of it.
● The cell afterwards is used for augmentation of image i.e creating more number of by the methods of tilting, shifting and manipulating the brightness. This is done in order to efficiently generalise the model.
● Afterwards the architecture of the model is defined by specifying multiple layers and number of classes.
print(num_of_samples)
plt.figure(figsize=(12, 4))
plt.bar(range(0, num_classes), num_of_samples)
plt.title("Distribution of the training dataset")
plt.xlabel("Class number")
plt.ylabel("Number of images")
plt.show()
[169, 1770, 1838, 1163, 1586, 1494, 337, 1164, 1146, 1176, 1623, 1065, 1697, 1746, 620, 513, 325, 913, 1006, 163, 289, 266, 313, 401, 210, 1221, 495, 203, 447, 219, 378, 639, 193, 567, 347, 955, 318, 178, 1668, 246, 306, 195, 191]
def grayscale(img):
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
return img
def equalize(img):
img = cv2.equalizeHist(img)
return img
def preprocessing(img):
img = grayscale(img)
img = equalize(img)
img = img / 255 # image normalization
return img
X_train = np.array(list(map(preprocessing, X_train)))
X_validation = np.array(list(map(preprocessing, X_validation)))
X_test = np.array(list(map(preprocessing, X_test)))
● Certain parameters are now defined before training the model like:
i) Batch size value: Represents the number of images to be processed in a single step.
ii) Steps per epoch value: Represents the number of steps to be processed in a single epoch. This can be calculated using the formula:
steps per epoch = floor (number of images in training folder/batch size value)
iii) Number of epochs value: Represents the number of times to pass through the entire training folder.
X_train=X_train.reshape(-1,32,32,1)
X_validation=X_validation.reshape(-1,32,32,1)
X_test=X_test.reshape(-1,32,32,1)
#Augmentation of images
dataGen = ImageDataGenerator(width_shift_range=0.1,
height_shift_range=0.1,
zoom_range=0.2,
shear_range=0.1,
rotation_range=10)
dataGen.fit(X_train)
batches = dataGen.flow(X_train, y_train,batch_size=20)
X_batch, y_batch = next(batches)
y_train = to_categorical(y_train, noOfClasses)
y_validation = to_categorical(y_validation, noOfClasses)
y_test = to_categorical(y_test, noOfClasses)
● Afterwards the model is trained by passing these values into an inbuilt function. The time of the training depends upon the number of training images, number of epochs and the GPU capacity of the system.
def seq_Model():
no_Of_Filters = 60
size_of_Filter = (5, 5)
size_of_Filter2 = (3, 3)
size_of_pool = (2, 2)
no_Of_Nodes = 500
model = Sequential()
model.add((Conv2D(no_Of_Filters, size_of_Filter, input_shape=(32, 32, 1),
activation='relu')))
model.add((Conv2D(no_Of_Filters, size_of_Filter, activation='relu')))
model.add(MaxPooling2D(pool_size=size_of_pool))
model.add((Conv2D(no_Of_Filters // 2, size_of_Filter2, activation='relu')))
model.add((Conv2D(no_Of_Filters // 2, size_of_Filter2, activation='relu')))
model.add(MaxPooling2D(pool_size=size_of_pool))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(no_Of_Nodes, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(noOfClasses, activation='softmax'))
model.compile(Adam(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
return model
model = seq_Model()
print(model.summary())
history=model.fit(dataGen.flow(X_train,y_train,batch_size=batch_size_val),steps_per_epoch=steps_per_epoch_val,epochs=epochs_val,validation_data=(X_validation,y_validation),shuffle=1)
● After the training the accuracy and loss of the model is visualised.
plt.figure(1)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.legend(['training','validation'])
plt.title('loss')
plt.xlabel('epoch')
plt.figure(2)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.legend(['training','validation'])
plt.title('Acurracy')
plt.xlabel('epoch')
plt.show()
● Finally, the accuracy on the test dataset is displayed and the model is saved for further usage.
score =model.evaluate(X_test,y_test,verbose=0)
print('Test Score:',score[0])
print('Test Accuracy:',score[1])
#save model
model.save('traffif_sign_model.h5')
Test Score: 0.03907863795757294
Test Accuracy: 0.987758219242096
Testing the trained model :
● A file named “live_testing.py” has been provided inside the GitHub repository given above which is to be used for the purpose of testing the model and traffic sign recognition.
import numpy as np
import cv2
from tensorflow import keras
threshold = 0.97 #
font = cv2.FONT_HERSHEY_SIMPLEX
model = keras.models.load_model('traffif_sign_model.h5')
def preprocess_img(imgBGR, erode_dilate=True):
rows, cols, _ = imgBGR.shape
imgHSV = cv2.cvtColor(imgBGR, cv2.COLOR_BGR2HSV)
Bmin = np.array([100, 43, 46])
Bmax = np.array([124, 255, 255])
img_Bbin = cv2.inRange(imgHSV, Bmin, Bmax)
Rmin1 = np.array([0, 43, 46])
Rmax1 = np.array([10, 255, 255])
img_Rbin1 = cv2.inRange(imgHSV, Rmin1, Rmax1)
Rmin2 = np.array([156, 43, 46])
Rmax2 = np.array([180, 255, 255])
img_Rbin2 = cv2.inRange(imgHSV, Rmin2, Rmax2)
img_Rbin = np.maximum(img_Rbin1, img_Rbin2)
img_bin = np.maximum(img_Bbin, img_Rbin)
if erode_dilate is True:
kernelErosion = np.ones((3, 3), np.uint8)
kernelDilation = np.ones((3, 3), np.uint8)
img_bin = cv2.erode(img_bin, kernelErosion, iterations=2)
img_bin = cv2.dilate(img_bin, kernelDilation, iterations=2)
return img_bin
def contour_detect(img_bin, min_area, max_area=-1, wh_ratio=1.2):
rects = []
contours, _ = cv2.findContours(img_bin.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
if len(contours) == 0:
return rects
max_area = img_bin.shape[0] * img_bin.shape[1] if max_area < 0 else max_area
for contour in contours:
area = cv2.contourArea(contour)
if area >= min_area and area <= max_area:
x, y, w, h = cv2.boundingRect(contour)
if(1.0 * w / h < wh_ratio and 1.0 * h / w < wh_ratio):
rects.append([x, y, w, h])
return rects
def preprocessing(img):
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
img = cv2.equalizeHist(img)
img = img / 255
return img
def class_disp(classNo):
sign_classes=['Speed Limit 20 km/h','Speed Limit 30 km/h','Speed Limit 50 km/h','Speed Limit 60 km/h','Speed Limit 70 km/h','Speed Limit 80 km/h','End of Speed Limit 80 km/h','Speed Limit 100 km/h','Speed Limit 120 km/h','No passing','No passing for vechiles over 3.5 metric tons','Right-of-way at the next intersection','Priority road','Yield','Stop','No vechiles','Vechiles over 3.5 metric tons prohibited','No entry','General caution','Dangerous curve to the left','Dangerous curve to the right','Double curve','Bumpy road','Slippery road','Road narrows on the right','Road work','Traffic signals','Pedestrians','Children crossing','Bicycles crossing','Beware of ice/snow','Wild animals crossing','End of all speed and passing limits','Turn right ahead','Turn left ahead','Ahead only','Go straight or right','Go straight or left','Keep right','Keep left','Roundabout mandatory','End of no passing','End of no passing by vechiles over 3.5 metric tons']
classNo=int(classNo)
return sign_classes[classNo]
if __name__ == "__main__":
cap = cv2.VideoCapture(1)
cols = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
rows = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
while (1):
ret, img = cap.read()
img_bin = preprocess_img(img, False)
#cv2.imshow("bin image", img_bin)
min_area = img_bin.shape[0] * img.shape[1] / (25 * 4)
rects = contour_detect(img_bin, min_area=min_area)
img_bbx = img.copy()
for rect in rects:
xc = int(rect[0] + rect[2] / 2)
yc = int(rect[1] + rect[3] / 2)
size = max(rect[2], rect[3])
x1 = max(0, int(xc - size / 2))
y1 = max(0, int(yc - size / 2))
x2 = min(cols, int(xc + size / 2))
y2 = min(rows, int(yc + size / 2))
if rect[2] > 100 and rect[3] > 100:
continue
#cv2.rectangle(img_bbx, (rect[0], rect[1]), (rect[0] + rect[2], rect[1] + rect[3]), (0, 0, 255), 2)
crop_img = np.asarray(img[y1:y2, x1:x2])
crop_img = cv2.resize(crop_img, (32, 32))
crop_img = preprocessing(crop_img)
cv2.imshow("afterprocessing", crop_img)
crop_img = crop_img.reshape(1, 32, 32, 1)
predictions = model.predict(crop_img)
classIndex=np.argmax(predictions, axis=-1)
probabilityValue = np.amax(predictions)
if probabilityValue > threshold:
cv2.putText(img_bbx, str(classIndex) + " " + str(class_disp(classIndex)), (rect[0], rect[1] - 10),
font, 0.75, (0, 0, 255), 2, cv2.LINE_AA)
cv2.putText(img_bbx, str(round(probabilityValue * 100, 2)) + "%", (rect[0], rect[1] - 40), font, 0.75,
(0, 0, 255), 2, cv2.LINE_AA)
print(class_disp(classIndex))
print(probabilityValue)
cv2.imshow("Final result", img_bbx)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
● The script works on real-time basis and displays the detected traffic sign.
● Initially in the script the path of the model and the threshold probability is to be defined. The term threshold probability represents the confidence level of the model for the prediction that it is making. As the threshold probability represents a probability value which implies that it’s range is from 0 to 1.
● Followed by this are multiple functions which are used in the script for obtaining the result. Following are the functions with a brief description about them:
i) preprocess_img: This function takes BGR channel image as input and converts it to HSV colour-space. Then each pixel in the HSV image is given a pixel value of either 1 or 0 depending upon its colour, following function represents the working:
(1) If pixel = red or blue -> assign pixel 1
else -> assign pixel 0
Finally, a binary image is returned using this function.
ii) contour_detect: This function takes binary image as input and returns rectangular contours. Contours basically is the parameter of object of interest in our case it is traffic signs.
iii) preprocessing: This function takes BGR image input and turns it into a grayscale image which has been normalised just like we did while training the dataset.