Loading

Lidar Car Detection

5 Fold Cross Validation Solution for Lidar Car Detection

A detailed solution for challenge Lidar Car Detection

g_mothy

5-Fold Cross Validation XGBoost Solution for Lidar Car Detection

Lidar Car Detection

Downloading Dataset

Installing aicrowd-cli

In [1]:
!pip install aicrowd-cli
%load_ext aicrowd.magic
Collecting aicrowd-cli
  Downloading aicrowd_cli-0.1.10-py3-none-any.whl (44 kB)
     |████████████████████████████████| 44 kB 2.3 MB/s 
Collecting pyzmq==22.1.0
  Downloading pyzmq-22.1.0-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 35.8 MB/s 
Collecting requests-toolbelt<1,>=0.9.1
  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 2.2 MB/s 
Collecting rich<11,>=10.0.0
  Downloading rich-10.9.0-py3-none-any.whl (211 kB)
     |████████████████████████████████| 211 kB 47.3 MB/s 
Requirement already satisfied: tqdm<5,>=4.56.0 in /usr/local/lib/python3.7/dist-packages (from aicrowd-cli) (4.62.0)
Requirement already satisfied: click<8,>=7.1.2 in /usr/local/lib/python3.7/dist-packages (from aicrowd-cli) (7.1.2)
Collecting requests<3,>=2.25.1
  Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 718 kB/s 
Requirement already satisfied: toml<1,>=0.10.2 in /usr/local/lib/python3.7/dist-packages (from aicrowd-cli) (0.10.2)
Collecting GitPython==3.1.18
  Downloading GitPython-3.1.18-py3-none-any.whl (170 kB)
     |████████████████████████████████| 170 kB 41.4 MB/s 
Collecting gitdb<5,>=4.0.1
  Downloading gitdb-4.0.7-py3-none-any.whl (63 kB)
     |████████████████████████████████| 63 kB 1.7 MB/s 
Requirement already satisfied: typing-extensions>=3.7.4.0 in /usr/local/lib/python3.7/dist-packages (from GitPython==3.1.18->aicrowd-cli) (3.7.4.3)
Collecting smmap<5,>=3.0.1
  Downloading smmap-4.0.0-py2.py3-none-any.whl (24 kB)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.25.1->aicrowd-cli) (2.0.4)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.25.1->aicrowd-cli) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.25.1->aicrowd-cli) (2021.5.30)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.25.1->aicrowd-cli) (1.24.3)
Collecting colorama<0.5.0,>=0.4.0
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in /usr/local/lib/python3.7/dist-packages (from rich<11,>=10.0.0->aicrowd-cli) (2.6.1)
Collecting commonmark<0.10.0,>=0.9.0
  Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
     |████████████████████████████████| 51 kB 6.0 MB/s 
Installing collected packages: smmap, requests, gitdb, commonmark, colorama, rich, requests-toolbelt, pyzmq, GitPython, aicrowd-cli
  Attempting uninstall: requests
    Found existing installation: requests 2.23.0
    Uninstalling requests-2.23.0:
      Successfully uninstalled requests-2.23.0
  Attempting uninstall: pyzmq
    Found existing installation: pyzmq 22.2.1
    Uninstalling pyzmq-22.2.1:
      Successfully uninstalled pyzmq-22.2.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires requests~=2.23.0, but you have requests 2.26.0 which is incompatible.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
Successfully installed GitPython-3.1.18 aicrowd-cli-0.1.10 colorama-0.4.4 commonmark-0.9.1 gitdb-4.0.7 pyzmq-22.1.0 requests-2.26.0 requests-toolbelt-0.9.1 rich-10.9.0 smmap-4.0.0
In [5]:
%aicrowd login
Please login here: https://api.aicrowd.com/auth/sstBUIRuRwdratzdCzqNquOIM-byD6gLXOgZSPWFQLA
API Key valid
Saved API Key successfully!
In [6]:
!rm -rf data
!mkdir data
%aicrowd ds dl -c lidar-car-detection -o data

Importing Libraries

In [11]:
import os
import random
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import plotly.graph_objects as go


import xgboost as xgb
import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor,ExtraTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import AdaBoostRegressor,BaggingRegressor
from sklearn.ensemble import ExtraTreesRegressor,GradientBoostingRegressor
from sklearn.model_selection import KFold

seed = 2020

Reading the dataset

In [13]:
# Reading the training dataset
data_dir = "./data"

train_data = np.load(os.path.join(data_dir,"train.npz"), allow_pickle=True)
train_data = train_data['train']

# Loading the test data
test_data = np.load(os.path.join(data_dir, "test.npz"), allow_pickle=True)
test_data = test_data['test']

train_data.shape, test_data.shape
Out[13]:
((400, 2), (601,))

Visualizing the dataset

In this section, we will be visualizing a sample 3D lidar data

In [9]:
# Getting a random 3D lidar sample data
INDEX = random.randint(0, train_data.shape[0])

# Getting the individual x,y and z points.
x = train_data[INDEX][0][:, 0].tolist()
y = train_data[INDEX][0][:, 1].tolist()
z = train_data[INDEX][0][:, 2].tolist()

# Label for the corrosponding sample ( no. of cars )
label  = train_data[INDEX][1]

# Generating the 3D graph
fig = go.Figure(data=[go.Scatter3d(x=x, y=y, z=z,
                                   mode='markers',
                                   marker=dict(
                                   size=1,       
                                   colorscale='Viridis',
                                   opacity=0.8))])
print("No. of cars : ", label)
fig.show()
No. of cars :  0

Splitting the dataset

In [15]:
# Getting the 3d points and flattening the points into 1d array
X = train_data[:, 0]
X = [i.flatten() for i in X]

# labels
y = train_data[:, 1]

# flattening the points into 1d array
X_test = [i.flatten() for i in test_data]

Training the model

5 Fold Cross Validation

In [34]:
K = 5
kf = KFold(n_splits = K, random_state = seed, shuffle = True)

xgb_preds = []
train = np.array(X)
target_train = np.array(y)
test = np.array(X_test)

train.shape,target_train.shape,test.shape
Out[34]:
((400, 116379), (400,), (601, 116379))
In [35]:
for train_index, test_index in kf.split(train):
    train_X, valid_X = train[train_index], train[test_index]
    train_y, valid_y = target_train[train_index], target_train[test_index]

    xgb_params = {
                    "objective":"reg:squarederror",
                    'colsample_bytree': 0.3, 
                    'learning_rate': 0.1,
                    'max_depth': 5, 
                    'alpha': 10}


    d_train = xgb.DMatrix(train_X, train_y)
    d_valid = xgb.DMatrix(valid_X, valid_y)
    d_test = xgb.DMatrix(test)
    
    watchlist = [(d_train, 'train'), (d_valid, 'valid')]
    model = xgb.train(xgb_params, 
                      d_train, 
                      5000,  
                      watchlist, 
                      maximize=True, 
                      verbose_eval=50, 
                      early_stopping_rounds=100)
                        
    xgb_pred = model.predict(d_test)
    xgb_preds.append(list(xgb_pred))
[0]	train-rmse:2.17905	valid-rmse:2.21413
Multiple eval metrics have been passed: 'valid-rmse' will be used for early stopping.

Will train until valid-rmse hasn't improved in 100 rounds.
[50]	train-rmse:0.38497	valid-rmse:1.55156
[100]	train-rmse:0.321756	valid-rmse:1.55663
Stopping. Best iteration:
[0]	train-rmse:2.17905	valid-rmse:2.21413

[0]	train-rmse:2.16768	valid-rmse:2.29408
Multiple eval metrics have been passed: 'valid-rmse' will be used for early stopping.

Will train until valid-rmse hasn't improved in 100 rounds.
[50]	train-rmse:0.391362	valid-rmse:1.24763
[100]	train-rmse:0.317668	valid-rmse:1.24112
Stopping. Best iteration:
[0]	train-rmse:2.16768	valid-rmse:2.29408

[0]	train-rmse:2.2032	valid-rmse:2.1623
Multiple eval metrics have been passed: 'valid-rmse' will be used for early stopping.

Will train until valid-rmse hasn't improved in 100 rounds.
[50]	train-rmse:0.396281	valid-rmse:1.19889
[100]	train-rmse:0.333375	valid-rmse:1.19564
Stopping. Best iteration:
[0]	train-rmse:2.2032	valid-rmse:2.1623

[0]	train-rmse:2.19541	valid-rmse:2.19773
Multiple eval metrics have been passed: 'valid-rmse' will be used for early stopping.

Will train until valid-rmse hasn't improved in 100 rounds.
[50]	train-rmse:0.381407	valid-rmse:1.32297
[100]	train-rmse:0.317776	valid-rmse:1.31624
Stopping. Best iteration:
[0]	train-rmse:2.19541	valid-rmse:2.19773

[0]	train-rmse:2.1927	valid-rmse:2.19097
Multiple eval metrics have been passed: 'valid-rmse' will be used for early stopping.

Will train until valid-rmse hasn't improved in 100 rounds.
[50]	train-rmse:0.399257	valid-rmse:1.57798
[100]	train-rmse:0.331795	valid-rmse:1.574
Stopping. Best iteration:
[0]	train-rmse:2.1927	valid-rmse:2.19097

In [36]:
preds=[]
for i in range(len(xgb_preds[0])):
    sum=0
    for j in range(K):
        sum+=xgb_preds[j][i]
    preds.append(sum / K)
In [45]:
submission = pd.DataFrame({"label":np.round(preds)})
submission
Out[45]:
label
0 2.0
1 2.0
2 3.0
3 3.0
4 2.0
... ...
596 2.0
597 2.0
598 3.0
599 2.0
600 3.0

601 rows × 1 columns

In [47]:
# Saving the predictions
!rm -rf assets
!mkdir assets
submission.to_csv(os.path.join("assets", "submission.csv"))
In [48]:
submission['label'].value_counts()
Out[48]:
2.0    235
3.0    212
1.0    117
4.0     32
0.0      5
Name: label, dtype: int64

Submitting our Predictions

In [ ]:

/usr/local/lib/python3.7/dist-packages/aicrowd/notebook/helpers.py:361: UserWarning: `%aicrowd` magic command can be used to save the notebook inside jupyter notebook/jupyterLab environment and also to get the notebook directly from the frontend without mounting the drive in colab environment. You can use magic command to skip mounting the drive and submit using the code below:
 %load_ext aicrowd.magic
%aicrowd notebook submit -c lidar-car-detection -a assets --no-verify
  warnings.warn(description + code)
Mounting Google Drive 💾
Your Google Drive will be mounted to access the colab notebook
Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.activity.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fexperimentsandconfigs%20https%3a%2f%2fwww.googleapis.com%2fauth%2fphotos.native&response_type=code

Enter your authorization code:
In [ ]:


Comments

You must login before you can post a comment.

Execute