Loading

ROVER CLASSIFICATION

Rover Classification Notebook

This one is for the workshop: "Introduction to Deep Learning!"

vyom999

image.png

Download Necessary Packages 📚

FastAI

fastai is a deep learning library which provides practitioners with high-level components that can quickly and easily provide state-of-the-art results in standard deep learning domains, and provides researchers with low-level components that can be mixed and matched to build new approaches. It aims to do both things without substantial compromises in ease of use, flexibility, or performance.

AIcrowd-CLI

AIcrowd Command Line Interface is basically a package that allows you to download datasets from the AIcrowd website and make submissions directly from colab!

In [ ]:
!pip install --upgrade fastai 
!pip install aicrowd-cli==0.1
Collecting fastai
  Downloading https://files.pythonhosted.org/packages/79/50/0b81742909d433ef5b67b4417597b41cd37b38a073714bc83dc150c81b2f/fastai-2.2.7-py3-none-any.whl (193kB)
     |████████████████████████████████| 194kB 7.4MB/s 
Requirement already satisfied, skipping upgrade: packaging in /usr/local/lib/python3.7/dist-packages (from fastai) (20.9)
Requirement already satisfied, skipping upgrade: scikit-learn in /usr/local/lib/python3.7/dist-packages (from fastai) (0.22.2.post1)
Collecting torchvision<0.9,>=0.8
  Downloading https://files.pythonhosted.org/packages/94/df/969e69a94cff1c8911acb0688117f95e1915becc1e01c73e7960a2c76ec8/torchvision-0.8.2-cp37-cp37m-manylinux1_x86_64.whl (12.8MB)
     |████████████████████████████████| 12.8MB 225kB/s 
Collecting fastcore<1.4,>=1.3.8
  Downloading https://files.pythonhosted.org/packages/0c/98/60404e2817cff113a6ae4023bc1772e23179408fdf7857fa410551758dfe/fastcore-1.3.19-py3-none-any.whl (53kB)
     |████████████████████████████████| 61kB 9.3MB/s 
Requirement already satisfied, skipping upgrade: pandas in /usr/local/lib/python3.7/dist-packages (from fastai) (1.1.5)
Requirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.7/dist-packages (from fastai) (2.23.0)
Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.7/dist-packages (from fastai) (1.4.1)
Requirement already satisfied, skipping upgrade: pyyaml in /usr/local/lib/python3.7/dist-packages (from fastai) (3.13)
Requirement already satisfied, skipping upgrade: fastprogress>=0.2.4 in /usr/local/lib/python3.7/dist-packages (from fastai) (1.0.0)
Requirement already satisfied, skipping upgrade: pillow>6.0.0 in /usr/local/lib/python3.7/dist-packages (from fastai) (7.0.0)
Requirement already satisfied, skipping upgrade: spacy<3 in /usr/local/lib/python3.7/dist-packages (from fastai) (2.2.4)
Requirement already satisfied, skipping upgrade: matplotlib in /usr/local/lib/python3.7/dist-packages (from fastai) (3.2.2)
Requirement already satisfied, skipping upgrade: pip in /usr/local/lib/python3.7/dist-packages (from fastai) (19.3.1)
Collecting torch<1.8,>=1.7.0
  Downloading https://files.pythonhosted.org/packages/90/5d/095ddddc91c8a769a68c791c019c5793f9c4456a688ddd235d6670924ecb/torch-1.7.1-cp37-cp37m-manylinux1_x86_64.whl (776.8MB)
     |████████████████████████████████| 776.8MB 22kB/s 
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->fastai) (2.4.7)
Requirement already satisfied, skipping upgrade: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->fastai) (1.0.1)
Requirement already satisfied, skipping upgrade: numpy>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->fastai) (1.19.5)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->fastai) (2.8.1)
Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->fastai) (2018.9)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->fastai) (2020.12.5)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->fastai) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->fastai) (2.10)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->fastai) (1.24.3)
Requirement already satisfied, skipping upgrade: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (1.0.5)
Requirement already satisfied, skipping upgrade: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (3.0.5)
Requirement already satisfied, skipping upgrade: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (4.41.1)
Requirement already satisfied, skipping upgrade: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (0.4.1)
Requirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (54.1.2)
Requirement already satisfied, skipping upgrade: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (2.0.5)
Requirement already satisfied, skipping upgrade: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (7.4.0)
Requirement already satisfied, skipping upgrade: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (0.8.2)
Requirement already satisfied, skipping upgrade: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (1.1.3)
Requirement already satisfied, skipping upgrade: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (1.0.0)
Requirement already satisfied, skipping upgrade: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy<3->fastai) (1.0.5)
Requirement already satisfied, skipping upgrade: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fastai) (0.10.0)
Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fastai) (1.3.1)
Requirement already satisfied, skipping upgrade: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch<1.8,>=1.7.0->fastai) (3.7.4.3)
Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->fastai) (1.15.0)
Requirement already satisfied, skipping upgrade: importlib-metadata>=0.20; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy<3->fastai) (3.7.2)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.20; python_version < "3.8"->catalogue<1.1.0,>=0.0.7->spacy<3->fastai) (3.4.1)
ERROR: torchtext 0.9.0 has requirement torch==1.8.0, but you'll have torch 1.7.1 which is incompatible.
Installing collected packages: torch, torchvision, fastcore, fastai
  Found existing installation: torch 1.8.0+cu101
    Uninstalling torch-1.8.0+cu101:
      Successfully uninstalled torch-1.8.0+cu101
  Found existing installation: torchvision 0.9.0+cu101
    Uninstalling torchvision-0.9.0+cu101:
      Successfully uninstalled torchvision-0.9.0+cu101
  Found existing installation: fastai 1.0.61
    Uninstalling fastai-1.0.61:
      Successfully uninstalled fastai-1.0.61
Successfully installed fastai-2.2.7 fastcore-1.3.19 torch-1.7.1 torchvision-0.8.2
Collecting aicrowd-cli==0.1
  Downloading https://files.pythonhosted.org/packages/ee/82/e38ebc9902b1392558d38c8c7af080ba5190898fe2a47220b68850fcbe6c/aicrowd_cli-0.1.0-py3-none-any.whl
Requirement already satisfied: click<8,>=7.1.2 in /usr/local/lib/python3.7/dist-packages (from aicrowd-cli==0.1) (7.1.2)
Collecting gitpython
  Downloading https://files.pythonhosted.org/packages/a6/99/98019716955ba243657daedd1de8f3a88ca1f5b75057c38e959db22fb87b/GitPython-3.1.14-py3-none-any.whl (159kB)
     |████████████████████████████████| 163kB 10.9MB/s 
Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from aicrowd-cli==0.1) (0.10.2)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from aicrowd-cli==0.1) (4.41.1)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from aicrowd-cli==0.1) (2.23.0)
Collecting rich
  Downloading https://files.pythonhosted.org/packages/15/e3/7929cc640f14bffc835bba5efb2f9177007e536429d9fe145765e5f221c4/rich-9.13.0-py3-none-any.whl (197kB)
     |████████████████████████████████| 204kB 18.0MB/s 
Collecting requests-toolbelt
  Downloading https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl (54kB)
     |████████████████████████████████| 61kB 8.6MB/s 
Collecting gitdb<5,>=4.0.1
  Downloading https://files.pythonhosted.org/packages/58/1a/b53069576a6b48a203ec8b7971ae5e455b1d801f5b7d086b986b8fc17186/gitdb-4.0.6-py3-none-any.whl (63kB)
     |████████████████████████████████| 71kB 9.3MB/s 
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->aicrowd-cli==0.1) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->aicrowd-cli==0.1) (2020.12.5)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->aicrowd-cli==0.1) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->aicrowd-cli==0.1) (1.24.3)
Collecting colorama<0.5.0,>=0.4.0
  Downloading https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl
Requirement already satisfied: typing-extensions<4.0.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from rich->aicrowd-cli==0.1) (3.7.4.3)
Collecting commonmark<0.10.0,>=0.9.0
  Downloading https://files.pythonhosted.org/packages/b1/92/dfd892312d822f36c55366118b95d914e5f16de11044a27cf10a7d71bbbf/commonmark-0.9.1-py2.py3-none-any.whl (51kB)
     |████████████████████████████████| 51kB 5.7MB/s 
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in /usr/local/lib/python3.7/dist-packages (from rich->aicrowd-cli==0.1) (2.6.1)
Collecting smmap<4,>=3.0.1
  Downloading https://files.pythonhosted.org/packages/d5/1e/6130925131f639b2acde0f7f18b73e33ce082ff2d90783c436b52040af5a/smmap-3.0.5-py2.py3-none-any.whl
Installing collected packages: smmap, gitdb, gitpython, colorama, commonmark, rich, requests-toolbelt, aicrowd-cli
Successfully installed aicrowd-cli-0.1.0 colorama-0.4.4 commonmark-0.9.1 gitdb-4.0.6 gitpython-3.1.14 requests-toolbelt-0.9.1 rich-9.13.0 smmap-3.0.5

!pip = A piece of software written in Python used to install stuff

install = install 👀

Download Data

The first step is to download out train test data. We will be training a model on the train data and make predictions on test data. We submit our predictions.

Please enter your API Key from https://www.aicrowd.com/participants/me

In [ ]:
API_KEY = 'YOUR_API_KEY' 
!aicrowd login --api-key $API_KEY
API Key valid
Saved API Key successfully!

Downloading the Data here! 🤓

In [ ]:
!aicrowd dataset download --challenge rover-classification
sample_submission.csv: 100% 164k/164k [00:00<00:00, 1.96MB/s]
test.zip: 100% 66.5M/66.5M [00:07<00:00, 9.43MB/s]
train.csv: 100% 689k/689k [00:00<00:00, 4.85MB/s]
train.zip: 100% 266M/266M [00:15<00:00, 17.2MB/s]
val.csv: 100% 65.0k/65.0k [00:00<00:00, 1.45MB/s]
val.zip: 100% 26.5M/26.5M [00:01<00:00, 14.3MB/s]
In [ ]:
!rm -rf data
!mkdir data

!unzip train.zip  -d data/train
!unzip val.zip -d data/val
!unzip test.zip  -d data/test

!mv train.csv data/train.csv
!mv val.csv data/val.csv
!mv sample_submission.csv data/sample_submission.csv

Import packages

In [ ]:
import pandas as pd
from fastai.vision.all import *
from fastai.data.core import *
import os

Load Data 🍞

  • We use pandas 🐼 library to load our data.
  • Pandas loads the data into dataframes and facilitates us to analyse the data.
  • Learn more about it here 🤓
In [ ]:
data_folder = "data"

We use read_csv to read the csv we have at hand. os.path.join means joining the path specified as "data" and joining it with "train.csv".

In [ ]:
train_df = pd.read_csv(os.path.join(data_folder, "train.csv"))

Visualize the data 👀

Lets take a view at the data.

In [ ]:
train_df

We will attach ".jpg" to the ImageIDs in order to make it easier to go forward from here:

In [ ]:
train_df['ImageID'] = train_df['ImageID'].astype(str)+".jpg"
train_df

What do the images look like?

Dataloader

To make our data ready for training a model, we need to create a DataLoaders object in fastai. It is just a wrapper around a training DataLoader and a validation DataLoader , so if you already have your own PyTorch dataloaders, you can create such an object directly.

FastAI allows you to create custom datasets and implement data loaders upon then. This makes programming in FastAI very flexible! ✈

There are other factory methods of ImageDataLoaders that could be more suitable for your problem, so make sure to check them all in vision.data.

In [ ]:
dls = ImageDataLoaders.from_df(train_df, path=os.path.join(data_folder, "train"), bs=8)
dls.show_batch()

TRAINING PHASE 🏋️

Then we can create a cnn_learner, which is a fastai object that combines the data and a model for training, and uses transfer learning to fine tune a pretrained model in just two lines of code:

What's a pretrained model?

What is a Pre-trained Model? Simply put, a pre-trained model is a model created by some one else to solve a similar problem. Instead of building a model from scratch to solve a similar problem, you use the model trained on other problem as a starting point.

And when you pick that model up to use it for your own task, it's called transfer learning!

What is Resnet18?

ResNet-18 is a convolutional neural network that is 18 layers deep. You can load a pretrained version of the network trained on more than a million images from the ImageNet database

In [ ]:
learn = cnn_learner(dls, resnet18, metrics=F1Score())

Train the Model 🏃🏽‍♀️

Fine tuning here refers to fitting the model to the data that we are providing to it 🙌🏽

In [ ]:
learn.fine_tune(1)

Again, we can have a look at some predictions with show_results:

In [ ]:
learn.show_results()

Testing Phase 😅

We are almost done. We trained and validated on the training data. Now its the time to predict on test set and make a submission.# Prediction on Evaluation Set

Load Test Set

Load the test data on which final submission is to be made.

In [ ]:
test_imgs_name = get_image_files(os.path.join(data_folder, "test"))

test_dls = dls.test_dl(test_imgs_name)

# Convert categorical values into label names
# Basically maps perservance as "1" and Curiosity as "0"
class_to_label_mapping = {v: k for v, k in enumerate(dls.vocab)}
print(class_to_label_mapping)

test_img_ids = [re.sub(r"\D", "", str(img_name)) for img_name in test_imgs_name]
In [ ]:
test_dls.show_batch()

Predict Test Set

Predict on the test set and you are all set to make the submission!

In [ ]:
_,_,results = learn.get_preds(dl = test_dls, with_decoded = True)

results = [class_to_label_mapping[i] for i in results.numpy()]

Save the prediction to csv

🚧 Note :

  • Do take a look at the submission format.
  • The submission file should contain a header.
  • Follow all submission guidelines strictly to avoid inconvenience.
In [ ]:
submission = pd.DataFrame({"ImageID":test_img_ids, "label":results})
submission
In [ ]:
submission.to_csv("submission.csv", index=False)

Making Direct Submission thought Aicrowd CLI

In [ ]:
!aicrowd submission create -c rover-classification -f submission.csv
Please accept challenge terms before making submission here: www.aicrowd.com/challenges/ai-blitz-7/challenge_rules

image.png

⁣          🎈🎈  ☁️
         🎈🎈🎈
 ☁️     🎈🎈🎈🎈
        🎈🎈🎈🎈
   ☁️    ⁣🎈🎈🎈
           \|/
           🏠   ☁️
   ☁️         ☁️

🌳🌹🏫🌳🏢🏢_🏢🏢🌳🌳


Comments

You must login before you can post a comment.

Execute