AI Blitz #6
Learn from started code
Using started codes shared by admin and improve accuracy for the image classifier problems
demarsylvain
Using started codes shared by admin and improve accuracy 98% for the image problems
Learn from started codes¶
- Motivation
As a chess player (♟♞ ELO ± 1,600) and data scientist (R pseudo expert), this AI Blitz caught my interest! But my expertise on image/video analysis is weak. Although I know the theory of neural network, I never have the chance to practice.
Could this challenge be a first introduction ?
--> Thanks to baseline scripts shared by the admin (big thank you to 👍 @Ashivani 👍), the answer is yes.
- Context
A) 3 of the 5 challenges are about image binary classification. From a chessboard picture, we would like to estimate which player (black or white):
- have more pieces?
- have more points?
- is the winner?
B) One challenge is about image transcription: describe FEN notation from a chessboard image.
C) The last one is about video transcription: descrive piece moves from a short video.
- AICrowd connexion 🔌
!pip install --upgrade fastai git+https://gitlab.aicrowd.com/yoogottamk/aicrowd-cli.git >/dev/null
%load_ext aicrowd.magic
API_KEY = '80f5b4c15de2bb95c6ef8b4dbf9264d8'
%aicrowd login --api-key $API_KEY
A) Image binary classification¶
For this king of problem, a starting solution is to used a pre-trained model. Even if it was not trained with chess pictures, its previous learnings could be robust enough for other problems. As chess pictures of these challenges are quite clean (similar size, easily readible, etc.), it could work without specifically building new layer.
The baseline propose to use AlexNet, a Convolutional Neural Network (CNN) designed for classify images. Let's see what it will done on these chess pictures.
Import Packages 📦¶
import pandas as pd
from fastai.vision.all import *
from fastai.data.core import *
import os
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
Access Data ♚♕♜♘♝♙¶
We start by downloading the zipfile, and unzip them. Let's start with the first challenge (Pieces).
- Pieces
%aicrowd dataset download --challenge chess-pieces -j 3
!rm -rf data
!mkdir data
!mkdir data/pieces
!unzip train.zip -d data/pieces/
!unzip val.zip -d data/pieces/
!unzip test.zip -d data/pieces/
!mv train.csv data/pieces/train.csv
!mv val.csv data/pieces/val.csv
!mv sample_submission.csv data/pieces/sample_submission.csv
We can visualize a summary table, containing image names and labels.
train_df = pd.read_csv("data/pieces/train.csv")
train_df['ImageID'] = train_df['ImageID'].astype(str)+".jpg"
train_df
We can also visualise some training images (chessboard with label)
dls = ImageDataLoaders.from_df(train_df, path="data/pieces/train", bs=8)
dls.show_batch()
Pre-trained model 💪¶
learn = cnn_learner(dls, alexnet, metrics = F1Score())
learn.fit(1)
For the 3 image binary classification problem (Pieces, Points and WinPrediction), the accuracy with this model is around 60 %. As it's my first Python neural network code, I start by playing with it, trying to improve the accuracy.
Improvments tests 🧪¶
There are several ways to (try to) improve such a solution:
Increase number of epochs of the neural network
learn.fit(4)Increase number of training images, by transforming existing ones for instance
aug_transforms(do_flip = False)Test different parameters of the neural network (learning rate for instance)
learn.lr_find() learn.fit_one_cycle(2, lr_max = 1e-3)Test other pre-trained classifier
Results 📝¶
- Pieces
With the different options described above, I managed to increase the accuracy for the first challenge - which players have more pieces - to an unexpected level (99.99 % 😜) using the pre-trained model ResNet-50 and only 3 epochs (see below). There are several participants with a similar performance (and it should increase day after day).
- Points and WinPrediction
With a similar approch, I reached accuracy of 98.8% for Points challenge, and 94% for WinPrediction. Some other participants manage to get better accuracy (close 100 %), meaning there are other improvments I have to do.
At the beginning of the training, accuracy increase with epoch number, but quickly (7-8 epochs) converge, meaning the algorithm will overfit if continue. Other kind of improvment should be explore.
- Code
dls = ImageDataLoaders.from_df(train_df, path="data/pieces/train", bs=8)
learn = cnn_learner(dls, models.resnet50, metrics = F1Score())
learn.fine_tune(3)
Submission ✉¶
We can use this solution to predict on the test dataset, and submit on the challenge to be sure our solution has a similar accuracy, and it didn't overfitting the training dataset.
test_imgs_name = get_image_files("data/pieces/test")
test_dls = dls.test_dl(test_imgs_name)
label_to_class_mapping = {v: k for v, k in enumerate(dls.vocab)}
test_img_ids = [re.sub(r"\D", "", str(img_name)) for img_name in test_imgs_name]
_,_,results = learn.get_preds(dl = test_dls, with_decoded = True)
results = [label_to_class_mapping[i] for i in results.numpy()]
submission = pd.DataFrame({"ImageID":test_img_ids, "label":results})
submission.to_csv("submission.csv", index=False)
%aicrowd submission create -c chess-pieces -f submission.csv
B) Image transcription¶
(on going)
Import Packages 📦¶
from tqdm.notebook import tqdm
import numpy as np
import os
import glob
import re
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from random import shuffle
from skimage.util.shape import view_as_blocks
from skimage import io, transform
import keras
from keras.applications.vgg16 import VGG16
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.convolutional import Convolution2D
import warnings
warnings.filterwarnings('ignore')
import pandas as pd
Access Data ♚♕♜♘♝♙¶
%aicrowd dataset download --challenge chess-configuration -j 3
!mkdir data
#!unzip train.zip -d data/config/
#!unzip val.zip -d data/config/
!unzip test.zip -d data/config/
!mv train.zip data/config/train.zip
!mv train.csv data/config/train.csv
!mv val.csv data/config/val.csv
!mv val.zip data/config/val.zip
!mv test.zip data/config/test.zip
!mv sample_submission.csv data/config/sample_submission.csv
train = glob.glob("data/config/train/*.jpg")
test = glob.glob("data/config/test/*.jpg")
train[0]
train_csv = pd.read_csv("data/config/train.csv")
train_csv['ImageID'] = train_csv['ImageID'].astype(str)+".jpg"
train_csv
f, axarr = plt.subplots(1,3, figsize=(80, 80))
for i in range(0,3):
axarr[i].set_title(train_csv['ImageID'][i] + '\n' + train_csv['label'][i], fontsize=50, pad=30)
axarr[i].imshow(mpimg.imread('data/config/train/' + train_csv['ImageID'][i]))
axarr[i].axis('off')