Loading

MABe Task 2: Annotation Style Transfer

[Task 2] Annotation Style Transfer [Baseline]

Baseline notebook for MABe Annotation Style Transfer Task

ashivani

AIcrowd-Logo

Join the communty!
chat on Discord

🐀🐀🐀🐀🐀🐀🐀🐀🐀🐀🐀🐁🐁🐁🐁🐁🐁🐁🐁🐁🐁
🐀 MABe Annotation Style Transfer: Baseline 🐁
🐀🐀🐀🐀🐀🐀🐀🐀🐀🐀🐀🐁🐁🐁🐁🐁🐁🐁🐁🐁🐁

How to use this notebook 📝

  1. Copy the notebook. This is a shared template and any edits you make here will not be saved. You should copy it into your own drive folder. For this, click the "File" menu (top-left), then "Save a Copy in Drive". You can edit your copy however you like.
  2. Link it to your AIcrowd account. In order to submit your predictions to AIcrowd, you need to provide your account's API key.

Setup AIcrowd Utilities 🛠

In [ ]:
!pip install -U aicrowd-cli

Install packages 🗃

Please add all pacakages installations in this section

In [ ]:
!pip install tensorflow-addons
Collecting tensorflow-addons
  Downloading https://files.pythonhosted.org/packages/74/e3/56d2fe76f0bb7c88ed9b2a6a557e25e83e252aec08f13de34369cd850a0b/tensorflow_addons-0.12.1-cp37-cp37m-manylinux2010_x86_64.whl (703kB)
     |████████████████████████████████| 706kB 11.5MB/s 
Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow-addons) (2.7.1)
Installing collected packages: tensorflow-addons
Successfully installed tensorflow-addons-0.12.1

Import necessary modules and packages 📚

In [ ]:
import numpy as np
import os

from tensorflow import keras
import tensorflow as tf
from keras.models import Sequential
import keras.layers as layers
import tensorflow_addons as tfa

import sklearn
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import train_test_split
from copy import deepcopy
import tqdm
import gc

Download the dataset 📲

Please get your API key from https://www.aicrowd.com/participants/me

In [ ]:
API_KEY = "53f83c19712d082628cb243559a0fb5c"
!aicrowd login --api-key $API_KEY
API Key valid
Saved API Key successfully!
In [ ]:
!aicrowd dataset download --challenge mabe-task-2-annotation-style-transfer
test-release.npy: 100% 1.83G/1.83G [01:54<00:00, 15.9MB/s]
sample-submission.npy: 100% 65.4M/65.4M [00:04<00:00, 16.1MB/s]
train.npy: 100% 135M/135M [00:07<00:00, 18.0MB/s]

Extract the downloaded dataset to data directory

In [ ]:
!rm -rf data
!mkdir data
 
!mv train.npy data/train.npy
!mv test-release.npy data/test.npy
!mv sample-submission.npy data/sample_submission.npy

Load Data

The dataset files are python dictionaries, this is a descirption of how the data is organized.

In [ ]:
train = np.load('data/train.npy',allow_pickle=True).item()
test = np.load('data/test.npy',allow_pickle=True).item()
sample_submission = np.load('data/sample_submission.npy',allow_pickle=True).item()

Dataset Specifications 💾

  • train.npy - Training set for the task, which follows the following schema:

           

  • test-release.npy - Test set for the task, which follows the following schema :

           

  • sample_submission.npy - Template for a sample submission which follows the following schema
{
    "<sequence_id-1>" : [0, 0, 1, 2, ...],
    "<sequence_id-2>" : [0, 1, 2, 0, ...]
}

Each key in the dictionary here refers to the unique sequence id obtained for the sequences in the test set. The value for each of the keys is expected to hold a list of corresponing annotations. The annotations are represented by the index of the corresponding annotation words in the vocabular provided in the test set.

How does the data look like? 🔍

Data overview

In [ ]:
print("Dataset keys - ", train.keys())
print("Vocabulary - ", train['vocabulary'])
print("Number of train Sequences - ", len(train['sequences']))
print("Number of test Sequences - ", len(test['sequences']))
Dataset keys -  dict_keys(['vocabulary', 'sequences'])
Vocabulary -  {'attack': 0, 'investigation': 1, 'mount': 2, 'other': 3}
Number of train Sequences -  30
Number of test Sequences -  458

Sample overview

In [ ]:
sequence_names = list(train["sequences"].keys())
sequence_key = sequence_names[0]
single_sequence = train["sequences"][sequence_key]
print("Sequence name - ", sequence_key)
print("Single Sequence keys", single_sequence.keys())
print(f"Number of Frames in {sequence_key} - ", len(single_sequence['annotations']))
print(f"Keypoints data shape of {sequence_key} - ", single_sequence['keypoints'].shape)
print(f"annotator_id of {sequence_key} - ", single_sequence['annotator_id'])
Sequence name -  43b6e939bd
Single Sequence keys dict_keys(['keypoints', 'annotator_id', 'annotations'])
Number of Frames in 43b6e939bd -  17415
Keypoints data shape of 43b6e939bd -  (17415, 2, 2, 7)
annotator_id of 43b6e939bd -  1

Whats different in Task 2

Task 2 is all about transferring the style of annotation for the same behaviors. The dataset contains "annotator_id" for each sequence.

In [ ]:
def anno_id_counts(dataset):
  all_annotator_ids = [dataset["sequences"][k]['annotator_id'] for k in dataset["sequences"]]
  unique_annotator_ids, annotator_id_counts = np.unique(all_annotator_ids, return_counts=True)
  for uaid, aic in zip(unique_annotator_ids, annotator_id_counts):
      print(f"Annotator id: {uaid} |  Number of sequences: {aic}")
  
print("Train")
anno_id_counts(train)
print()
print("Test")
anno_id_counts(test)
Train
Annotator id: 1 |  Number of sequences: 6
Annotator id: 2 |  Number of sequences: 6
Annotator id: 3 |  Number of sequences: 6
Annotator id: 4 |  Number of sequences: 6
Annotator id: 5 |  Number of sequences: 6

Test
Annotator id: 0 |  Number of sequences: 411
Annotator id: 1 |  Number of sequences: 13
Annotator id: 2 |  Number of sequences: 6
Annotator id: 3 |  Number of sequences: 4
Annotator id: 4 |  Number of sequences: 4
Annotator id: 5 |  Number of sequences: 20

Helper function for visualization 💁

Don't forget to run the cell 😉

In [ ]:
import matplotlib.pyplot as plt
from matplotlib import animation
from matplotlib import colors
from matplotlib import rc
 
rc('animation', html='jshtml')
 
# Note: Image processing may be slow if too many frames are animated.                
 
#Plotting constants
FRAME_WIDTH_TOP = 1024
FRAME_HEIGHT_TOP = 570
 
RESIDENT_COLOR = 'lawngreen'
INTRUDER_COLOR = 'skyblue'
 
PLOT_MOUSE_START_END = [(0, 1), (0, 2), (1, 3), (2, 3), (3, 4),
                        (3, 5), (4, 6), (5, 6), (1, 2)]
 
class_to_color = {'other': 'white', 'attack' : 'red', 'mount' : 'green',
                  'investigation': 'orange'}
 
class_to_number = {s: i for i, s in enumerate(train['vocabulary'])}
 
number_to_class = {i: s for i, s in enumerate(train['vocabulary'])}
 
def num_to_text(anno_list):
  return np.vectorize(number_to_class.get)(anno_list)
 
def set_figax():
    fig = plt.figure(figsize=(6, 4))
 
    img = np.zeros((FRAME_HEIGHT_TOP, FRAME_WIDTH_TOP, 3))
 
    ax = fig.add_subplot(111)
    ax.imshow(img)
 
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
 
    return fig, ax
 
def plot_mouse(ax, pose, color):
    # Draw each keypoint
    for j in range(7):
        ax.plot(pose[j, 0], pose[j, 1], 'o', color=color, markersize=5)
 
    # Draw a line for each point pair to form the shape of the mouse
 
    for pair in PLOT_MOUSE_START_END:
        line_to_plot = pose[pair, :]
        ax.plot(line_to_plot[:, 0], line_to_plot[
                :, 1], color=color, linewidth=1)
 
def animate_pose_sequence(video_name, keypoint_sequence, start_frame = 0, stop_frame = 100, 
                          annotation_sequence = None):
    # Returns the animation of the keypoint sequence between start frame
    # and stop frame. Optionally can display annotations.
    seq = keypoint_sequence.transpose((0,1,3,2))
 
    image_list = []
    
    counter = 0
    for j in range(start_frame, stop_frame):
        if counter%20 == 0:
          print("Processing frame ", j)
        fig, ax = set_figax()
        plot_mouse(ax, seq[j, 0, :, :], color=RESIDENT_COLOR)
        plot_mouse(ax, seq[j, 1, :, :], color=INTRUDER_COLOR)
        
        if annotation_sequence is not None:
          annot = annotation_sequence[j]
          annot = number_to_class[annot]
          plt.text(50, -20, annot, fontsize = 16, 
                   bbox=dict(facecolor=class_to_color[annot], alpha=0.5))
 
        ax.set_title(
            video_name + '\n frame {:03d}.png'.format(j))
 
        ax.axis('off')
        fig.tight_layout(pad=0)
        ax.margins(0)
 
        fig.canvas.draw()
        image_from_plot = np.frombuffer(fig.canvas.tostring_rgb(),
                                        dtype=np.uint8)
        image_from_plot = image_from_plot.reshape(
            fig.canvas.get_width_height()[::-1] + (3,)) 
 
        image_list.append(image_from_plot)
 
        plt.close()
        counter = counter + 1
 
    # Plot animation.
    fig = plt.figure()
    plt.axis('off')
    im = plt.imshow(image_list[0])
 
    def animate(k):
        im.set_array(image_list[k])
        return im,
    ani = animation.FuncAnimation(fig, animate, frames=len(image_list), blit=True)
    return ani
 
def plot_annotation_strip(annotation_sequence, start_frame = 0, stop_frame = 100, title="Behavior Labels"):
  # Plot annotations as a annotation strip.
 
  # Map annotations to a number.
  annotation_num = []
  for item in annotation_sequence[start_frame:stop_frame]:
    annotation_num.append(class_to_number[item])
 
  all_classes = list(set(annotation_sequence[start_frame:stop_frame]))
 
  cmap = colors.ListedColormap(['red', 'orange', 'green', 'white'])
  bounds=[-0.5,0.5,1.5, 2.5, 3.5]
  norm = colors.BoundaryNorm(bounds, cmap.N)
 
  height = 200
  arr_to_plot = np.repeat(np.array(annotation_num)[:,np.newaxis].transpose(),
                                                  height, axis = 0)
  
  fig, ax = plt.subplots(figsize = (16, 3))
  ax.imshow(arr_to_plot, interpolation = 'none',cmap=cmap, norm=norm)
 
  ax.set_yticks([])
  ax.set_xlabel('Frame Number')
  plt.title(title)
 
  import matplotlib.patches as mpatches
 
  legend_patches = []
  for item in all_classes:
    legend_patches.append(mpatches.Patch(color=class_to_color[item], label=item))
 
  plt.legend(handles=legend_patches,loc='center left', bbox_to_anchor=(1, 0.5))
 
  plt.tight_layout()

Visualize the mouse movements🎥

Sample visualization for plotting pose gifs.

In [ ]:
keypoint_sequence = single_sequence['keypoints']
annotation_sequence = single_sequence['annotations']

ani = animate_pose_sequence(sequence_key,
                            keypoint_sequence, 
                            start_frame = 3000,
                            stop_frame = 3100,
                            annotation_sequence = annotation_sequence)

# Display the animaion on colab
ani
Processing frame  3000
Processing frame  3020
Processing frame  3040
Processing frame  3060
Processing frame  3080
Out[ ]:

Showing a section of the validation data (Index needs to be selected for a full video)

In [ ]:
annotation_sequence = single_sequence['annotations']
text_sequence = num_to_text(annotation_sequence)
 
plot_annotation_strip(
    text_sequence,
    start_frame=0,
    stop_frame=len(annotation_sequence) + 1000
)

Basic EDA 🤓

There are 5 annotators in the train set, all of them label videos slightly differently, here we look at the percentages of each video annotator wise. Note the annotations from different annotators are not for the same videos.

Each sequence has different amounts of each behavior, here we get the percentage of frames of each behavior in each sequence. We can use this to split the dataset for validation in a stratified way.

In [ ]:
# Function for showing dataframes nicely on jupyter
from IPython.display import display, HTML
def pretty_print_dataframe(df):
  display(HTML(df.to_html()))
In [ ]:
vocabulary = train['vocabulary']
def get_percentage(sequence_key):
  anno_seq = num_to_text(train['sequences'][sequence_key]['annotations'])
  counts = {k: np.mean(np.array(anno_seq) == k) for k in vocabulary}
  return counts

anno_percentages = {k: get_percentage(k) for k in train['sequences']}

anno_perc_df = pd.DataFrame(anno_percentages).T
anno_perc_df['annotator_id'] = [seq['annotator_id'] for k, seq in train['sequences'].items()]
print("Percentage of frames in every sequence for every class")
for anno in anno_perc_df['annotator_id'].unique():
  pretty_print_dataframe(anno_perc_df[anno_perc_df['annotator_id'] == anno])
Percentage of frames in every sequence for every class
attack investigation mount other annotator_id
43b6e939bd 0.112374 0.255067 0.000000 0.632558 1
5e11824914 0.000000 0.166901 0.311344 0.521754 1
e23a8798d2 0.000000 0.121332 0.193692 0.684975 1
9aa7834334 0.104795 0.228481 0.000000 0.666724 1
f636d8ce50 0.042356 0.407999 0.000000 0.549646 1
6ff4efa5af 0.000000 0.163359 0.267215 0.569426 1
attack investigation mount other annotator_id
164fea19b9 0.000000 0.241118 0.128168 0.630714 2
d74df16abe 0.063488 0.453967 0.022614 0.459931 2
49a06c058d 0.074868 0.167778 0.001093 0.756260 2
8272b0514e 0.000000 0.290838 0.049824 0.659338 2
5b0c4bcc5b 0.000000 0.236804 0.026842 0.736354 2
a9797dafad 0.155806 0.120652 0.006901 0.716641 2
attack investigation mount other annotator_id
37685eaeab 0.000000 0.182941 0.118722 0.698337 3
1d1fbf825b 0.000000 0.347086 0.117151 0.535763 3
dfc28b400a 0.000000 0.228521 0.348493 0.422986 3
3d85aa9112 0.077094 0.081905 0.002703 0.838298 3
468f81c924 0.102584 0.046779 0.000000 0.850637 3
92d65c5928 0.264395 0.053320 0.000000 0.682285 3
attack investigation mount other annotator_id
55336c1c10 0.000000 0.424183 0.056618 0.519199 4
5dc3bea88e 0.000000 0.175984 0.112424 0.711592 4
f00a01799b 0.233328 0.006733 0.000000 0.759939 4
52ac2359fb 0.139976 0.030842 0.000000 0.829181 4
8060ae3dcf 0.107315 0.075904 0.000000 0.816781 4
37c778250f 0.000000 0.174082 0.089087 0.736831 4
attack investigation mount other annotator_id
0cc9e91c16 0.036092 0.162057 0.000000 0.801851 5
ae2efcbabd 0.000000 0.188581 0.029831 0.781588 5
20bcf954d4 0.079431 0.266523 0.000000 0.654046 5
99af4715e3 0.039112 0.229705 0.000000 0.731183 5
f8e8e4a375 0.000000 0.448194 0.011593 0.540213 5
a104a6d76c 0.000000 0.345275 0.029140 0.625585 5

Percentage Frames of all behaviors

Lets look at the class imbalance for every annotator

In [ ]:
for annotator_id in anno_perc_df['annotator_id'].unique():
  all_annotations = []
  for sk, sequence in train['sequences'].items():
    if not sequence['annotator_id'] == annotator_id:
      continue
    annotations = sequence['annotations']
    all_annotations.extend(list(annotations))
  all_annotations = num_to_text(all_annotations)
  classes, counts = np.unique(all_annotations, return_counts=True)
  print("Annotator: ", annotator_id)
  percentages = {"Behavior": classes, "Percentage Frames": counts/len(all_annotations)}
  pretty_print_dataframe(pd.DataFrame(percentages))
Annotator:  1
Behavior Percentage Frames
0 attack 0.032468
1 investigation 0.205776
2 mount 0.161987
3 other 0.599769
Annotator:  2
Behavior Percentage Frames
0 attack 0.052742
1 investigation 0.258164
2 mount 0.033984
3 other 0.655110
Annotator:  3
Behavior Percentage Frames
0 attack 0.080907
1 investigation 0.134947
2 mount 0.095047
3 other 0.689099
Annotator:  4
Behavior Percentage Frames
0 attack 0.086887
1 investigation 0.116234
2 mount 0.044560
3 other 0.752319
Annotator:  5
Behavior Percentage Frames
0 attack 0.030254
1 investigation 0.264150
2 mount 0.010272
3 other 0.695323

Training The Model 🏋️‍♂️

The given MABe dataset contain many sequences of time series data, each frame has its own behavior label. Training on just a single frame does not give good results due to less information.

So here past and future frames are also added to each input. But also all the frames are not concatenated as as the boundaries of the past and future frames need to stay separate for each video.

Seeding helper

Its good practice to seed before every run, that way its easily reproduced.

In [ ]:
def seed_everything(seed):
  np.random.seed(seed)
  os.environ['PYTHONHASHSEED'] = str(seed)
  tf.random.set_seed(seed)

seed=2021
seed_everything(seed)

Generator 🔌

The generator is used to take input winodws from each sequence after randomly sampling frames.

It also provides code for augmentations

  1. Random rotation
  2. Random translate

🚧 Note that these augmentations are applied in the same across all frames in a selected window, e.g - Random rotation by 10 degrees will rotate all frames in the input window by the same angle.

In [ ]:
class MABe_Generator(keras.utils.Sequence):
    def __init__(self, pose_dict, 
                 batch_size, dim, 
                 use_conv, num_classes, augment=False,
                 class_to_number=None,
                 past_frames=0, future_frames=0, 
                 frame_gap=1, shuffle=False,
                 mode='fit'):
        self.batch_size = batch_size
        self.video_keys = list(pose_dict.keys())
        self.dim = dim
        self.use_conv = use_conv
        self.past_frames = past_frames
        self.future_frames = future_frames
        self.frame_gap = frame_gap
        self.shuffle = shuffle
        self.num_classes=num_classes
        self.augment = augment
        self.mode = mode

        self.class_to_number = class_to_number

        self.video_indexes = []
        self.frame_indexes = []
        self.X = {}
        if self.mode == 'fit':
          self.y = []
        self.pad = self.past_frames * self.frame_gap
        future_pad = self.future_frames * self.frame_gap
        pad_width = (self.pad, future_pad), (0, 0), (0, 0), (0, 0)
        self.seq_lengths = {}
        for vc, key in enumerate(self.video_keys):
          if self.mode == 'fit':
            anno = pose_dict[key]['annotations']
            self.y.extend(anno)
          nframes = len(pose_dict[key]['keypoints'])
          self.video_indexes.extend([vc for _ in range(nframes)])
          self.frame_indexes.extend(range(nframes))
          self.X[key] = np.pad(pose_dict[key]['keypoints'], pad_width)
          self.seq_lengths[key] = nframes
        
        if self.mode == 'fit':
          self.y = np.array(self.y)
        
        self.X_dtype = self.X[key].dtype

        self.indexes = list(range(len(self.frame_indexes)))

        if self.mode == 'predict':
          extra_predicts = -len(self.indexes) % self.batch_size # So that last part is not missed
          self.indexes.extend(self.indexes[:extra_predicts])
          self.indexes = np.array(self.indexes)
        
        self.on_epoch_end()

    def __len__(self):
        return len(self.indexes) // self.batch_size

    def augment_fn(self, x):
      # Rotate
      angle = (np.random.rand()-0.5) * (np.pi * 2)
      c, s = np.cos(angle), np.sin(angle)
      rot = np.array([[c, -s], [s, c]])
      x = np.dot(x, rot)

      # Shift - All get shifted together
      shift = (np.random.rand(2)-0.5) * 2 * 0.25
      x = x + shift
      return x

    def __getitem__(self, index):
        bs = self.batch_size
        indexes = self.indexes[index*bs:(index+1)*bs]
        X = np.empty((bs, *self.dim), self.X_dtype)
        if self.mode == 'predict':
          vkey_fi_list = []
        for bi, idx in enumerate(indexes):
          vkey = self.video_keys[self.video_indexes[idx]]
          fi = self.frame_indexes[idx]
          if self.mode == 'predict':
            vkey_fi_list.append((vkey, fi))
          fi = fi + self.pad
          start = fi - self.past_frames*self.frame_gap
          stop = fi + (self.future_frames + 1)*self.frame_gap
          assert start >= 0

          Xi = self.X[vkey][start:stop:self.frame_gap].copy()
          
          if self.augment:
            Xi = self.augment_fn(Xi)
          X[bi] = np.reshape(Xi, self.dim)
          

        if self.mode == 'fit':
          y_vals = self.y[indexes]
          # Converting to one hot because F1 callback needs one hot
          y = np.zeros( (bs,self.num_classes), np.float32)
          y[np.arange(bs), y_vals] = 1
          return X, y

        elif self.mode == 'predict':
          return X, vkey_fi_list

    def on_epoch_end(self):
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

Trainer 🏋️

The trainer class implements a unified interface for using the datagenerator.

It supports fully connected or 1D convolutional networks, as well as other hyperparameters for the model and the generator.

In [ ]:
class Trainer:
  def __init__(self, *,
               train_data,
               val_data,
               test_data,
               feature_dim, 
               batch_size, 
               num_classes,
               augment=False,
               class_to_number=None,
               past_frames=0, 
               future_frames=0,
               frame_gap=1, 
               use_conv=False):
    flat_dim = np.prod(feature_dim)
    if use_conv:
      input_dim = ((past_frames + future_frames + 1), flat_dim,)
    else:
      input_dim = (flat_dim * (past_frames + future_frames + 1),)

    self.input_dim = input_dim
    self.use_conv=use_conv
    self.num_classes=num_classes

    c2n = {'other': 0,'investigation': 1,
                'attack' : 2, 'mount' : 3}
    self.class_to_number = class_to_number or c2n

    self.train_generator = MABe_Generator(train_data, 
                                      batch_size=batch_size, 
                                      dim=input_dim,
                                      num_classes=num_classes, 
                                      past_frames=past_frames, 
                                      future_frames=future_frames,
                                      class_to_number=self.class_to_number,
                                      use_conv=use_conv,
                                      frame_gap=frame_gap,
                                      augment=augment,
                                      shuffle=True,
                                      mode='fit')

    self.val_generator = MABe_Generator(val_data, 
                                        batch_size=batch_size, 
                                        dim=input_dim, 
                                        num_classes=num_classes, 
                                        past_frames=past_frames,
                                        future_frames=future_frames,
                                        use_conv=use_conv,
                                        class_to_number=self.class_to_number,
                                        frame_gap=frame_gap,
                                        augment=False,
                                        shuffle=False,
                                        mode='fit')
    
    self.test_generator = MABe_Generator(test_data, 
                                        batch_size=1024, 
                                        dim=input_dim, 
                                        num_classes=num_classes, 
                                        past_frames=past_frames,
                                        future_frames=future_frames,
                                        use_conv=use_conv,
                                        class_to_number=self.class_to_number,
                                        frame_gap=frame_gap,
                                        augment=False,
                                        shuffle=False,
                                        mode='predict')
  
  def delete_model(self):
    self.model = None
  
  def initialize_model(self, layer_channels=(512, 256), dropout_rate=0., 
                       learning_rate=1e-3, conv_size=5):

    def add_dense_bn_activate(model, out_dim, activation='relu', drop=0.):
      model.add(layers.Dense(out_dim))
      model.add(layers.BatchNormalization())
      model.add(layers.Activation('relu'))
      if drop > 0:
        model.add(layers.Dropout(rate=drop))
      return model
    
    def add_conv_bn_activate(model, out_dim, activation='relu', conv_size=3, drop=0.):
      model.add(layers.Conv1D(out_dim, conv_size))
      model.add(layers.BatchNormalization())
      model.add(layers.Activation('relu'))
      model.add(layers.MaxPooling1D(2, 2))
      if drop > 0:
        model.add(layers.Dropout(rate=drop))
      return model

    model = Sequential()
    model.add(layers.Input(self.input_dim))
    model.add(layers.BatchNormalization())
    for ch in layer_channels:
      if self.use_conv:
        model = add_conv_bn_activate(model, ch, conv_size=conv_size,
                                     drop=dropout_rate)
      else:
        model = add_dense_bn_activate(model, ch, drop=dropout_rate)
    model.add(layers.Flatten())
    model.add(layers.Dense(self.num_classes, activation='softmax'))

    metrics = [tfa.metrics.F1Score(num_classes=self.num_classes)]
    optimizer = keras.optimizers.Adam(lr=learning_rate)
    model.compile(loss='categorical_crossentropy', 
                  optimizer=optimizer,
                  metrics=metrics)

    self.model = model

  def _set_model(self, model):
      """ Set an external, provide initialized and compiled keras model """
      self.model = model

  def train(self, epochs=20, class_weight=None):
    if self.model is None:
      print("Please Call trainer.initialize_model first")
      return
    self.model.fit(self.train_generator,
          validation_data=self.val_generator,
          epochs=epochs,
          class_weight=class_weight)
        
  def get_validation_labels(self, on_test_set=False):
    y_val = []
    for _, y in self.val_generator:
      y_val.extend(list(y))
    y_val = np.argmax(np.array(y_val), axis=-1)
    return y_val

  def get_validation_predictions(self):
    y_val_pred = self.model.predict(self.val_generator)
    y_val_pred = np.argmax(y_val_pred, axis=-1)
    return y_val_pred

  def get_validation_metrics(self):
    y_val = self.get_validation_labels()
    y_val_pred = self.get_validation_predictions()

    f1_scores = sklearn.metrics.f1_score(y_val, y_val_pred,average=None)
    rec_scores = sklearn.metrics.precision_score(y_val, y_val_pred,average=None)
    prec_scores = sklearn.metrics.recall_score(y_val, y_val_pred,average=None)
    classes = list(self.class_to_number.keys())
    metrics = pd.DataFrame({"Class": classes, "F1": f1_scores, "Precision": prec_scores, "Recall": rec_scores})
    return metrics
  
  def get_test_predictions(self):
    all_test_preds = {}
    for vkey in self.test_generator.video_keys:
      nframes = self.test_generator.seq_lengths[vkey]
      all_test_preds[vkey] = np.zeros(nframes, dtype=np.int32)

    for X, vkey_fi_list in tqdm.tqdm(self.test_generator):
      test_pred = self.model.predict(X)
      test_pred = np.argmax(test_pred, axis=-1)

      for p, (vkey, fi) in zip(test_pred, vkey_fi_list):
        all_test_preds[vkey][fi] = p
    return all_test_preds

Preprocess

We'll normalize the data based on the information that the frame size is 1024x570

The original data is of shape (sequence length, mouse, x y coordinate, keypoint) = (length, 2, 2, 7)

We'll swap the x y and the keypoint axis, which will help in rotation augmentation.

Preprocess and Split data

In [ ]:
def normalize_data(orig_pose_dictionary):
  for key in orig_pose_dictionary:
    X = orig_pose_dictionary[key]['keypoints']
    X = X.transpose((0,1,3,2)) #last axis is x, y coordinates
    X[..., 0] = X[..., 0]/1024
    X[..., 1] = X[..., 1]/570
    orig_pose_dictionary[key]['keypoints'] = X
  return orig_pose_dictionary
In [ ]:
def split_validation(orig_pose_dictionary, vocabulary, seed=2021, 
                       test_size=0.5, split_videos=False):
  if split_videos:
    pose_dictionary = {}
    for key in orig_pose_dictionary:
      key_pt1 = key + '_part1'
      key_pt2 = key + '_part2'
      anno_len = len(orig_pose_dictionary[key]['annotations'])
      split_idx = anno_len//2
      pose_dictionary[key_pt1] = {
          'annotations': orig_pose_dictionary[key]['annotations'][:split_idx],
          'keypoints': orig_pose_dictionary[key]['keypoints'][:split_idx]}
      pose_dictionary[key_pt2] = {
          'annotations': orig_pose_dictionary[key]['annotations'][split_idx:],
          'keypoints': orig_pose_dictionary[key]['keypoints'][split_idx:]}
  else:
    pose_dictionary = orig_pose_dictionary
  
  def get_percentage(sequence_key):
    anno_seq = num_to_text(pose_dictionary[sequence_key]['annotations'])
    counts = {k: np.mean(np.array(anno_seq) == k) for k in vocabulary}
    return counts

  anno_percentages = {k: get_percentage(k) for k in pose_dictionary}

  anno_perc_df = pd.DataFrame(anno_percentages).T

  rng_state = np.random.RandomState(seed)
  try:
    idx_train, idx_val = train_test_split(anno_perc_df.index,
                                      stratify=anno_perc_df['attack'] > 0, 
                                      test_size=test_size,
                                      random_state=rng_state)
  except:
    idx_train, idx_val = train_test_split(anno_perc_df.index,
                                      test_size=test_size,
                                      random_state=rng_state)
    
  train_data = {k : pose_dictionary[k] for k in idx_train}
  val_data = {k : pose_dictionary[k] for k in idx_val}
  return train_data, val_data, anno_perc_df

Train function and inference

This below function is specific for Task 2, it has a set of hyperparameters we found with some tuning. Though results can be improved with further tuning.

It has the option to use a pretrained model from Task1.

It also generates the submission dictionary after training is completed.

In [ ]:
def run_task2(results_dir, dataset, vocabulary, test_data, annotator_id, pretrained_file=None, seed=2021):
  HPARAMS = {}
  val_size = HPARAMS["val_size"] = 0.3
  normalize = HPARAMS["normalize"] = True
  HPARAMS["seed"] = seed
  seed_everything(seed)
  split_videos = HPARAMS["split_videos"] = True

  if normalize:
    dataset = normalize_data(deepcopy(dataset))
    test_data = normalize_data(deepcopy(test_data))

  train_data, val_data, anno_perc_df = split_validation(dataset, 
                                                        seed=seed,
                                                        vocabulary=vocabulary,
                                                        test_size=val_size, 
                                                        split_videos=split_videos)                               
  num_classes = len(anno_perc_df.keys())
  feature_dim = HPARAMS["feature_dim"] = (2,7,2)

  # Generator parameters
  past_frames = HPARAMS["past_frames"] = 50
  future_frames = HPARAMS["future_frames"] = 50
  frame_gap = HPARAMS["frame_gap"] = 1
  use_conv = HPARAMS["use_conv"] = True
  batch_size = HPARAMS["batch_size"] = 128

  # Model parameters
  dropout_rate = HPARAMS["dropout_rate"] = 0.5
  learning_rate = HPARAMS["learning_rate"] = 5e-5
  layer_channels = HPARAMS["layer_channels"] = (128, 64, 32)
  conv_size = HPARAMS["conv_size"] = 5
  augment = HPARAMS["augment"] = True
  class_to_number = HPARAMS['class_to_number'] = vocabulary
  epochs = HPARAMS["epochs"] = 10

  trainer = Trainer(train_data=train_data,
                    val_data=val_data,
                    test_data=test_data,
                    feature_dim=feature_dim, 
                    batch_size=batch_size, 
                    num_classes=num_classes,
                    augment=augment,
                    class_to_number=class_to_number,
                    past_frames=past_frames, 
                    future_frames=future_frames,
                    frame_gap=frame_gap,
                    use_conv=use_conv)

  trainer.initialize_model(layer_channels=layer_channels,
                          dropout_rate=dropout_rate,
                          learning_rate=learning_rate,
                          conv_size=conv_size)
  
  
  if pretrained_file and os.path.exists(pretrained_file):
    HPARAMS['pretrained_file'] = pretrained_file
    
    # Freeze all layers except last and Batchnorm
    trainer.model = keras.models.load_model(pretrained_file)
    for idx, layer in enumerate(trainer.model.layers[:-1]):
      if not isinstance(layer, layers.BatchNormalization):
        trainer.model.layers[idx].trainable = False
    
    # Train linear probe
    linear_probe_lr = HPARAMS['linear_probe_lr'] = learning_rate
    trainer.model.optimizer.learning_rate.assign(linear_probe_lr)
    linear_probe_epochs = HPARAMS['linear_probe_epochs'] = 5
    trainer.train(epochs=linear_probe_epochs)

    # Unfreeze all layers
    for idx, layer in enumerate(trainer.model.layers[:-1]):
      trainer.model.layers[idx].trainable = True
    trainer.model.optimizer.learning_rate.assign(learning_rate)

  trainer.train(epochs=epochs)
  trainer.model.save(f'{results_dir}/task2_{annotator_id}.h5')
  np.save(f"{results_dir}/task2_{annotator_id}_hparams", HPARAMS)

  val_metrics = trainer.get_validation_metrics()
  test_results = trainer.get_test_predictions()

  np.save(f"{results_dir}/task2_{annotator_id}_test_results", test_results) 
  val_metrics.to_csv(f"{results_dir}/task2_{annotator_id}_metrics_val.csv", index=False)
  return test_results

Run Training for Task 2 🏃‍♂️

🚧🚧🚧🚧🚧🚧🚧🚧🚧🚧🚧🚧🚧

Since Task 2 has few sequences, it helps to use a pretrained model from Task 1

You can refer to the Task 1 baseline to get the model for that.

In [ ]:
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [ ]:
results_dir = '.'
# Need to take this file from task 1, also make sure all network params and generator parameters are same
pretrained_file = "/content/drive/MyDrive/aicrowd_mabe_models/task1_augmented.h5" 
# pretrained_file = None # If you want to skip the pretrained model part - uncomment this
anno_ids = np.unique([train["sequences"][k]['annotator_id'] for k in train["sequences"]])
submission = {}
for annotator in anno_ids:
  train_data_annotator = {skey: seq for skey, seq in train['sequences'].items() 
                            if seq['annotator_id'] == annotator}
  test_data_annotator = {skey: seq for skey, seq in test['sequences'].items() 
                            if seq['annotator_id'] == annotator}
  vocabulary = train['vocabulary']
  annotator_results =  run_task2(results_dir, 
                                  train_data_annotator,
                                  vocabulary,
                                  test_data_annotator,
                                  annotator_id='annotator%i'%annotator, 
                                  pretrained_file=pretrained_file)
  submission.update(annotator_results)
Epoch 1/5
721/721 [==============================] - 38s 28ms/step - loss: 0.2493 - f1_score: 0.8389 - val_loss: 0.2447 - val_f1_score: 0.8449
Epoch 2/5
721/721 [==============================] - 20s 27ms/step - loss: 0.2323 - f1_score: 0.8469 - val_loss: 0.2531 - val_f1_score: 0.8405
Epoch 3/5
721/721 [==============================] - 19s 27ms/step - loss: 0.2282 - f1_score: 0.8479 - val_loss: 0.2355 - val_f1_score: 0.8506
Epoch 4/5
721/721 [==============================] - 19s 27ms/step - loss: 0.2215 - f1_score: 0.8515 - val_loss: 0.2422 - val_f1_score: 0.8477
Epoch 5/5
721/721 [==============================] - 19s 27ms/step - loss: 0.2204 - f1_score: 0.8539 - val_loss: 0.2386 - val_f1_score: 0.8509
Epoch 1/10
721/721 [==============================] - 20s 27ms/step - loss: 0.2170 - f1_score: 0.8548 - val_loss: 0.2410 - val_f1_score: 0.8497
Epoch 2/10
721/721 [==============================] - 19s 27ms/step - loss: 0.2163 - f1_score: 0.8583 - val_loss: 0.2471 - val_f1_score: 0.8488
Epoch 3/10
721/721 [==============================] - 19s 27ms/step - loss: 0.2150 - f1_score: 0.8581 - val_loss: 0.2450 - val_f1_score: 0.8524
Epoch 4/10
721/721 [==============================] - 19s 27ms/step - loss: 0.2137 - f1_score: 0.8571 - val_loss: 0.2461 - val_f1_score: 0.8505
Epoch 5/10
721/721 [==============================] - 19s 27ms/step - loss: 0.2118 - f1_score: 0.8592 - val_loss: 0.2431 - val_f1_score: 0.8513
Epoch 6/10
721/721 [==============================] - 19s 27ms/step - loss: 0.2124 - f1_score: 0.8579 - val_loss: 0.2443 - val_f1_score: 0.8500
Epoch 7/10
721/721 [==============================] - 19s 27ms/step - loss: 0.2077 - f1_score: 0.8607 - val_loss: 0.2392 - val_f1_score: 0.8555
Epoch 8/10
721/721 [==============================] - 19s 27ms/step - loss: 0.2074 - f1_score: 0.8639 - val_loss: 0.2517 - val_f1_score: 0.8524
Epoch 9/10
721/721 [==============================] - 19s 27ms/step - loss: 0.2036 - f1_score: 0.8668 - val_loss: 0.2363 - val_f1_score: 0.8546
Epoch 10/10
721/721 [==============================] - 19s 27ms/step - loss: 0.2065 - f1_score: 0.8641 - val_loss: 0.2367 - val_f1_score: 0.8525
100%|██████████| 281/281 [00:36<00:00,  7.65it/s]
Epoch 1/5
729/729 [==============================] - 20s 27ms/step - loss: 0.3159 - f1_score: 0.7451 - val_loss: 0.2677 - val_f1_score: 0.8355
Epoch 2/5
729/729 [==============================] - 19s 26ms/step - loss: 0.2973 - f1_score: 0.7609 - val_loss: 0.2688 - val_f1_score: 0.8380
Epoch 3/5
729/729 [==============================] - 19s 26ms/step - loss: 0.2905 - f1_score: 0.7640 - val_loss: 0.2631 - val_f1_score: 0.8349
Epoch 4/5
729/729 [==============================] - 19s 26ms/step - loss: 0.2853 - f1_score: 0.7748 - val_loss: 0.2677 - val_f1_score: 0.8344
Epoch 5/5
729/729 [==============================] - 19s 26ms/step - loss: 0.2830 - f1_score: 0.7751 - val_loss: 0.2609 - val_f1_score: 0.8334
Epoch 1/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2779 - f1_score: 0.7770 - val_loss: 0.2662 - val_f1_score: 0.8314
Epoch 2/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2777 - f1_score: 0.7786 - val_loss: 0.2637 - val_f1_score: 0.8298
Epoch 3/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2772 - f1_score: 0.7793 - val_loss: 0.2691 - val_f1_score: 0.8343
Epoch 4/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2770 - f1_score: 0.7789 - val_loss: 0.2643 - val_f1_score: 0.8358
Epoch 5/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2730 - f1_score: 0.7848 - val_loss: 0.2658 - val_f1_score: 0.8336
Epoch 6/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2722 - f1_score: 0.7840 - val_loss: 0.2691 - val_f1_score: 0.8390
Epoch 7/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2700 - f1_score: 0.7883 - val_loss: 0.2662 - val_f1_score: 0.8392
Epoch 8/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2694 - f1_score: 0.7874 - val_loss: 0.2652 - val_f1_score: 0.8354
Epoch 9/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2689 - f1_score: 0.7884 - val_loss: 0.2660 - val_f1_score: 0.8343
Epoch 10/10
729/729 [==============================] - 19s 26ms/step - loss: 0.2664 - f1_score: 0.7906 - val_loss: 0.2613 - val_f1_score: 0.8414
100%|██████████| 148/148 [00:20<00:00,  7.38it/s]
Epoch 1/5
517/517 [==============================] - 15s 28ms/step - loss: 0.6916 - f1_score: 0.6681 - val_loss: 0.6139 - val_f1_score: 0.6845
Epoch 2/5
517/517 [==============================] - 14s 27ms/step - loss: 0.5593 - f1_score: 0.6952 - val_loss: 0.6051 - val_f1_score: 0.6883
Epoch 3/5
517/517 [==============================] - 14s 27ms/step - loss: 0.5234 - f1_score: 0.7071 - val_loss: 0.6027 - val_f1_score: 0.6934
Epoch 4/5
517/517 [==============================] - 14s 27ms/step - loss: 0.5057 - f1_score: 0.7144 - val_loss: 0.6078 - val_f1_score: 0.7006
Epoch 5/5
517/517 [==============================] - 14s 27ms/step - loss: 0.4905 - f1_score: 0.7159 - val_loss: 0.5991 - val_f1_score: 0.7061
Epoch 1/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4769 - f1_score: 0.7248 - val_loss: 0.5918 - val_f1_score: 0.7082
Epoch 2/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4745 - f1_score: 0.7263 - val_loss: 0.5988 - val_f1_score: 0.7130
Epoch 3/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4621 - f1_score: 0.7291 - val_loss: 0.5901 - val_f1_score: 0.7152
Epoch 4/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4577 - f1_score: 0.7298 - val_loss: 0.5877 - val_f1_score: 0.7175
Epoch 5/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4524 - f1_score: 0.7332 - val_loss: 0.5834 - val_f1_score: 0.7199
Epoch 6/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4488 - f1_score: 0.7326 - val_loss: 0.5745 - val_f1_score: 0.7226
Epoch 7/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4482 - f1_score: 0.7324 - val_loss: 0.5851 - val_f1_score: 0.7216
Epoch 8/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4401 - f1_score: 0.7375 - val_loss: 0.5756 - val_f1_score: 0.7230
Epoch 9/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4383 - f1_score: 0.7358 - val_loss: 0.5708 - val_f1_score: 0.7240
Epoch 10/10
517/517 [==============================] - 14s 27ms/step - loss: 0.4342 - f1_score: 0.7385 - val_loss: 0.5688 - val_f1_score: 0.7266
100%|██████████| 76/76 [00:09<00:00,  7.64it/s]
Epoch 1/5
596/596 [==============================] - 17s 27ms/step - loss: 0.6075 - f1_score: 0.6766 - val_loss: 0.4478 - val_f1_score: 0.6209
Epoch 2/5
596/596 [==============================] - 16s 27ms/step - loss: 0.4887 - f1_score: 0.6981 - val_loss: 0.4376 - val_f1_score: 0.6460
Epoch 3/5
596/596 [==============================] - 16s 26ms/step - loss: 0.4625 - f1_score: 0.7063 - val_loss: 0.4315 - val_f1_score: 0.6579
Epoch 4/5
596/596 [==============================] - 15s 26ms/step - loss: 0.4452 - f1_score: 0.7119 - val_loss: 0.4353 - val_f1_score: 0.6595
Epoch 5/5
596/596 [==============================] - 16s 26ms/step - loss: 0.4280 - f1_score: 0.7200 - val_loss: 0.4243 - val_f1_score: 0.6646
Epoch 1/10
596/596 [==============================] - 15s 26ms/step - loss: 0.4239 - f1_score: 0.7228 - val_loss: 0.4214 - val_f1_score: 0.6698
Epoch 2/10
596/596 [==============================] - 16s 26ms/step - loss: 0.4164 - f1_score: 0.7295 - val_loss: 0.4187 - val_f1_score: 0.6689
Epoch 3/10
596/596 [==============================] - 16s 26ms/step - loss: 0.4057 - f1_score: 0.7341 - val_loss: 0.4187 - val_f1_score: 0.6714
Epoch 4/10
596/596 [==============================] - 15s 26ms/step - loss: 0.4035 - f1_score: 0.7343 - val_loss: 0.4198 - val_f1_score: 0.6717
Epoch 5/10
596/596 [==============================] - 16s 26ms/step - loss: 0.3989 - f1_score: 0.7377 - val_loss: 0.4168 - val_f1_score: 0.6722
Epoch 6/10
596/596 [==============================] - 16s 26ms/step - loss: 0.3969 - f1_score: 0.7405 - val_loss: 0.4112 - val_f1_score: 0.6800
Epoch 7/10
596/596 [==============================] - 16s 26ms/step - loss: 0.3937 - f1_score: 0.7394 - val_loss: 0.4148 - val_f1_score: 0.6762
Epoch 8/10
596/596 [==============================] - 16s 26ms/step - loss: 0.3907 - f1_score: 0.7444 - val_loss: 0.4101 - val_f1_score: 0.6809
Epoch 9/10
596/596 [==============================] - 16s 26ms/step - loss: 0.3877 - f1_score: 0.7461 - val_loss: 0.4158 - val_f1_score: 0.6814
Epoch 10/10
596/596 [==============================] - 16s 26ms/step - loss: 0.3849 - f1_score: 0.7468 - val_loss: 0.4151 - val_f1_score: 0.6823
100%|██████████| 75/75 [00:09<00:00,  7.61it/s]
Epoch 1/5
533/533 [==============================] - 15s 26ms/step - loss: 0.2820 - f1_score: 0.6544 - val_loss: 0.2108 - val_f1_score: 0.8165
Epoch 2/5
533/533 [==============================] - 14s 26ms/step - loss: 0.2477 - f1_score: 0.6912 - val_loss: 0.1994 - val_f1_score: 0.8288
Epoch 3/5
533/533 [==============================] - 14s 26ms/step - loss: 0.2424 - f1_score: 0.6925 - val_loss: 0.1975 - val_f1_score: 0.8333
Epoch 4/5
533/533 [==============================] - 14s 25ms/step - loss: 0.2405 - f1_score: 0.6879 - val_loss: 0.1944 - val_f1_score: 0.8212
Epoch 5/5
533/533 [==============================] - 14s 25ms/step - loss: 0.2367 - f1_score: 0.7007 - val_loss: 0.1961 - val_f1_score: 0.8192
Epoch 1/10
533/533 [==============================] - 14s 26ms/step - loss: 0.2328 - f1_score: 0.7021 - val_loss: 0.1932 - val_f1_score: 0.8164
Epoch 2/10
533/533 [==============================] - 14s 26ms/step - loss: 0.2310 - f1_score: 0.7003 - val_loss: 0.1924 - val_f1_score: 0.8113
Epoch 3/10
533/533 [==============================] - 14s 26ms/step - loss: 0.2293 - f1_score: 0.7087 - val_loss: 0.1903 - val_f1_score: 0.8116
Epoch 4/10
533/533 [==============================] - 14s 26ms/step - loss: 0.2304 - f1_score: 0.7109 - val_loss: 0.1903 - val_f1_score: 0.8167
Epoch 5/10
533/533 [==============================] - 14s 26ms/step - loss: 0.2263 - f1_score: 0.7082 - val_loss: 0.1937 - val_f1_score: 0.8131
Epoch 6/10
533/533 [==============================] - 14s 26ms/step - loss: 0.2251 - f1_score: 0.7149 - val_loss: 0.1909 - val_f1_score: 0.8195
Epoch 7/10
533/533 [==============================] - 14s 25ms/step - loss: 0.2225 - f1_score: 0.7257 - val_loss: 0.1885 - val_f1_score: 0.8240
Epoch 8/10
533/533 [==============================] - 14s 26ms/step - loss: 0.2237 - f1_score: 0.7154 - val_loss: 0.1925 - val_f1_score: 0.8181
Epoch 9/10
533/533 [==============================] - 14s 26ms/step - loss: 0.2235 - f1_score: 0.7259 - val_loss: 0.1932 - val_f1_score: 0.8192
Epoch 10/10
533/533 [==============================] - 14s 26ms/step - loss: 0.2266 - f1_score: 0.7251 - val_loss: 0.1924 - val_f1_score: 0.8250
100%|██████████| 356/356 [00:46<00:00,  7.64it/s]
In [ ]:
# Test set for task 2 has many sequences with annotator 0
# these are not used for scoring task 2, hence submit random predictions for these
for sequence_id, sequence in test["sequences"].items():
  if sequence_id in submission: # skip the ones where prediction is done
    continue
  keypoint_sequence = sequence['keypoints']
  submission[sequence_id] = np.random.randint(4, size=len(sequence['keypoints']))

Validate the submission ✅

The submssion should follow these constraints:

  1. It should be a dictionary
  2. It should be have same keys as sample_submission
  3. The lengths of the arrays are same
  4. All values are intergers

You can use the helper function below to check these

In [ ]:
def validate_submission(submission, sample_submission):
    if not isinstance(submission, dict):
      print("Submission should be dict")
      return False

    if not submission.keys() == sample_submission.keys():
      print("Submission keys don't match")
      return False
    
    for key in submission:
      sv = submission[key]
      ssv = sample_submission[key]
      if not len(sv) == len(ssv):
        print(f"Submission lengths of {key} doesn't match")
        return False
    
    for key, sv in submission.items():
      if not all(isinstance(x, (np.int32, np.int64, int)) for x in list(sv)):
        print(f"Submission of {key} is not all integers")
        return False
    
    print("All tests passed")
    return True
In [ ]:
validate_submission(submission, sample_submission)
All tests passed
Out[ ]:
True

Save the prediction as npy 📨

In [ ]:
np.save("submission.npy", submission)

Submit to AIcrowd 🚀

In [ ]:
!aicrowd submission create -c mabe-task-2-annotation-style-transfer -f submission.npy
submission.npy ━━━━━━━━━━━━━━━━━━━━━━ 100.0%61.6/61.6 MB2.3 MB/s0:00:00
                                                                    ╭─────────────────────────╮                                                                    
                                                                    │ Successfully submitted! │                                                                    
                                                                    ╰─────────────────────────╯                                                                    
                                                                          Important links                                                                          
┌──────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│  This submission │ https://www.aicrowd.com/challenges/multi-agent-behavior-representation-modeling-measurement-and-applications/submissions/125579              │
│                  │                                                                                                                                              │
│  All submissions │ https://www.aicrowd.com/challenges/multi-agent-behavior-representation-modeling-measurement-and-applications/submissions?my_submissions=true │
│                  │                                                                                                                                              │
│      Leaderboard │ https://www.aicrowd.com/challenges/multi-agent-behavior-representation-modeling-measurement-and-applications/leaderboards                    │
│                  │                                                                                                                                              │
│ Discussion forum │ https://discourse.aicrowd.com/c/multi-agent-behavior-representation-modeling-measurement-and-applications                                    │
│                  │                                                                                                                                              │
│   Challenge page │ https://www.aicrowd.com/challenges/multi-agent-behavior-representation-modeling-measurement-and-applications                                 │
└──────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
{'submission_id': 125579, 'created_at': '2021-03-08T16:44:25.075Z'}

Comments

You must login before you can post a comment.

Execute