Activity
Ratings Progression
Challenge Categories
Challenges Entered
Behavioral Representation Learning from Animal Poses.
Latest submissions
Latest submissions
A benchmark for image-based food recognition
Latest submissions
What data should you label to get the most value for your money?
Latest submissions
ASCII-rendered single-player dungeon crawl game
Latest submissions
Airborne Object Tracking Challenge
Latest submissions
Multi-Agent RL for Trains
Latest submissions
Machine Learning for detection of early onset of Alzheimers
Latest submissions
Sample Efficient Reinforcement Learning in Minecraft
Latest submissions
Measure sample efficiency and generalization in reinforcement learning using procedurally generated environments
Latest submissions
Robustness and teamwork in a massively multiagent environment
Latest submissions
3D Seismic Image Interpretation by Machine Learning
Latest submissions
See Allgraded | 82855 | ||
graded | 82679 | ||
graded | 82664 |
Latest submissions
Play in a realistic insurance market, compete for profit!
Latest submissions
Multi-Agent Reinforcement Learning on Trains
Latest submissions
A dataset and open-ended challenge for music recommendation research
Latest submissions
A benchmark for image-based food recognition
Latest submissions
See Allgraded | 109590 | ||
failed | 109586 | ||
failed | 109578 |
Sample-efficient reinforcement learning in Minecraft
Latest submissions
Latest submissions
Predicting smell of molecular compounds
Latest submissions
See Allgraded | 101429 | ||
graded | 101420 | ||
graded | 81550 |
Classify images of snake species from around the world
Latest submissions
See Allfailed | 5795 | ||
graded | 392 | ||
failed | 391 |
Find all the aircraft!
Latest submissions
5 Problems 21 Days. Can you solve it all?
Latest submissions
5 Puzzles 21 Days. Can you solve it all?
Latest submissions
Robots that learn to interact with the environment autonomously
Latest submissions
5 PROBLEMS 3 WEEKS. CAN YOU SOLVE THEM ALL?
Latest submissions
Grouping/Sorting players into their respective teams
Latest submissions
Reinforcement Learning on Musculoskeletal Models
Latest submissions
See Allgraded | 18446 | ||
graded | 10385 | ||
graded | 10099 |
Disentanglement: from simulation to real-world
Latest submissions
See Allgraded | 21050 | ||
graded | 13148 | ||
graded | 13117 |
Sample-efficient reinforcement learning in Minecraft
Latest submissions
See Allgraded | 18575 | ||
graded | 15152 | ||
graded | 11232 |
Multi Agent Reinforcement Learning on Trains.
Latest submissions
See Allfailed | 18455 | ||
failed | 18370 | ||
failed | 18105 |
Visual SLAM in challenging environments
Latest submissions
Latest submissions
ACL-BioNLP Shared Task
Latest submissions
5 Problems 15 Days. Can you solve it all?
Latest submissions
Project 2: Road extraction from satellite images
Latest submissions
Robots that learn to interact with the environment autonomously
Latest submissions
See Allgraded | 11208 | ||
failed | 11198 | ||
failed | 11193 |
Immitation Learning for Autonomous Driving
Latest submissions
A new benchmark for Artificial Intelligence (AI) research in Reinforcement Learning
Latest submissions
See Allgraded | 7057 | ||
graded | 664 | ||
graded | 663 |
ACL-BioNLP Shared Task
Latest submissions
Predict if users will skip or listen to the music they're streamed
Latest submissions
Spot the Boson
Latest submissions
5 PROBLEMS 3 WEEKS. CAN YOU SOLVE THEM ALL?
Latest submissions
Predict viewer reactions from a large-scale video dataset!
Latest submissions
Detect Multi-Animal Behaviors from a large, hand-labeled dataset.
Latest submissions
Multi-Agent Reinforcement Learning on Trains
Latest submissions
Participant | Rating |
---|---|
![]() |
0 |
![]() |
0 |
![]() |
0 |
![]() |
0 |
![]() |
|
![]() |
|
![]() |
0 |
![]() |
0 |
![]() |
0 |
![]() |
|
![]() |
132 |
Participant | Rating |
---|
ESCI Challenge for Improving Product Search

π¨π¨π¨ Dataset update - `v0.2` released!
About 1 month ago@yrquni : Yes, that is correct. We do so, to avoid confusion across different versions of the datasets floating around. And the versioning system we use at the moment is for the combined datasets of all the tasks.

Downloading data via AIcrowd CLI
About 1 month agoHi @HAGE2000 ,
Sorry for the confusion. The ordering of the columns technically doesnβt matter. In future releases of the data, we will make sure to ensure the consistency between the released data and the description in the Overview.
Best,
Mohanty

π Announcing Community Contribution Prizes ππ₯½
About 2 months agoWe are excited to announce the following Community Contribution Prizes :
- 2 x Oculus Quest 2
- 2 x DJI Mavic Mini FMK
How to Participate?
- The Community Contribution Prizes will be awarded based on the discretion of the organizers, and will be awarded to the teams/individuals, that among other actions, are most active in the community.
- The prizes typically go to individuals or teams that are extremely active in the community, share resources - or even answer questions - that benefit the whole community greatly!
- So, share your posts widely to spread the word and get active in the community!
- You can make multiple submissions, but you are only eligible for the Community Contribution Prize once.
- In case of resources that are created, your work needs to be published under a license of your choice, and on a platform that allows other participants to access and use it.
Notebooks, Blog Posts, Tutorials, Screencasts, Youtube Videos, or even your active responses on the challenge forums - everything is eligible for the Community Contribution Prizes.
We are looking forward to see everything you create!
All the best!

The 'product_id' field is not unique in the product table
About 2 months agoEach product_id
can have different descriptions etc depending on the product_locale
. So please use the product_locale
to match with the query_locale
when trying to map the product_catalogue
to the training set or the test set.
Best,
Mohanty

π± Why is there a `product_id` column in the Task 1 test set ? π±
About 2 months agoThe v0.1
of the Task-1 test set only had the following columns : query_id
, query
, query_locale
.
In the v0.2
release, we introduced a separate column, the product_id
, and there has been some confusion as to why !
We understand, and Let us explain !
The reason the v2.0
test set of Task 1 includes pairs of query_id
and product_id
, is because we only have the esci_label
s for those specific query_id
and product_id
pairs, and we can only consider those pairs for computing the nDCG score. The other query_id
and product_id
pairs that do not appear in the test set will not be considered for computing the nDCG score, since we do not have the corresponding esci_label
.
This helps participants obtain a more meaningful nDCG score.
To help clarify this better, lets imagine a situation where we only want the top three ranked products for the query included in the test set:
Version 0.1 dataset
Input:
query_1, "some query", us
Output:
query_1, product_11
query_1, product_7
query_1, product_9
If we do not have the esci_label
for any of those three pairs (say for query_1, product_7
pair), then we will omit those pairs, and compute the nDCG using only the query_1, product_11
and query_1, product_9
pairs.
Version 0.2 dataset:
Input:
query_1, product_1, locale_us
query_1, product_3, locale_us
query_1, product_10, locale_us
Output:
query_1, product_10
query_1, product_1,
query_1, product_3
query_1, product_39
In the output, we have all the pairs annotated except the query_1, product_39
pair.
But if the participants already had the information of which query-product
pairs had esci_label
s available, then they should not have included the query_1, product_39
pair to begin with, as it is not included the test set.
So in this case, we will compute the nDCG score considering the 3 ranked products that were included in the test set, and not the query_1, product_39
pair which was not included in the test set.
In other words, the question to answer is, how best to βorderβ the test set of Task-1, to optimize your nDCG score.
Hope that clarifies the confusion.
If you have any more queries, please do not hesitate to reach out to us.
Best,
Mohanty

π Datasets Released & Submissions Open π
About 2 months agoThis should be addressed in the v2.0
release of the dataset.

π¨π¨π¨ Dataset update - `v0.2` released!
About 2 months agoHello everyone!
We updated the dataset to address the duplicate query_id
issue and released v0.2
version of the dataset.
Changes in the new version
- Additional
product_id
column in task 1 test set.- More information about the
product_id
column is available at : π± Why is there a `product_id` column in the Task 1 test set ? π±
- More information about the
- All the CSV files except the sample submission are zipped. But you should still be able to load them in pandas. For example,
import pandas as pd df = pd.read_csv("path/to/test_public-v0.2.csv.zip")
More details about the dataset is available at the updated post here : π Datasets Released & Submissions Open π
Best of Luck!

π Datasets Released & Submissions Open π
About 2 months agoDataset Released
NOTE: This post has been updated to reflect the changes to the dataset. Please use
v0.2
version of the dataset starting 1st April, 2020.
The Shopping Queries Dataset for the Amazon KDD Cup 2022 - ESCI Challenge for Improving Product Search has been released.
You can access the datasets for each of the Tracks on the Resources Page.
The datasets contain the following files :
.
βββ task_1_query-product_ranking
β βββ product_catalogue-v0.2.csv.zip
β βββ sample_submission-v0.2.csv
β βββ test_public-v0.2.csv.zip
β βββ train-v0.2.csv.zip
βββ task_2_multiclass_product_classification
β βββ product_catalogue-v0.2.csv.zip
β βββ sample_submission-v0.2.csv
β βββ test_public-v0.2.csv.zip
β βββ train-v0.2.csv.zip
βββ task_3_product_substitute_identification
βββ product_catalogue-v0.2.csv.zip
βββ sample_submission-v0.2.csv
βββ test_public-v0.2.csv.zip
βββ train-v0.2.csv.zip
The product_catalogue-v0.2.csv.zip
for all the tasks has the following columns : product_id
, product_title
, product_description
, product_bullet_point
, product_brand
, product_color_name
, product_locale
Task 1
-
train-v0.2.csv.zip
contains the following columns :query_id
,query
,query_locale
,product_id
,esci_label
-
test_public-v0.2.csv.zip
contains the following columns :query_id
,query
,query_locale
,product_id
-
sample_submission-v0.2.csv.zip
contains the following columns :query_id
,product_id
Task 2
-
train-v0.2.csv.zip
contains the following columns :example_id
,query
,product_id
,query_locale
,esci_label
-
test_public-v0.2.csv.zip
contains the following columns :example_id
,query
,product_id
,query_locale
-
sample_submission-v0.2.csv.zip
contains the following columns :example_id
,esci_label
Task 3
-
train-v0.2.csv.zip
contains the following columns :example_id
,query
,product_id
,query_locale
,substitute_label
-
test_public-v0.2.csv.zip
contains the following columns :example_id
,query
,product_id
,query_locale
-
sample_submission-v0.2.csv.zip
contains the following columns :example_id
,substitute_label
Download via CLI [more commands]
aicrowd datasets download -c esci-challenge-for-improving-product-search
# if you don't have AIcrowd CLI installed
pip install -U aicrowd-cli
Submissions
You can make the submissions by clicking on the Create Submission button on the challenge page. Please do remember to select the correct Task from the drop down before submitting.
The Create Submission button is only accessible after you accept the challenge rules by clicking on the Participate
button.
We very much recommend making a first submission using the included sample_submission
files for each of the tracks.
Best of Luck !

π Welcome to Amazon KDD Cup '22!
2 months agoWe are in the final stages of data prep and validation. The current plans are to release the datasets on 28th March, Monday. Stay tuned
Data Purchasing Challenge 2022

Clarification and Updates to Challenge Rules
About 2 months agoThe current challenge rules outline that the Round start and end dates are :
- Round 1: Feb 4th, 2022, 12:00:00 pm UTC β February 28th, 2022, 12:00:00 pm UTC
- Round 2: March 1st, 2022, 12:00:00 pm UTC β April 5th, 2022 12:00:00 pm UTC
and the cash prizes for the Top-3 teams are as follows :
- 1st Prize: : USD 6,500
- 2nd Prize: : USD 4,500
- 3rd Prize: : USD 3,000
While, the information on the Challenge Overview page states that the Round start and end dates are :
- Round 1: Feb 4th, 2022, 12:00:00 pm UTC β February 28th, 2022, 12:00:00 pm UTC
- Round 2: March 3rd, 2022, 12:00:00 pm UTC β April 7th, 2022 12:00:00 pm UTC
and the cash prizes for the Top-3 teams are as follows :
- 1st Prize: : USD 6,000
- 2nd Prize: : USD 4,500
- 3rd Prize: : USD 3,000
We confirm that the information on the Challenge Overview page reflect the most updated information about both the Round start-end times and the prizes for the top teams.
For consistency, we are updating the Challenge Rules to reflect the same, and this will require all the participants to re-accept the rules (by Clicking on the Participate button on the Challenge Overview page) before they can make any further submissions.
Apologies for any confusion, and best of luck with the competition.
Best,
Mohanty

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!
3 months ago@moto : No, unfortunately you still cannot check in your pre-trained models. The rules around pre-trained models stay the same as it was in Round 1.

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!
3 months ago@moto : You will still have to train your models from scratch, then use your trained models for making your purchasing decisions. We then take the labels that you purchased, and use it along with the training set in our training pipeline to compute the final scores.
The training pipeline we introduce is just an elaborate evaluation function to βassess the quality of the purchased labelsβ.
Wrt the prediction_phase interfaces, you will still have to submit the same, as the evaluators do a series of integration tests to ensure that the prediction_phase interface works as expected - while we do not use this function in the current evaluation pipeline, it will allow us to do a more elaborate analysis of the submissions at a later point.

Outside code cell submission
3 months agoWe will be posting an end to end notebook submission example soon with a baseline included. Hopefully that should be helpful.
@shivam is the main POC for the same.

Video tutorial on using this repo
3 months agoThanks Gaurav. This is super helpful.
We will be releasing some changes to the dataset and the problem statement later this week.
And we will release the official screencast videos after.
Best,
Mohanty

Potential loop hole in purchasing phase
3 months agoThanks @gaurav_singhal for bringing this up.
The classes available to you for local development are different from the implementation of the classes used in the evaluation setup.
In the evaluation setup, a drop-in replacement of the ProtectedDataset
class is used, which interfaces with a remote service that actually knows the true labels labels. So this aspect of protecting the true labels during the purchasing phase has been taken care of in the evaluators.

Sklearn Accuracy Score is identical with exact match ratio
3 months agoThe original intent was to use a strict measure of accuracy, meaning a prediction is counted as correct when it exactly matches the ground truth labels for all the classes. However the sklearn accuracy_score actually computes the exact match ratio, as it computes the accuracy for all labels separately and then takes the mean across each.
We will discuss internally, and re-assess if we update the accuracy metric, or leave it as is for this round.

ππ»ββοΈ Frequently Asked Questions (FAQs)
4 months agoHi everyone,
We hope you are having a great time taking part in the ZEW Data Purchasing Challenge 2022.
1.
: Can I submit pre-trained models in my submission ?
No, in this challenge, you are not allowed to include any custom pre-trained models in your submission.
Your submission can however use any of the pre-trained models already included in torchvision.models (which will be available in the local cache).
Before your submission is evaluated, all files greater than a threshold size will be scrubbed from your submission repository.
If your submission repository contains more than a threshold number of files, your submission will fail automatically.
During your submission, your submitted code will not have any network access,
so trying to access any model weights hosted externally will also lead to a failed submission.
2
: When will the cash prizes be awarded ?
The cash prizes will be awarded based on the final leaderboard from the Round 2 of the competition - after a manual verification of the said submission, to ensure that the submission conforms to the Rules of the competition.
3
: When can I submit an entry for the Community Contribution Prize ?
You can submit an entry for the Community Contribution Prize throughout the competition, up until a week before the closing date of Round 2.
4
What compute resources will I have access to during the evaluation ?
Your code submissions will have access to 4 CPUS
, 16 GB RAM
and 1 NVIDIA T4 GPU
during the evaluation. The GPU is optional to use, and you can configure your need for the GPU in the aicrowd.json
at the root of your submission repository.

Allowance of Pre-trained Model
4 months agoClever ! But we are scrubbing all files in the submitted repo which are larger than a threshold size. And if you include a base64 encoded model which has the same filesize as any arbitrary python code file, that would be a great achievement
You could however split your base64 encoded model into many many little files, and even call them as .py files , but then your submission will be disqualified during the manual inspection of the code before any prizes are awarded.
We will however clarify in the challenge rules, that any pre-trained models (other than the ones included in torchvision) are not allowed. We will try our best to technically filter out submissions which disrespect the rule, and we will definitely disqualify any submissions which disrespect this rule during the manual code inspection, if by chance, some clever submissions fall through the cracks.
We would anyway appreciate, if participants respect the spirit of the competition
Best,
Mohanty

Allowance of Pre-trained Model
4 months ago@wufanyou : Only pre-trained models included in pytorch(via torchvision are allowed at the moment.
If you include any model weights in your submission, they will be scrubbed before the evaluation begins.
Best,
Mohanty
π’ Important Message for the Participants regarding Task-1 π’
18 days agoPlease note that the Task-1 Public Test set (
v0.2
release) contains some overlapping samples from the Task-2 Training set (v0.2
release).If the systems are using an integrated training set from all the tasks, the participants will need to be conscious that their system might be βmemorizingβ the training data and getting artificially high results on the Task-1 Public dataset.
While we allow the training data from all tasks to be used for training any task, we want to emphasize that re-using or memorizing the labels of training data in another task is not advisable. It should be noted that there will not be any such overlap in the Private Test set (which will be used to determine the final winners and the rankings on the leaderboard for that task).
Since the goal of this task is to build a general query-product ranking/classification models, it will be beneficial for the participants to use only the training sets given for each task separately when building their systems or evaluate the performance of their systems without using any overlapping data.
With this in mind, starting from today onwards, we will also be showing another column (same performance metric as before) on the leaderboard excluding any overlapping test data. This should help the participants understand the generalization ability of their systems.
The score computed on the complete Public Test set will be available under the
NDCG (full test set)
column, and the score computed on the subset of the Public Test set (with the overlapping test data removed), will be available under theNDCG (clean)
column. The leaderboards will be sorted using theNDCG (clean)
.