📢 Important Message for the Participants regarding Task-1 📢

Please note that the Task-1 Public Test set (v0.2 release) contains some overlapping samples from the Task-2 Training set (v0.2 release).

If the systems are using an integrated training set from all the tasks, the participants will need to be conscious that their system might be “memorizing” the training data and getting artificially high results on the Task-1 Public dataset.

While we allow the training data from all tasks to be used for training any task, we want to emphasize that re-using or memorizing the labels of training data in another task is not advisable. It should be noted that there will not be any such overlap in the Private Test set (which will be used to determine the final winners and the rankings on the leaderboard for that task).

Since the goal of this task is to build a general query-product ranking/classification models, it will be beneficial for the participants to use only the training sets given for each task separately when building their systems or evaluate the performance of their systems without using any overlapping data.

With this in mind, starting from today onwards, we will also be showing another column (same performance metric as before) on the leaderboard excluding any overlapping test data. This should help the participants understand the generalization ability of their systems.

The score computed on the complete Public Test set will be available under the NDCG (full test set) column, and the score computed on the subset of the Public Test set (with the overlapping test data removed), will be available under the NDCG (clean) column. The leaderboards will be sorted using the NDCG (clean).

Leaderboard snippet (ranks at the time of posting it)

image1260×1346 113 KB

View post 4

🚨🚨🚨 Dataset update - `v0.2` released!

@yrquni : Yes, that is correct. We do so, to avoid confusion across different versions of the datasets floating around. And the versioning system we use at the moment is for the combined datasets of all the tasks.

View post 0

Downloading data via AIcrowd CLI

Hi @HAGE2000 ,

Sorry for the confusion. The ordering of the columns technically doesn’t matter. In future releases of the data, we will make sure to ensure the consistency between the released data and the description in the Overview.

Best,
Mohanty

View post 1

📝 Announcing Community Contribution Prizes 🚁🥽

Community Contribution Prize1920×1011 88 KB

We are excited to announce the following Community Contribution Prizes :

• 2 x Oculus Quest 2
• 2 x DJI Mavic Mini FMK

How to Participate?

• The Community Contribution Prizes will be awarded based on the discretion of the organizers, and will be awarded to the teams/individuals, that among other actions, are most active in the community.
• The prizes typically go to individuals or teams that are extremely active in the community, share resources - or even answer questions - that benefit the whole community greatly!
• So, share your posts widely to spread the word and get active in the community!
• You can make multiple submissions, but you are only eligible for the Community Contribution Prize once.
• In case of resources that are created, your work needs to be published under a license of your choice, and on a platform that allows other participants to access and use it.

Notebooks, Blog Posts, Tutorials, Screencasts, Youtube Videos, or even your active responses on the challenge forums - everything is eligible for the Community Contribution Prizes.

We are looking forward to see everything you create!

All the best!

View post 1

The 'product_id' field is not unique in the product table

Each product_id can have different descriptions etc depending on the product_locale. So please use the product_locale to match with the query_locale when trying to map the product_catalogue to the training set or the test set.

More Details

Best,
Mohanty

View post 0

😱 Why is there a `product_id` column in the Task 1 test set ? 😱

The v0.1 of the Task-1 test set only had the following columns : query_id, query, query_locale.

In the v0.2 release, we introduced a separate column, the product_id, and there has been some confusion as to why !

We understand, and Let us explain !

The reason the v2.0 test set of Task 1 includes pairs of query_id and product_id, is because we only have the esci_labels for those specific query_id and product_id pairs, and we can only consider those pairs for computing the nDCG score. The other query_id and product_id pairs that do not appear in the test set will not be considered for computing the nDCG score, since we do not have the corresponding esci_label.

This helps participants obtain a more meaningful nDCG score.

To help clarify this better, lets imagine a situation where we only want the top three ranked products for the query included in the test set:

Version 0.1 dataset

Input:

query_1, "some query", us

Output:

query_1, product_11

query_1, product_7

query_1, product_9

If we do not have the esci_label for any of those three pairs (say for query_1, product_7 pair), then we will omit those pairs, and compute the nDCG using only the query_1, product_11 and query_1, product_9 pairs.

Version 0.2 dataset:

Input:

query_1, product_1, locale_us

query_1, product_3, locale_us

query_1, product_10, locale_us

Output:

query_1, product_10

query_1, product_1,

query_1, product_3

query_1, product_39

In the output, we have all the pairs annotated except the query_1, product_39 pair.

But if the participants already had the information of which query-product pairs had esci_labels available, then they should not have included the query_1, product_39 pair to begin with, as it is not included the test set.

So in this case, we will compute the nDCG score considering the 3 ranked products that were included in the test set, and not the query_1, product_39 pair which was not included in the test set.

In other words, the question to answer is, how best to “order” the test set of Task-1, to optimize your nDCG score.

Hope that clarifies the confusion.

If you have any more queries, please do not hesitate to reach out to us.

Best,

Mohanty

View post 2

🚀 Datasets Released & Submissions Open 🚀

This should be addressed in the v2.0 release of the dataset.

View post 1

🚨🚨🚨 Dataset update - `v0.2` released!

Hello everyone!

We updated the dataset to address the duplicate query_id issue and released v0.2 version of the dataset.

Changes in the new version

• Additional product_id column in task 1 test set.
  • More information about the product_id column is available at : 😱 Why is there a `product_id` column in the Task 1 test set ? 😱
• All the CSV files except the sample submission are zipped. But you should still be able to load them in pandas. For example,

  import pandas as pd
  df = pd.read_csv("path/to/test_public-v0.2.csv.zip")
  

More details about the dataset is available at the updated post here : 🚀 Datasets Released & Submissions Open 🚀

Best of Luck!

View post 3

🚀 Datasets Released & Submissions Open 🚀

Dataset Released

NOTE: This post has been updated to reflect the changes to the dataset. Please use v0.2 version of the dataset starting 1st April, 2020.

The Shopping Queries Dataset for the Amazon KDD Cup 2022 - ESCI Challenge for Improving Product Search has been released.

You can access the datasets for each of the Tracks on the Resources Page.

The datasets contain the following files :

.
├── task_1_query-product_ranking
│   ├── product_catalogue-v0.2.csv.zip 
│   ├── sample_submission-v0.2.csv
│   ├── test_public-v0.2.csv.zip
│   └── train-v0.2.csv.zip
├── task_2_multiclass_product_classification
│   ├── product_catalogue-v0.2.csv.zip
│   ├── sample_submission-v0.2.csv
│   ├── test_public-v0.2.csv.zip
│   └── train-v0.2.csv.zip
└── task_3_product_substitute_identification
    ├── product_catalogue-v0.2.csv.zip
    ├── sample_submission-v0.2.csv
    ├── test_public-v0.2.csv.zip
    └── train-v0.2.csv.zip

The product_catalogue-v0.2.csv.zip for all the tasks has the following columns : product_id, product_title, product_description, product_bullet_point, product_brand, product_color_name, product_locale

Task 1

• train-v0.2.csv.zip contains the following columns : query_id, query, query_locale, product_id, esci_label
• test_public-v0.2.csv.zip contains the following columns : query_id, query, query_locale, product_id
• sample_submission-v0.2.csv.zip contains the following columns : query_id, product_id

Task 2

• train-v0.2.csv.zip contains the following columns : example_id, query, product_id, query_locale, esci_label
• test_public-v0.2.csv.zip contains the following columns : example_id, query, product_id, query_locale
• sample_submission-v0.2.csv.zip contains the following columns : example_id, esci_label

Task 3

• train-v0.2.csv.zip contains the following columns : example_id, query, product_id, query_locale, substitute_label
• test_public-v0.2.csv.zip contains the following columns : example_id, query, product_id, query_locale
• sample_submission-v0.2.csv.zip contains the following columns : example_id, substitute_label

Download via CLI [more commands]

aicrowd datasets download -c esci-challenge-for-improving-product-search

# if you don't have AIcrowd CLI installed
pip install -U aicrowd-cli

Submissions

You can make the submissions by clicking on the Create Submission button on the challenge page. Please do remember to select the correct Task from the drop down before submitting.

The Create Submission button is only accessible after you accept the challenge rules by clicking on the Participate button.

We very much recommend making a first submission using the included sample_submission files for each of the tracks.

Best of Luck !

View post 5

👋 Welcome to Amazon KDD Cup '22!

We are in the final stages of data prep and validation. The current plans are to release the datasets on 28th March, Monday. Stay tuned

View post 1

Clarification and Updates to Challenge Rules

The current challenge rules outline that the Round start and end dates are :

• Round 1: Feb 4th, 2022, 12:00:00 pm UTC – February 28th, 2022, 12:00:00 pm UTC
• Round 2: March 1st, 2022, 12:00:00 pm UTC – April 5th, 2022 12:00:00 pm UTC

and the cash prizes for the Top-3 teams are as follows :

• 1st Prize: : USD 6,500
• 2nd Prize: : USD 4,500
• 3rd Prize: : USD 3,000

While, the information on the Challenge Overview page states that the Round start and end dates are :

• Round 1: Feb 4th, 2022, 12:00:00 pm UTC – February 28th, 2022, 12:00:00 pm UTC
• Round 2: March 3rd, 2022, 12:00:00 pm UTC – April 7th, 2022 12:00:00 pm UTC

and the cash prizes for the Top-3 teams are as follows :

• 1st Prize: : USD 6,000
• 2nd Prize: : USD 4,500
• 3rd Prize: : USD 3,000

We confirm that the information on the Challenge Overview page reflect the most updated information about both the Round start-end times and the prizes for the top teams.

For consistency, we are updating the Challenge Rules to reflect the same, and this will require all the participants to re-accept the rules (by Clicking on the Participate button on the Challenge Overview page) before they can make any further submissions.

Apologies for any confusion, and best of luck with the competition.

Best,
Mohanty

View post 4

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

@moto : No, unfortunately you still cannot check in your pre-trained models. The rules around pre-trained models stay the same as it was in Round 1.

View post 1

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

@moto : You will still have to train your models from scratch, then use your trained models for making your purchasing decisions. We then take the labels that you purchased, and use it along with the training set in our training pipeline to compute the final scores.

The training pipeline we introduce is just an elaborate evaluation function to “assess the quality of the purchased labels”.

Wrt the prediction_phase interfaces, you will still have to submit the same, as the evaluators do a series of integration tests to ensure that the prediction_phase interface works as expected - while we do not use this function in the current evaluation pipeline, it will allow us to do a more elaborate analysis of the submissions at a later point.

View post 0

Outside code cell submission

We will be posting an end to end notebook submission example soon with a baseline included. Hopefully that should be helpful.
@shivam is the main POC for the same.

View post 1

Video tutorial on using this repo

Thanks Gaurav. This is super helpful.

We will be releasing some changes to the dataset and the problem statement later this week.
And we will release the official screencast videos after.

Best,
Mohanty

View post 2

Potential loop hole in purchasing phase

Thanks @gaurav_singhal for bringing this up.

The classes available to you for local development are different from the implementation of the classes used in the evaluation setup.

In the evaluation setup, a drop-in replacement of the ProtectedDataset class is used, which interfaces with a remote service that actually knows the true labels labels. So this aspect of protecting the true labels during the purchasing phase has been taken care of in the evaluators.

View post 1

Sklearn Accuracy Score is identical with exact match ratio

The original intent was to use a strict measure of accuracy, meaning a prediction is counted as correct when it exactly matches the ground truth labels for all the classes. However the sklearn accuracy_score actually computes the exact match ratio, as it computes the accuracy for all labels separately and then takes the mean across each.
We will discuss internally, and re-assess if we update the accuracy metric, or leave it as is for this round.

View post 0

🙋🏻‍♂️ Frequently Asked Questions (FAQs)

Hi everyone,

We hope you are having a great time taking part in the ZEW Data Purchasing Challenge 2022.

1. : Can I submit pre-trained models in my submission ?

No, in this challenge, you are not allowed to include any custom pre-trained models in your submission.
Your submission can however use any of the pre-trained models already included in torchvision.models (which will be available in the local cache).

Before your submission is evaluated, all files greater than a threshold size will be scrubbed from your submission repository.
If your submission repository contains more than a threshold number of files, your submission will fail automatically.

During your submission, your submitted code will not have any network access,
so trying to access any model weights hosted externally will also lead to a failed submission.

2 : When will the cash prizes be awarded ?

The cash prizes will be awarded based on the final leaderboard from the Round 2 of the competition - after a manual verification of the said submission, to ensure that the submission conforms to the Rules of the competition.

3 : When can I submit an entry for the Community Contribution Prize ?

You can submit an entry for the Community Contribution Prize throughout the competition, up until a week before the closing date of Round 2.

4 What compute resources will I have access to during the evaluation ?

Your code submissions will have access to 4 CPUS, 16 GB RAM and 1 NVIDIA T4 GPU during the evaluation. The GPU is optional to use, and you can configure your need for the GPU in the aicrowd.json at the root of your submission repository.

View post 1

Allowance of Pre-trained Model

Clever ! But we are scrubbing all files in the submitted repo which are larger than a threshold size. And if you include a base64 encoded model which has the same filesize as any arbitrary python code file, that would be a great achievement
You could however split your base64 encoded model into many many little files, and even call them as .py files , but then your submission will be disqualified during the manual inspection of the code before any prizes are awarded.

We will however clarify in the challenge rules, that any pre-trained models (other than the ones included in torchvision) are not allowed. We will try our best to technically filter out submissions which disrespect the rule, and we will definitely disqualify any submissions which disrespect this rule during the manual code inspection, if by chance, some clever submissions fall through the cracks.

We would anyway appreciate, if participants respect the spirit of the competition

Best,
Mohanty

View post 0

Allowance of Pre-trained Model

@wufanyou : Only pre-trained models included in pytorch(via torchvision are allowed at the moment.
If you include any model weights in your submission, they will be scrubbed before the evaluation begins.

Best,
Mohanty

View post 0