Loading
11 Follower
0 Following
mohanty
Sharada Mohanty

Organization

AIcrowd

Location

Geneva, CH

Badges

4
3
3

Connect

Activity

May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Mon
Wed
Fri

Ratings Progression

Loading...

Challenge Categories

Loading...

Challenges Entered

Latest submissions

See All
graded 178805
graded 178804
graded 178803

Behavioral Representation Learning from Animal Poses.

Latest submissions

No submissions made in this challenge.

A benchmark for image-based food recognition

Latest submissions

No submissions made in this challenge.

What data should you label to get the most value for your money?

Latest submissions

No submissions made in this challenge.

ASCII-rendered single-player dungeon crawl game

Latest submissions

No submissions made in this challenge.

Airborne Object Tracking Challenge

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

Machine Learning for detection of early onset of Alzheimers

Latest submissions

No submissions made in this challenge.

Sample Efficient Reinforcement Learning in Minecraft

Latest submissions

No submissions made in this challenge.

Measure sample efficiency and generalization in reinforcement learning using procedurally generated environments

Latest submissions

No submissions made in this challenge.

Robustness and teamwork in a massively multiagent environment

Latest submissions

No submissions made in this challenge.

3D Seismic Image Interpretation by Machine Learning

Latest submissions

See All
graded 82855
graded 82679
graded 82664

Latest submissions

No submissions made in this challenge.

Play in a realistic insurance market, compete for profit!

Latest submissions

No submissions made in this challenge.

Multi-Agent Reinforcement Learning on Trains

Latest submissions

No submissions made in this challenge.

A dataset and open-ended challenge for music recommendation research

Latest submissions

No submissions made in this challenge.

A benchmark for image-based food recognition

Latest submissions

See All
graded 109590
failed 109586
failed 109578

Sample-efficient reinforcement learning in Minecraft

Latest submissions

No submissions made in this challenge.

Predicting smell of molecular compounds

Latest submissions

See All
graded 101429
graded 101420
graded 81550

Latest submissions

See All
failed 5795
graded 392
failed 391

Latest submissions

No submissions made in this challenge.

5 Problems 21 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

5 Puzzles 21 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

Robots that learn to interact with the environment autonomously

Latest submissions

No submissions made in this challenge.

5 PROBLEMS 3 WEEKS. CAN YOU SOLVE THEM ALL?

Latest submissions

No submissions made in this challenge.

Grouping/Sorting players into their respective teams

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
graded 18446
graded 10385
graded 10099

Latest submissions

See All
graded 21050
graded 13148
graded 13117

Sample-efficient reinforcement learning in Minecraft

Latest submissions

See All
graded 18575
graded 15152
graded 11232

Multi Agent Reinforcement Learning on Trains.

Latest submissions

See All
failed 18455
failed 18370
failed 18105

Recognise Handwritten Digits

Latest submissions

See All
failed 60098

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

5 Problems 15 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

Project 2: Road extraction from satellite images

Latest submissions

No submissions made in this challenge.

Robots that learn to interact with the environment autonomously

Latest submissions

See All
graded 11208
failed 11198
failed 11193

Latest submissions

No submissions made in this challenge.

A new benchmark for Artificial Intelligence (AI) research in Reinforcement Learning

Latest submissions

See All
graded 7057
graded 664
graded 663

Latest submissions

See All
graded 1137
graded 1135
graded 1134

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
graded 9169
graded 9164

Latest submissions

See All
graded 9168
graded 9163

Latest submissions

See All
graded 9167
failed 9165
failed 9162

Predict if users will skip or listen to the music they're streamed

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

5 PROBLEMS 3 WEEKS. CAN YOU SOLVE THEM ALL?

Latest submissions

No submissions made in this challenge.

Predict viewer reactions from a large-scale video dataset!

Latest submissions

No submissions made in this challenge.

Detect Multi-Animal Behaviors from a large, hand-labeled dataset.

Latest submissions

No submissions made in this challenge.

Multi-Agent Reinforcement Learning on Trains

Latest submissions

No submissions made in this challenge.
Participant Rating
krishna_kaushik 0
amitabh 0
singstad90 0
omkarkur 0
vrv
Shubhamaicrowd
LazyPanda 0
SHARDA 0
semih_catal 0
shivam
20161302_animesh 132
Participant Rating

ESCI Challenge for Improving Product Search

πŸ“’ Important Message for the Participants regarding Task-1 πŸ“’

18 days ago

Please note that the Task-1 Public Test set (v0.2 release) contains some overlapping samples from the Task-2 Training set (v0.2 release).

If the systems are using an integrated training set from all the tasks, the participants will need to be conscious that their system might be β€œmemorizing” the training data and getting artificially high results on the Task-1 Public dataset.

While we allow the training data from all tasks to be used for training any task, we want to emphasize that re-using or memorizing the labels of training data in another task is not advisable. It should be noted that there will not be any such overlap in the Private Test set (which will be used to determine the final winners and the rankings on the leaderboard for that task).

Since the goal of this task is to build a general query-product ranking/classification models, it will be beneficial for the participants to use only the training sets given for each task separately when building their systems or evaluate the performance of their systems without using any overlapping data.

With this in mind, starting from today onwards, we will also be showing another column (same performance metric as before) on the leaderboard excluding any overlapping test data. This should help the participants understand the generalization ability of their systems.

The score computed on the complete Public Test set will be available under the NDCG (full test set) column, and the score computed on the subset of the Public Test set (with the overlapping test data removed), will be available under the NDCG (clean) column. The leaderboards will be sorted using the NDCG (clean).


:point_right: Leaderboard snippet (ranks at the time of posting it)

🚨🚨🚨 Dataset update - `v0.2` released!

About 1 month ago

@yrquni : Yes, that is correct. We do so, to avoid confusion across different versions of the datasets floating around. And the versioning system we use at the moment is for the combined datasets of all the tasks.

Downloading data via AIcrowd CLI

About 1 month ago

Hi @HAGE2000 ,

Sorry for the confusion. The ordering of the columns technically doesn’t matter. In future releases of the data, we will make sure to ensure the consistency between the released data and the description in the Overview.

Best,
Mohanty

πŸ“ Announcing Community Contribution Prizes 🚁πŸ₯½

About 2 months ago

We are excited to announce the following Community Contribution Prizes :

:memo: How to Participate?

  • The Community Contribution Prizes will be awarded based on the discretion of the organizers, and will be awarded to the teams/individuals, that among other actions, are most active in the community.
  • The prizes typically go to individuals or teams that are extremely active in the community, share resources - or even answer questions - that benefit the whole community greatly!
  • So, share your posts widely to spread the word and get active in the community!
  • You can make multiple submissions, but you are only eligible for the Community Contribution Prize once.
  • In case of resources that are created, your work needs to be published under a license of your choice, and on a platform that allows other participants to access and use it.

Notebooks, Blog Posts, Tutorials, Screencasts, Youtube Videos, or even your active responses on the challenge forums - everything is eligible for the Community Contribution Prizes.

We are looking forward to see everything you create!

Mavic-Air-in-action-GIF vr

All the best!

The 'product_id' field is not unique in the product table

About 2 months ago

Each product_id can have different descriptions etc depending on the product_locale. So please use the product_locale to match with the query_locale when trying to map the product_catalogue to the training set or the test set.

:point_right: More Details

Best,
Mohanty

😱 Why is there a `product_id` column in the Task 1 test set ? 😱

About 2 months ago

The v0.1 of the Task-1 test set only had the following columns : query_id, query, query_locale.

In the v0.2 release, we introduced a separate column, the product_id, and there has been some confusion as to why !

We understand, and Let us explain !

The reason the v2.0 test set of Task 1 includes pairs of query_id and product_id, is because we only have the esci_labels for those specific query_id and product_id pairs, and we can only consider those pairs for computing the nDCG score. The other query_id and product_id pairs that do not appear in the test set will not be considered for computing the nDCG score, since we do not have the corresponding esci_label.

This helps participants obtain a more meaningful nDCG score.

To help clarify this better, lets imagine a situation where we only want the top three ranked products for the query included in the test set:

Version 0.1 dataset

Input:


query_1, "some query", us

Output:


query_1, product_11

query_1, product_7

query_1, product_9

If we do not have the esci_label for any of those three pairs (say for query_1, product_7 pair), then we will omit those pairs, and compute the nDCG using only the query_1, product_11 and query_1, product_9 pairs.

Version 0.2 dataset:

Input:


query_1, product_1, locale_us

query_1, product_3, locale_us

query_1, product_10, locale_us

Output:


query_1, product_10

query_1, product_1,

query_1, product_3

query_1, product_39

In the output, we have all the pairs annotated except the query_1, product_39 pair.

But if the participants already had the information of which query-product pairs had esci_labels available, then they should not have included the query_1, product_39 pair to begin with, as it is not included the test set.

So in this case, we will compute the nDCG score considering the 3 ranked products that were included in the test set, and not the query_1, product_39 pair which was not included in the test set.

In other words, the question to answer is, how best to β€œorder” the test set of Task-1, to optimize your nDCG score.

Hope that clarifies the confusion.

If you have any more queries, please do not hesitate to reach out to us.

Best,

Mohanty

πŸš€ Datasets Released & Submissions Open πŸš€

About 2 months ago

This should be addressed in the v2.0 release of the dataset.

🚨🚨🚨 Dataset update - `v0.2` released!

About 2 months ago

Hello everyone!

We updated the dataset to address the duplicate query_id issue and released v0.2 version of the dataset.


Changes in the new version

  • Additional product_id column in task 1 test set.
  • All the CSV files except the sample submission are zipped. But you should still be able to load them in pandas. For example,
    import pandas as pd
    df = pd.read_csv("path/to/test_public-v0.2.csv.zip")
    

More details about the dataset is available at the updated post here : πŸš€ Datasets Released & Submissions Open πŸš€

Best of Luck!

πŸš€ Datasets Released & Submissions Open πŸš€

About 2 months ago

Dataset Released :rocket::rocket::rocket:

:rotating_light::rotating_light::rotating_light: NOTE: This post has been updated to reflect the changes to the dataset. Please use v0.2 version of the dataset starting 1st April, 2020.

The Shopping Queries Dataset for the Amazon KDD Cup 2022 - ESCI Challenge for Improving Product Search has been released.

You can access the datasets for each of the Tracks on the Resources Page.

The datasets contain the following files :

.
β”œβ”€β”€ task_1_query-product_ranking
β”‚   β”œβ”€β”€ product_catalogue-v0.2.csv.zip 
β”‚   β”œβ”€β”€ sample_submission-v0.2.csv
β”‚   β”œβ”€β”€ test_public-v0.2.csv.zip
β”‚   └── train-v0.2.csv.zip
β”œβ”€β”€ task_2_multiclass_product_classification
β”‚   β”œβ”€β”€ product_catalogue-v0.2.csv.zip
β”‚   β”œβ”€β”€ sample_submission-v0.2.csv
β”‚   β”œβ”€β”€ test_public-v0.2.csv.zip
β”‚   └── train-v0.2.csv.zip
└── task_3_product_substitute_identification
    β”œβ”€β”€ product_catalogue-v0.2.csv.zip
    β”œβ”€β”€ sample_submission-v0.2.csv
    β”œβ”€β”€ test_public-v0.2.csv.zip
    └── train-v0.2.csv.zip

The product_catalogue-v0.2.csv.zip for all the tasks has the following columns : product_id, product_title, product_description, product_bullet_point, product_brand, product_color_name, product_locale

Task 1

  • train-v0.2.csv.zip contains the following columns : query_id, query, query_locale, product_id, esci_label
  • test_public-v0.2.csv.zip contains the following columns : query_id, query, query_locale, product_id
  • sample_submission-v0.2.csv.zip contains the following columns : query_id, product_id

Task 2

  • train-v0.2.csv.zip contains the following columns : example_id, query, product_id, query_locale, esci_label
  • test_public-v0.2.csv.zip contains the following columns : example_id, query, product_id, query_locale
  • sample_submission-v0.2.csv.zip contains the following columns : example_id, esci_label

Task 3

  • train-v0.2.csv.zip contains the following columns : example_id, query, product_id, query_locale, substitute_label
  • test_public-v0.2.csv.zip contains the following columns : example_id, query, product_id, query_locale
  • sample_submission-v0.2.csv.zip contains the following columns : example_id, substitute_label

Download via CLI [more commands]

aicrowd datasets download -c esci-challenge-for-improving-product-search

# if you don't have AIcrowd CLI installed
pip install -U aicrowd-cli

Submissions :rocket:

You can make the submissions by clicking on the Create Submission button on the challenge page. Please do remember to select the correct Task from the drop down before submitting.

The Create Submission button is only accessible after you accept the challenge rules by clicking on the Participate button.

We very much recommend making a first submission using the included sample_submission files for each of the tracks.

Best of Luck !

πŸ‘‹ Welcome to Amazon KDD Cup '22!

2 months ago

We are in the final stages of data prep and validation. The current plans are to release the datasets on 28th March, Monday. Stay tuned :rocket:

Data Purchasing Challenge 2022

Clarification and Updates to Challenge Rules

About 2 months ago

The current challenge rules outline that the Round start and end dates are :

  • Round 1: Feb 4th, 2022, 12:00:00 pm UTC – February 28th, 2022, 12:00:00 pm UTC
  • Round 2: March 1st, 2022, 12:00:00 pm UTC – April 5th, 2022 12:00:00 pm UTC

and the cash prizes for the Top-3 teams are as follows :

  • 1st Prize: : USD 6,500
  • 2nd Prize: : USD 4,500
  • 3rd Prize: : USD 3,000

While, the information on the Challenge Overview page states that the Round start and end dates are :

  • Round 1: Feb 4th, 2022, 12:00:00 pm UTC – February 28th, 2022, 12:00:00 pm UTC
  • Round 2: March 3rd, 2022, 12:00:00 pm UTC – April 7th, 2022 12:00:00 pm UTC

and the cash prizes for the Top-3 teams are as follows :

  • 1st Prize: : USD 6,000
  • 2nd Prize: : USD 4,500
  • 3rd Prize: : USD 3,000

We confirm that the information on the Challenge Overview page reflect the most updated information about both the Round start-end times and the prizes for the top teams.

For consistency, we are updating the Challenge Rules to reflect the same, and this will require all the participants to re-accept the rules (by Clicking on the Participate button on the Challenge Overview page) before they can make any further submissions.

Apologies for any confusion, and best of luck with the competition.

Best,
Mohanty

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

3 months ago

@moto : No, unfortunately you still cannot check in your pre-trained models. The rules around pre-trained models stay the same as it was in Round 1.

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

3 months ago

@moto : You will still have to train your models from scratch, then use your trained models for making your purchasing decisions. We then take the labels that you purchased, and use it along with the training set in our training pipeline to compute the final scores.

The training pipeline we introduce is just an elaborate evaluation function to β€œassess the quality of the purchased labels”.

Wrt the prediction_phase interfaces, you will still have to submit the same, as the evaluators do a series of integration tests to ensure that the prediction_phase interface works as expected - while we do not use this function in the current evaluation pipeline, it will allow us to do a more elaborate analysis of the submissions at a later point.

Outside code cell submission

3 months ago

We will be posting an end to end notebook submission example soon with a baseline included. Hopefully that should be helpful.
@shivam is the main POC for the same.

Video tutorial on using this repo

3 months ago

Thanks Gaurav. This is super helpful.

We will be releasing some changes to the dataset and the problem statement later this week.
And we will release the official screencast videos after.

Best,
Mohanty

Potential loop hole in purchasing phase

3 months ago

Thanks @gaurav_singhal for bringing this up.

The classes available to you for local development are different from the implementation of the classes used in the evaluation setup.

In the evaluation setup, a drop-in replacement of the ProtectedDataset class is used, which interfaces with a remote service that actually knows the true labels labels. So this aspect of protecting the true labels during the purchasing phase has been taken care of in the evaluators.

Sklearn Accuracy Score is identical with exact match ratio

3 months ago

The original intent was to use a strict measure of accuracy, meaning a prediction is counted as correct when it exactly matches the ground truth labels for all the classes. However the sklearn accuracy_score actually computes the exact match ratio, as it computes the accuracy for all labels separately and then takes the mean across each.
We will discuss internally, and re-assess if we update the accuracy metric, or leave it as is for this round.

πŸ™‹πŸ»β€β™‚οΈ Frequently Asked Questions (FAQs)

4 months ago

Hi everyone,

We hope you are having a great time taking part in the ZEW Data Purchasing Challenge 2022.

 


1. :sparkles: : Can I submit pre-trained models in my submission ?

No, in this challenge, you are not allowed to include any custom pre-trained models in your submission.
Your submission can however use any of the pre-trained models already included in torchvision.models (which will be available in the local cache).

Before your submission is evaluated, all files greater than a threshold size will be scrubbed from your submission repository.
If your submission repository contains more than a threshold number of files, your submission will fail automatically.

During your submission, your submitted code will not have any network access,
so trying to access any model weights hosted externally will also lead to a failed submission.

2 :sparkles: : When will the cash prizes be awarded ?

The cash prizes will be awarded based on the final leaderboard from the Round 2 of the competition - after a manual verification of the said submission, to ensure that the submission conforms to the Rules of the competition.

3 :sparkles: : When can I submit an entry for the Community Contribution Prize ?

You can submit an entry for the Community Contribution Prize throughout the competition, up until a week before the closing date of Round 2.

4 :sparkles: What compute resources will I have access to during the evaluation ?

Your code submissions will have access to 4 CPUS, 16 GB RAM and 1 NVIDIA T4 GPU during the evaluation. The GPU is optional to use, and you can configure your need for the GPU in the aicrowd.json at the root of your submission repository.

Allowance of Pre-trained Model

4 months ago

Clever ! But we are scrubbing all files in the submitted repo which are larger than a threshold size. And if you include a base64 encoded model which has the same filesize as any arbitrary python code file, that would be a great achievement :tada:
You could however split your base64 encoded model into many many little files, and even call them as .py files :wink:, but then your submission will be disqualified during the manual inspection of the code before any prizes are awarded.

We will however clarify in the challenge rules, that any pre-trained models (other than the ones included in torchvision) are not allowed. We will try our best to technically filter out submissions which disrespect the rule, and we will definitely disqualify any submissions which disrespect this rule during the manual code inspection, if by chance, some clever submissions fall through the cracks.

We would anyway appreciate, if participants respect the spirit of the competition :tada:

Best,
Mohanty

Allowance of Pre-trained Model

4 months ago

@wufanyou : Only pre-trained models included in pytorch(via torchvision are allowed at the moment.
If you include any model weights in your submission, they will be scrubbed before the evaluation begins.

Best,
Mohanty

mohanty has not provided any information yet.