Scores of all tasks made public 💡

Hi Everyone,

We’ve decided to make the scores of all the tasks and borda ranks of each submission public. Now you can view the scores of all the tasks. Our original motivation was to prevent overfitting on the tasks in the spirit of the challenge being unsupervised representation learning. But we understand that its frustrating for participants to not have any source of feedback on the tasks and why one of their submissions does better than another.

P.S: The scores visible are on the public split of the data. (Ignore that the names say private). The scores on the private split will be available after the competition ends, and will be used for selecting the winners (No changes here)

P.P.S: Negative scores indicate MSE scores, these are made negative because the borda system requires “higher is better”. So you can ignore the negative sign when looking at the MSE scores.

For those interested, here’s a short description on how the selection system works (taken from this discussion from AIcrowd Discord)

The selection system we have works like this:
First your own submissions are borda ranked, the best among these is selected, then your submission is borda ranked against other top submissions from each team. This is done to prevent any team from having multiple entries to increase their rank gap based on a single task.

As a consequence of this, you are also competing against your submissions in the borda system. In this case, your new submissions may perform slightly better in average borda rank against your own submissions, because it can significantly outperform on a few of the hidden tasks.

Note that the above case may cause the a submission with lower average f1 score to be selected.

View post 1

Round 2 Deadlines updated 🚀

Hi Everyone,

Happy to announce that we’ve extended Round 2 by 1 and a half months after receiving your feedback that the current timeline was too short to work with video data. Looking forward to your solutions!

Round 2 now ends on 3rd July 2022.

View post 1

🚀 Share your solutions! 🚀

Hi Everyone,

Thank you for participating in the Data Purchasing Challenge, it has been a unique journey for this one-of-a-kind challenge. We’re excited to know your ideas and solutions. Regardless of whether you won, any ideas you share on the discussion forum are highly appreciated.

Sharing your solutions also helps you reflect upon your learnings throughout the competition.

I’ll summarize all the solutions that are shared into this post, as I’m sure they’ll be tremendously useful for participants of future Data Purchasing Challanges (yes there will be more )

Please also share failed ideas, as all machine learning afficianados know, negative samples are just as important.

Looking forward to your solutions!

View post 1

[Announcement] Leaderboard Winners

Hi @xiaozhou_wang, glad you’ll be sharing your solutions. Congratulations on bagging the top spot. As @Camaro suggested, please make a discourse post, and you can also make your submissions repository on AIcrowd Gitlab public if you want to share your full submissions journey.

View post 2

:rotating_light: Select submissions for final evaluation

Hi @chuifeng

No, all the runs are completed. The leaderboard is final. The best scoring submission of each participant from the full runs was used. The “successful entries” column is generic feature of the leaderboard, since leaderboard was calculated offline and the final results added separately, it shows 1.

Please check this sheet with the full scores of all the runs.

View post 0

:rotating_light: Select submissions for final evaluation

Hi @leocd

The highest scoring among the selected submissions will be used.

View post 2

:rotating_light: Select submissions for final evaluation

Hi everyone,

Thanks for participating in the Data Purchasing Challenge!

We hope you had fun with the problem and the dataset. In this next phase, we will be evaluating your selected submissions on the full hidden dataset.

‎ℹ️ Know more about it here:

• https://discourse.aicrowd.com/t/important-details-about-end-of-competition-evaluations/7512
• https://discourse.aicrowd.com/t/code-for-end-of-competition-training-pipelines/7547

Deadline for filling this form is 08 April 2022, 23:59 UTC (earlier the better, thanks )

Select Submission ID for Full Evaluation

View post 3

Code for End of Competition Training pipelines

Hi @sergey_zlobin , all the compute budget pairs will be run with the purchase phase first. Then the post purchase evaluations will be run on each of the purchased sets. As is already the case, there is no time limit on the post purchase training as its entirely controlled by us.

View post 1

Code for End of Competition Training pipelines

Hi Everyone,

Below are the links to the end of competition training pipelines that will be used for the final winner selection. Each pipeline will be run for 2 seeds. The code release is for your feedback, so please feel free to go through them.

In case you have no idea what I’m talking about, please check this post.

If there are any critical issues we’ll fix them.

I tried many different combinations and rejected some if the gap between random and all label purchase was too low, apologies if your favourite combination didn’t make it.

Check the commit history easily navigating the changes.

View post 6

IMPORTANT: Details about end of competition evaluations 🎯

@Camaro Probably no extension as far as I know. Will inform in case of any changes.

View post 0

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

@Camaro

Yes what you say makes sense, although on the other end a very strong model was getting nearly as good as “all label purchase” scores with just random purchase, so the dataset needed more difficulty, important lessons learnt. In any case, I agree with your definition of useful, for now we’ve come up with the end of competition evaluations scheme. Please check this recent post.

View post 1

IMPORTANT: Details about end of competition evaluations 🎯

Hi Everyone,

I’ve seen multiple people’s posts suggesting changes in the purchase phase training pipeline. This post is to clarify some details about the end of competition evaluations and explain some its implications.

First of all, thanks a lot to everyone who have provided feedback for the training pipeline. It has been very valuable to us for the challenge design.

TL;Dr - Please submit your best purchase strategies, you’ll need to select them for end of competition evaluations which will run 5 post purchase training pipelines.

Current problems

Many have pointed out that the training pipeline does not provide good scores. I’ll break these down some categories.

• Concerns about the best purchases not being incentivised correctly

  1. Model is too weak and cannot learn hard examples - This is a legitimate concern, though I do not currently know the scale to which it is applicable. We’re investigating this.
  2. Not using GaussianBlur in testing changes the optimal labels - Currently I have not found any strong evidence of this, but would love to discuss further.
  3. Model does not converge due to low epochs hence too stochastic - I did not find the stochasticity in scores to be too much and feel it is in expected practical limits. However, in the later part of this post I’ll be addressing this further.

  These are the most important concerns we’re looking to resolve. Some of them are also difficult to measure, and the discussion becomes qualitative and opinionated. We want to resolve it in the most quantitative and fair way possible.

• Concerns that the score is too low

  1. The feature layers are frozen
  2. The model is too small and plateaus
  3. GaussianBlur is not used in test

  For these, I’d like to reiterate that the score is not important, rather, maximising the score by making the best purchase strategies is the goal of the competition.

End of competition Evaluations

In addition to changing the dataset for end of competition evaluations. We will run select submissions on multiple training pipelines in the post purchase phase.

The detailed steps are given below:

1. Eligible teams will select two of their submissions to evaluate - Eligibility criteria to be announced soon, it will be based on Round 2 leaderboard.
2. Each submission will run through the pre-train and the purchase phase on the end of competition dataset.
3. The same purchased labels will be put through 5 training pipelines - Details to be released soon.
4. Each training pipeline will be run for 2 seeds and scores averaged, to address any stochasticity in scores.
5. To avoid issues due to difference of average scores from different training pipelines, a Borda ranking system will be used.

We hope this will inceltivize participants to select the best purchase strategies and not optimize for the current training pipeline. We’re unable to incorporate this setup during the live round due to the prohibitive cost of compute for each run.

Training pipeline survey

Please vote here for your favoured schemes. Note that is not a vote to select the training pipelines, just a survey of participant’s preferences, but we’ll take the results very seriously.

Please feel free to suggest any other changes if I have missed them.

View post 4

Title for the post

Body for the post, which will be edited soon.

View post 0

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

Hi @tfriedel

Seems I missed your message. The point on ruling out the most useful images, do you feel its a huge issue with the current training pipeline?

We do want the solutions to be as agnostic to the training pipeline while buying the best images possible. Yes its not completely possible to make things training agnostic, but that is the spirit of the competition we’d like to promote. If you’re finding that you’re deliberately having to remove too much of the useful images, please let me know.

View post 0

Baseline Released :gift:

We hope you have been following the development around the challenge.

Today, we are finally releasing a baseline, which was promised in the last townhall.

Did you miss the townhall?
You can check about it here: 📹 Town Hall Recording & Resources from top participants

About the Baseline

The baseline contains fast heuristic implementations of some simple ideas.

• Purchase images with more labels - For multilabel datasets, often having images with more than one label gives a boost for deep learning models.
• Purchase uncertain images - Purchase images which have the most uncertainty in their predictions. While many methods exists to measure uncertainty, a simple output probability based heuristic method is used here.
• Purchase images to balance labels - Well balanced datasets can improve model performance in deep learning. We set a uniform target distribution and try to purchase labels to get closer to that distribution. The provided code can try to purchase labels to any target distribution.

Learn more about the implementation here.
Jump to the codebase directly here.
Looking for more resources to explore? Check out this thread.

Ok, talk is easy, how does it perform?

Well, as it stands, the baseline is currently on 5th position on the whole leaderboard.

image1210×254 44.2 KB

Do you have additional questions?

Feel free to drop them in this thread and we can reply them asap.
What are you waiting for? Let’s get started with the submissions!

View post 6

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

Hi @Camaro

Thanks for the clarification and useful explanation. I’ll consider this and try some experiments. One issue with designing the challenge is it needs balance between good training pipeline, score gap for better data, and compute constraints. All these need to be satisfied while iterating our synthetic data to match these constraints. We’ll try to improve the training pipeline accordingly if we’re able to match them in a reasonable way.

View post 1

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

Hi @Camaro

I understand your concern about underfitting. However, this challenge is a bit non-traditional, the data is completely synthetic, and the purpose is solely the methods used for research only. The final model trained is not of importance to any real-world setting, only the algorithms you develop.

Also, the purpose of the training pipeline is to give a level playing field to all participants to that they focus the purchase strategies. Whether the final loss value is reached is not important as long as better data purchased produces better scores using the same training pipeline, which is what we’ve tried to setup with Round 2.

Your goal is to improve the score by purchasing better labels, the score may be limited by the training setup once the best labels are purchased, but that is not yet the case in my opinion.

View post 0

Why there is no GaussianBlur in test transform?

Hi @chuifeng

Yes the GaussianBlur in training should have been applied with a probability value during training to get higher scores. Or as you suggest, could be used during test which will also increase the score.

Though I’d still like to stress that the gap in scores between random purchase and the all-label purchase is the criteria that is more important that the raw scores.

You can refer to the discussion here:

View post 0

:aicrowd: [Update] Round 2 of Data Purchasing Challenge is now live!

Hi @tfriedel

Currently, I do not see this much spread in scores when testing on the private set which is used for leaderboard. Nevertheless, thanks for pointing this out, will check further on the score spread across the budgets used.

Also note that the final end of competition evaluations will not use the dataset used on the leaderboard now, but a different dataset sampled from the same distribution. The end of competition evalutions will also feature more exhaustive evaluations with many more models. Hence overfitting the leaderboard is likely to hurt participants when the final evaluations take place. We’ll make communications about this clearer in case this isn’t properly explained.

View post 1

Evaluations are being re-run with balanced class weights

Hi Everyone,

As you might have recently got a e-mail for, the leaderboard is currently being re-run with balanced class weights. The scores might be unstable until all the runs finish. Please do not worry if you suddenly see your ranks jumping around too much. They should stabilize once all the runs are done.

View post 1