Evaluation Score Update

TL;DR: We are re-calculating scores using the correct normalization. We don’t expect any significant changes in the ranking, only on the numerical values.

Dear Participants - We are making an amendment to the evaluation score shown on the leaderboard. Unfortunately we noticed that score computation had a normalisation discrepancy with that of Melting Pot 2.0. We don’t expect the ranking of any of the submissions to change, only the numerical score. The Melting Pot 2.0 tech report defines a normalization of return for a scenario to be zero for the baseline with the lowest focal per-capita return, and one for the highest one. The normalization currently reported used the exploiter return from Melting Point 2.0 as the upper bound and random performance as lower bound. For most scenarios these are the same, but there are some for which this is not the case.

Going forward we are updating the leaderboard score to use the normalization from Melting Pot 2.0. We will use this updated normalization for the generalization score to be displayed from Nov 1 too.

Please note: This change will only reflect as a scale factor on overall score where in your performance on some scenarios may slightly decrease by the same scale for all teams and hence we do not expect any significant change in the current leaderboard ranking due to this. We will be using the new score moving forward for both development and generalization phase and also final evaluation.

View post 0

Select submissions for final leaderboard

Hi Everyone,

All teams need to select upto 3 submissions for their final leaderboard entries using the form on this page.

View post 0

Git Clone Runs infinitely

Hi Everyone,

We’re looking into what’s wrong with the submission script. In the meantime, please use the advice from @hca97. The same is also described in the starter kit. Please note the you need to push to the correct remote repo. I think for most participants, this would be set to aicrowd instead of origin, please check the name of your remote repo using git remote -v before trying to push the tag.

View post 0

Submission failed during Inference

@saidinesh_pola @gavinwangshuo , both of these failed due to timeout.

View post 0

Submission failed during Inference

@hca97 No it is not the first prediction, I see many print statements before this with varying time values.

View post 1

Submission failed during Inference

@fkemeth Can you give me the submission ids where you get the failure and I’ll check if it’s due to timeout or something else.

View post 0

Submission failed during Inference

@hca97

The timeout on these submission does indeed cross 2 seconds for one instance. I believe these are your print statements?

Totla Time  2018.3565616607666 ms

Unfortunately we can’t relax the timing criteria, if possible try to reduce the time taken by your model.

View post 0

Submissions are quite unstable

@MPWARE @tfriedel the time is measured per image. Upon request from the organizer, we have increased this number to 2 seconds per image.

@MPWARE , would it be possible to give a link to your submissions where same image got different times, as its not expected to vary based on cluster workload, your submission container gets the full node.

View post 1

Support for central_agent=False

Hi Everyone,

We now support setting central_agent=False in the Citylearn Environment. Since the creation of the environment is not directly controlled by participants. You need to provide a separate key in the aicrowd.json file in your starter kits, to let us know to use non central agents.

Things to keep in mind:

• The actions you provide will be passed to the environment as is, so make sure to use the correct action format when using non central agent.
• Make sure the aicrowd.json is formatted correctly as a valid json file.
• Make sure to use “False” in quotes when setting central agent to False in the json file.

All the best for the challenge.

View post 0

How to load a trained model during AIcrowd evaluation?

@skywuuuu You an use git-lfs using the instructions provided in the starter kit.

View post 0

Error with submissions in MDX23

Okay it’s fixed and I’ve rerun it here, you should get the scores once its completed.

View post 0

Error with submissions in MDX23

Is it not this repo?

View post 0

Error with submissions in MDX23

Hi @ZFTurbo

The old base image seems to not be available anymore, can you change to this base image in the Dockerfile

FROM nvidia/cuda:11.7.1-cudnn8-runtime-ubuntu20.04

View post 0

💬 Feedback & Suggestions

@CPMP 5 submissions per team should be enforced by the UI on a per day basis (00:00 to 00:00 UTC fixed window). We have two settings that the organizers can choose from, 24 hour rolling window and 24 hour fixed window. For this challenge its set to fixed window.

Is your team able to make more than 5 submissions in the fixed window?

View post 0

💬 Feedback & Suggestions

On Kaggle a merge is possible only if the sum of submission counts does not exceed 5 times the number of days spent on the competition so far.

Apologies, I wasn’t aware of this.

I’ll mention that we generally do not run competitions without private test sets, which is where this is a much bigger issue. This was an oversight on our part.

I checked the leaderboards and see that there is one team that has managed to make a lot of submissions doing this. After the team merge deadline we’ll check for any more unfair merges, and let the Amazon team decide if they want to provide everyone else with additional submissions to keep things fair.

View post 0

💬 Feedback & Suggestions

Agreed, but this is the system we have, it applies to everyone. If I understand correctly it is also the system on some other platforms.

I understand it’s an outsized advantage for this challenge as it’s not a code based challenge and doesn’t have a private test set.

We’re open to alternative ideas for future challenges, please share your feedback here or in dm.

View post 0

💬 Feedback & Suggestions

Hi @BenediktSchifferer

Yes, every team is limited to 5 submissions per day. After merging teams, submissions limits will apply to the whole team.

View post 0

💬 Feedback & Suggestions

@ECNU_Wei_Zhang Please ask all you team members to dm me and confirm that they want to leave the team.

View post 0

💬 Feedback & Suggestions

@ECNU_Wei_Zhang Okay your team has been removed

View post 2

Clarification and More Information for Phase 2

@ZhipengZhang Thanks for letting me know. Leaderboards should be displayed correctly now.

View post 0