Activity
Ratings Progression
Challenge Categories
Challenges Entered
A benchmark for image-based food recognition
Latest submissions
See Allfailed | 172430 | ||
graded | 172229 | ||
failed | 172228 |
Using AI For Buildingβs Energy Management
Latest submissions
See Allfailed | 193327 | ||
failed | 193315 | ||
failed | 193310 |
What data should you label to get the most value for your money?
Latest submissions
See Allfailed | 178246 | ||
failed | 177490 | ||
failed | 177425 |
Behavioral Representation Learning from Animal Poses.
Latest submissions
Airborne Object Tracking Challenge
Latest submissions
ASCII-rendered single-player dungeon crawl game
Latest submissions
See Allgraded | 155140 | ||
graded | 147319 |
Latest submissions
Machine Learning for detection of early onset of Alzheimers
Latest submissions
5 Puzzles 21 Days. Can you solve it all?
Latest submissions
Sample Efficient Reinforcement Learning in Minecraft
Latest submissions
The first, open autonomous racing challenge.
Latest submissions
See Allgraded | 176785 | ||
graded | 176487 | ||
graded | 176466 |
Measure sample efficiency and generalization in reinforcement learning using procedurally generated environments
Latest submissions
See Allsubmitted | 90059 | ||
graded | 83575 | ||
failed | 81249 |
Self-driving RL on DeepRacer cars - From simulation to real world
Latest submissions
Robustness and teamwork in a massively multiagent environment
Latest submissions
5 Puzzles 21 Days. Can you solve it all?
Latest submissions
Latest submissions
Play in a realistic insurance market, compete for profit!
Latest submissions
See Allgraded | 125874 | ||
graded | 121934 | ||
failed | 116909 |
5 Puzzles 21 Days. Can you solve it all?
Latest submissions
Multi-Agent Reinforcement Learning on Trains
Latest submissions
A dataset and open-ended challenge for music recommendation research
Latest submissions
A benchmark for image-based food recognition
Latest submissions
See Allgraded | 114994 | ||
graded | 114972 | ||
failed | 114971 |
Latest submissions
Sample-efficient reinforcement learning in Minecraft
Latest submissions
Latest submissions
See Allfailed | 124981 | ||
failed | 124727 | ||
failed | 124726 |
5 Puzzles, 3 Weeks. Can you solve them all? π
Latest submissions
Multi-agent RL in game environment. Train your Derklings, creatures with a neural network brain, to fight for you!
Latest submissions
Predicting smell of molecular compounds
Latest submissions
Classify images of snake species from around the world
Latest submissions
Find all the aircraft!
Latest submissions
5 Problems 21 Days. Can you solve it all?
Latest submissions
5 Puzzles 21 Days. Can you solve it all?
Latest submissions
5 Puzzles, 3 Weeks | Can you solve them all?
Latest submissions
5 PROBLEMS 3 WEEKS. CAN YOU SOLVE THEM ALL?
Latest submissions
Grouping/Sorting players into their respective teams
Latest submissions
Latest submissions
Sample-efficient reinforcement learning in Minecraft
Latest submissions
Multi Agent Reinforcement Learning on Trains.
Latest submissions
Latest submissions
See Allgraded | 191633 | ||
submitted | 191628 | ||
submitted | 191622 |
Latest submissions
See Allgraded | 60315 | ||
graded | 60314 |
Latest submissions
5 Problems 15 Days. Can you solve it all?
Latest submissions
Project 2: Road extraction from satellite images
Latest submissions
Project 2: build our own text classifier system, and test its performance.
Latest submissions
Predict if users will skip or listen to the music they're streamed
Latest submissions
Identifying relevant concepts in a large corpus of medical images
Latest submissions
Latest submissions
Latest submissions
See Allgraded | 67702 | ||
graded | 67701 | ||
graded | 67600 |
Latest submissions
Predict if users will skip or listen to the music they're streamed
Latest submissions
Latest submissions
Latest submissions
Predicting wine quality
Latest submissions
Predict whether an individual will be back to prison
Latest submissions
Latest submissions
Analyse Sentiment From Sound Clips
Latest submissions
Reinforcement Learning, IIT-M, assignment 1
Latest submissions
5 puzzles and 1 week to solve them!
Latest submissions
Latest submissions
Multi-Agent Reinforcement Learning on Trains
Latest submissions
Latest submissions
Localization, SLAM, Place Recognition, Visual Navigation, Loop Closure Detection
Latest submissions
Identify Words from silent video inputs.
Latest submissions
A Challenge on Continual Learning using Real-World Imagery
Latest submissions
Use an RL agent to build a structure with natural language inputs
Latest submissions
Participant | Rating |
---|---|
![]() |
0 |
![]() |
0 |
![]() |
0 |
![]() |
0 |
![]() |
0 |
![]() |
0 |
![]() |
0 |
![]() |
0 |
![]() |
136 |
![]() |
0 |
![]() |
0 |
![]() |
261 |
Participant | Rating |
---|---|
![]() |
0 |
![]() |
|
![]() |
136 |
![]() |
261 |
ESCI Challenge for Improving Product Search
Data Purchasing Challenge 2022
[Resolution] Bugs With Getting Started Of Round 2
Over 2 years ago@gaurav_singhal We updated the dataset file names and the extracted directory names. The zip files should extract debug
, training
, unlabelled
and validation
directories. Also updated the colab notebook with the new dataset paths.
Thanks for your help!
[Resolution] Bugs With Getting Started Of Round 2
Over 2 years ago@gaurav_singhal The updated notebook already has some code changes in the magic box cell on the lines of your suggested changes. Sharing the link to the updated notebook just in case,
[Resolution] Bugs With Getting Started Of Round 2
Over 2 years agoHello @gaurav_singhal
Thanks for pointing these out.
We will update the file names to match the names shown in the listing.
Regarding the colab notebook part, are you using the notebook that was released for round 1? We have an updated notebook for round 2. The updated notebook should have all these fixes along with additional code needed to run the post_purchase_training_phase
. Can you try using the new notebook in case if you havenβt yet?
Learn-to-Race: Autonomous Racing Virtual Challenge
L2R Docker container
Over 2 years agoHello @nandan_tumu
Unfortunately, aicrowd/learn-to-race:base
is not available publicly and only works when you submit your code to AIcrowd.
The base image is based on ubuntu and has l2r repo along with the simulator. The image is not released publicly as it has the data need for VegasNorthRoad
circuit (the private track to be used for next round).
You can specify any packages that need be installed using apt-get
in the apt.txt
file.
Example: apt.txt Β· master Β· Learn to Race / l2r-starter-kit Β· GitLab
The image has conda pre-installed, so you can also specify any conda commands that you want to run.
Example: Dockerfile Β· master Β· Learn to Race / l2r-starter-kit Β· GitLab
Finally, you can specify the pip packages in the requirements.txt
file.
Example: requirements.txt Β· master Β· Learn to Race / l2r-starter-kit Β· GitLab
You can update the Dockerfile in your repo as you seem fit except for the base image part (the first line of the Dockerfile). In case you are facing any issues in setting up the runtime dependencies during the evaluation, please feel to post on the forums or reach out to us.
DL Frameworks supported during evaluation
Over 2 years agoHello @max333
Is there some default CUDA installed? Or we have to modify this file to get any CUDA?
We do not have CUDA installed by default. You can install the version that you need by uncommenting the lines in the Dockerfile as mentioned in the above reply. Please feel free to reach out to us or the organizers if you need any further help with setting this up.
I guess also the torch version in requirements.txt should be changed to a CUDA version.
Yes, you can update the requirements.txt based in your need.
Also is there some info on the server hardware (GPU, number of cores).
The evaluations run on AWS g4dn.xlarge
nodes. They have 4 vCPUs, 16 GB RAM and 1x Nvidia Tesla T4 GPU.
DL Frameworks supported during evaluation
Over 2 years agoHello @rsakthivel
You can choose any framework. The easiest way to do this is to specify the framework/library in your requirements.txt
file.
If you need to choose a CUDA version, you can uncomment one of these lines based on your requirement.
You can also edit the Dockerfile as you see fit. However, the base image needs to be aicrowd/learn-to-race:base
.
KeyError: βsuccess_rateβ
Over 2 years agoHello all!
The way we register lap wise metrics had a bug that was causing the evaluator to skip registering a few metrics if the lap was completed by the end of the first episode. This issue is fixed and all the effected submissions were re-evaluated. Please let us know if you are still facing this issue or any of your submissions did not get re-evaluated.
KeyError: βsuccess_rateβ
Over 2 years ago@denis9 @boliu0 Thanks for reporting this. Can you check if this fixes the issue?
If this commit doesnβt fix the issue, it would help us pinpoint the problem if you can share the traceback for when this exception is raised.
How to run the simulator on Mac OS using Docker?
Over 2 years agoHello @themaroonknight
The base docker image mentioned in the dockerfile is a protected image that contains a few evaluation tracks. Unfortunately, you canβt use it to build the image locally and only works on evaluation servers.
The simulator needs an Nvidia graphics card to run and the simulator binaries are built for Linux. So running it on Mac is not feasible at this point. However, as @siddha_ganju mentioned, you can start an EC2 instance on AWS (g4dn.xlarge
instance) and train your agents.
RL Project 2021-56807e
Hit a connection error, exiting silently
Almost 3 years agoHello @tahir_javed_cs20d407
Can you give us a few submission IDs so that we can debug this further on our end?
Hit a connection error, exiting silently
Almost 3 years agoHello @tahir_javed_cs20d407
This happens when the evaluation times out (2 hours is the current timeout) or the evaluation takes up too much of memory.
Why is the evaluation failing at evaluation rollout phase?
Almost 3 years agoHello @utsav_dey_cs20s009
The issue should be fixed now. We are re-evaluating the effected submissions.
Evaluation failing at last stage
Almost 3 years agoHello @tahir_javed_cs20d407
The issue should be fixed now. We are re-evaluating the effected submissions.
Evaluation failing at last stage
Almost 3 years agoHello @tahir_javed_cs20d407
Thanks for pointing this out. We are looking into the issue and will soon fix it.
NeurIPS 2021 - The NetHack Challenge
Are the specs for the machine that the evaluations are run on available anywhere?
Almost 3 years agoHello @jon_grantham
is that without considering time spent in the agent?
Yes, it is without considering the time spent in the agent.
How much of that is network latency? I.e., if we have an agent capable of performing faster than that, does all of the agent compute time get hidden under the latency?
This includes the network latency, processing delay and everything that is needed to send a request and get the response. Any latency/compute time on the agent will be added to this.
For example, if you have something like
import aicrowd_gym
from tqdm import trange
env = aicrowd_gym.make("NetHackChallenge-v0")
env.reset()
for _ in trange(1000000):
_, _, done, _ = env.step(1)
if done:
env.reset()
This should give you a throughput of 1500-2000 iterations per second during the evaluation.
Are the specs for the machine that the evaluations are run on available anywhere?
Almost 3 years agoHello @jon_grantham
The evaluations run on AWS EC2 instances. The resources available are as follows
GPU enabled flag in aicrowd.json
|
vCPUs | Memory | GPU | AWS instance type |
---|---|---|---|---|
true |
4 | 16 GB | NVIDIA T4 | g4dn.xlarge |
false |
4 | 16 GB | - | m5.xlarge |
Note:
During the evaluations, you will receive a proxy NetHack env object instead of the actual environment. This proxy object talks to the actual NetHack env over the network and returns the values as needed. We do this to prevent participants from tampering with the env. This also adds an overhead. Based on our benchmarks, a single env should roughly give a throughput of 1500-2000 steps/second. Using something like a batched env increases the throughput.
Hope this helps and please feel free to reach out to us for any help.
Iterating over Dict spaces in aicrowd-gym
About 3 years agoHello @anssi
Thanks for sharing this with us.
- Allow access to observation/action space variables that exist in normal Gym (e.g.
spaces
for Dict obs)
We will soon update the evaluator to allow access to a few more attributes. At the moment the following attributes/methods can be accessed.
On action space:
(
"contains",
"dtype",
"sample",
"shape",
"n",
)
On observation space:
(
"bounded_above",
"bounded_below",
"contains",
"dtype",
"high",
"is_bounded",
"low",
"sample",
"shape",
"n",
)
- This is perhaps result of bad programming from my end, but make sure variables that do not exist in the environment return appropriate exceptions. For example, I did
hasattr("something", env)
.something
was not a variable in the environment, but with aicrowd-gym it was set toNone
. This causedhasattr
to return True and subsequently things failing.
I believe this happens with gym.Env
objects as well. For example, if I do
import gym
env = gym.make("CartPolve-v0")
setattr(env, "something", "some value")
it doesnβt return any error. If possible, can you share with a simple example the expected behaviour?
Note: During the evaluation, the env object you have access to is a proxy object and not the actual gym.Env
instance. So setting attributes from your code will not set any attributes on the actual env object. If you have any specific use case for doing this, please feel free to reach out to us and we can try to accommodate your use case.
Edit: Sorry, misread the last part. We will try to get the read access to all the env attributes in our future releases.
Music Demixing Challenge ISMIR 2021
Can I use GPU for prediction?
About 3 years agoHello @woosung_choi @Li_Xue_Han
Sorry, my bad. GPUs are not available for this competition. Apologies again for the confusion.
Can I use GPU for prediction?
About 3 years agoHello @Li_Xue_Han
Edit: GPUs are not available for this competition
You can use GPUs for inference. Please refer How to enable GPU for your submission? for more information on using GPUs for your submission.
Notebooks
-
Solution for submission 128368 A detailed solution for submission 128368 submitted for challenge IIT-M RL-ASSIGNMENT-2-TAXIjyotishΒ· Over 3 years ago
-
[Baseline] Detectron2 starter kit for food recognition π A beginner friendly notebook kick start your instance segmentation skills with detectron2jyotishΒ· Over 3 years ago
BUG in evaluation timing
About 2 years agoHello @dami
Thanks for reporting this issue. It is a minor glitch on our side that showed the wrong end time on the issue page. We released a fix for this and future submissions should show the correct end time. Please note that these time stamps are not used for timeouts.