Loading
12 Follower
4 Following
jyotish
Jyotish

Organization

AIcrowd

Location

Geneva, CH

Badges

8
3
3

Connect

Activity

Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Mon
Wed
Fri

Ratings Progression

Loading...

Challenge Categories

Loading...

Challenges Entered

Latest submissions

See All
graded 192426
failed 192410
submitted 192407

Latest submissions

See All
failed 193327
failed 193315
failed 193310

What data should you label to get the most value for your money?

Latest submissions

See All
failed 178246
failed 177490
failed 177425

A benchmark for image-based food recognition

Latest submissions

See All
failed 172430
graded 172229
failed 172228

Behavioral Representation Learning from Animal Poses.

Latest submissions

No submissions made in this challenge.

Airborne Object Tracking Challenge

Latest submissions

No submissions made in this challenge.

ASCII-rendered single-player dungeon crawl game

Latest submissions

See All
graded 155140
graded 147319

Machine Learning for detection of early onset of Alzheimers

Latest submissions

No submissions made in this challenge.

5 Puzzles 21 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

Sample Efficient Reinforcement Learning in Minecraft

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
graded 176785
graded 176487
graded 176466

Measure sample efficiency and generalization in reinforcement learning using procedurally generated environments

Latest submissions

See All
submitted 90059
graded 83575
failed 81249

Self-driving RL on DeepRacer cars - From simulation to real world

Latest submissions

No submissions made in this challenge.

Robustness and teamwork in a massively multiagent environment

Latest submissions

No submissions made in this challenge.

3D Seismic Image Interpretation by Machine Learning

Latest submissions

See All
failed 99353

5 Puzzles 21 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

Play in a realistic insurance market, compete for profit!

Latest submissions

See All
graded 125874
graded 121934
failed 116909

5 Puzzles 21 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

Multi-Agent Reinforcement Learning on Trains

Latest submissions

No submissions made in this challenge.

A dataset and open-ended challenge for music recommendation research

Latest submissions

No submissions made in this challenge.

A benchmark for image-based food recognition

Latest submissions

See All
graded 114994
graded 114972
failed 114971

Latest submissions

No submissions made in this challenge.

Sample-efficient reinforcement learning in Minecraft

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
failed 124981
failed 124727
failed 124726

5 Puzzles, 3 Weeks. Can you solve them all? 😉

Latest submissions

No submissions made in this challenge.

Multi-agent RL in game environment. Train your Derklings, creatures with a neural network brain, to fight for you!

Latest submissions

No submissions made in this challenge.

Predicting smell of molecular compounds

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

5 Problems 21 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

5 Puzzles 21 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

5 Puzzles, 3 Weeks | Can you solve them all?

Latest submissions

No submissions made in this challenge.

5 PROBLEMS 3 WEEKS. CAN YOU SOLVE THEM ALL?

Latest submissions

No submissions made in this challenge.

Grouping/Sorting players into their respective teams

Latest submissions

No submissions made in this challenge.

Dog breed classification

Latest submissions

No submissions made in this challenge.

5 Problems 15 Days. Can you solve it all?

Latest submissions

See All
failed 71051
failed 71041

Sample-efficient reinforcement learning in Minecraft

Latest submissions

No submissions made in this challenge.

Multi Agent Reinforcement Learning on Trains.

Latest submissions

No submissions made in this challenge.

Recognise Handwritten Digits

Latest submissions

See All
graded 191633
submitted 191628
submitted 191622

Crowdsourced Map Land Cover Prediction

Latest submissions

See All
graded 60315
graded 60314

Latest submissions

No submissions made in this challenge.

5 Problems 15 Days. Can you solve it all?

Latest submissions

No submissions made in this challenge.

Project 2: Road extraction from satellite images

Latest submissions

No submissions made in this challenge.

Project 2: build our own text classifier system, and test its performance.

Latest submissions

No submissions made in this challenge.

Predict if users will skip or listen to the music they're streamed

Latest submissions

No submissions made in this challenge.

Identifying relevant concepts in a large corpus of medical images

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

5 PROBLEMS 3 WEEKS. CAN YOU SOLVE THEM ALL?

Latest submissions

See All
failed 77264

Real Time Mask Detection

Latest submissions

See All
graded 67702
graded 67701
graded 67600

Latest submissions

No submissions made in this challenge.

Predict if users will skip or listen to the music they're streamed

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

Solve the jigsaw and finish the picture!

Latest submissions

No submissions made in this challenge.

Predicting wine quality

Latest submissions

No submissions made in this challenge.

Predict whether an individual will be back to prison

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

Analyse Sentiment From Sound Clips

Latest submissions

No submissions made in this challenge.

Predict viewer reactions from a large-scale video dataset!

Latest submissions

See All
graded 124097

Reinforcement Learning, IIT-M, assignment 1

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
failed 156316

5 puzzles and 1 week to solve them!

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
graded 128368

Latest submissions

No submissions made in this challenge.

Multi-Agent Reinforcement Learning on Trains

Latest submissions

No submissions made in this challenge.

Latest submissions

No submissions made in this challenge.

Train your RL agents

Latest submissions

See All
graded 165868
failed 162152

Localization, SLAM, Place Recognition, Visual Navigation, Loop Closure Detection

Latest submissions

No submissions made in this challenge.

Identify Words from silent video inputs.

Latest submissions

No submissions made in this challenge.

Latest submissions

See All
failed 195996
submitted 195995
failed 183788

A Challenge on Continual Learning using Real-World Imagery

Latest submissions

No submissions made in this challenge.

Use an RL agent to build a structure with natural language inputs

Latest submissions

No submissions made in this challenge.
Participant Rating
BhaviD 0
will_kwan 0
lars12llt 0
jansi_rani_s_v 0
branden_murray 0
saketha_ramanujam 0
vrv 0
jerome_patel 0
shivam
cadabullos 0
krishna_kaushik 0
unnikrishnan.r 261
Participant Rating
vrv 0
aicrowd-bot
shivam
unnikrishnan.r 261

ESCI Challenge for Improving Product Search

BUG in evaluation timing

5 months ago

Hello @dami

Thanks for reporting this issue. It is a minor glitch on our side that showed the wrong end time on the issue page. We released a fix for this and future submissions should show the correct end time. Please note that these time stamps are not used for timeouts.

Data Purchasing Challenge 2022

[Resolution] Bugs With Getting Started Of Round 2

9 months ago

@gaurav_singhal We updated the dataset file names and the extracted directory names. The zip files should extract debug, training, unlabelled and validation directories. Also updated the colab notebook with the new dataset paths.

Thanks for your help!

[Resolution] Bugs With Getting Started Of Round 2

9 months ago

@gaurav_singhal The updated notebook already has some code changes in the magic box cell on the lines of your suggested changes. Sharing the link to the updated notebook just in case,

[Resolution] Bugs With Getting Started Of Round 2

9 months ago

Hello @gaurav_singhal

Thanks for pointing these out.

We will update the file names to match the names shown in the listing.


Regarding the colab notebook part, are you using the notebook that was released for round 1? We have an updated notebook for round 2. The updated notebook should have all these fixes along with additional code needed to run the post_purchase_training_phase. Can you try using the new notebook in case if you haven’t yet?

Learn-to-Race: Autonomous Racing Virtual Challenge

L2R Docker container

10 months ago

Hello @nandan_tumu

Unfortunately, aicrowd/learn-to-race:base is not available publicly and only works when you submit your code to AIcrowd.

The base image is based on ubuntu and has l2r repo along with the simulator. The image is not released publicly as it has the data need for VegasNorthRoad circuit (the private track to be used for next round).


You can specify any packages that need be installed using apt-get in the apt.txt file.

Example: apt.txt · master · Learn to Race / l2r-starter-kit · GitLab


The image has conda pre-installed, so you can also specify any conda commands that you want to run.

Example: Dockerfile · master · Learn to Race / l2r-starter-kit · GitLab


Finally, you can specify the pip packages in the requirements.txt file.

Example: requirements.txt · master · Learn to Race / l2r-starter-kit · GitLab


You can update the Dockerfile in your repo as you seem fit except for the base image part (the first line of the Dockerfile). In case you are facing any issues in setting up the runtime dependencies during the evaluation, please feel to post on the forums or reach out to us.

DL Frameworks supported during evaluation

10 months ago

Hello @max333

Is there some default CUDA installed? Or we have to modify this file to get any CUDA?

We do not have CUDA installed by default. You can install the version that you need by uncommenting the lines in the Dockerfile as mentioned in the above reply. Please feel free to reach out to us or the organizers if you need any further help with setting this up.

I guess also the torch version in requirements.txt should be changed to a CUDA version.

Yes, you can update the requirements.txt based in your need.

Also is there some info on the server hardware (GPU, number of cores).

The evaluations run on AWS g4dn.xlarge nodes. They have 4 vCPUs, 16 GB RAM and 1x Nvidia Tesla T4 GPU.

DL Frameworks supported during evaluation

10 months ago

Hello @rsakthivel

You can choose any framework. The easiest way to do this is to specify the framework/library in your requirements.txt file.

If you need to choose a CUDA version, you can uncomment one of these lines based on your requirement.

You can also edit the Dockerfile as you see fit. However, the base image needs to be aicrowd/learn-to-race:base.

KeyError: ‘success_rate’

10 months ago

Hello all!

The way we register lap wise metrics had a bug that was causing the evaluator to skip registering a few metrics if the lap was completed by the end of the first episode. This issue is fixed and all the effected submissions were re-evaluated. Please let us know if you are still facing this issue or any of your submissions did not get re-evaluated.

KeyError: ‘success_rate’

10 months ago

@denis9 @boliu0 Thanks for reporting this. Can you check if this fixes the issue?

If this commit doesn’t fix the issue, it would help us pinpoint the problem if you can share the traceback for when this exception is raised.

How to run the simulator on Mac OS using Docker?

12 months ago

Hello @themaroonknight

The base docker image mentioned in the dockerfile is a protected image that contains a few evaluation tracks. Unfortunately, you can’t use it to build the image locally and only works on evaluation servers.

The simulator needs an Nvidia graphics card to run and the simulator binaries are built for Linux. So running it on Mac is not feasible at this point. However, as @siddha_ganju mentioned, you can start an EC2 instance on AWS (g4dn.xlarge instance) and train your agents.

RL Project 2021-56807e

Hit a connection error, exiting silently

About 1 year ago

Hello @tahir_javed_cs20d407

Can you give us a few submission IDs so that we can debug this further on our end?

Hit a connection error, exiting silently

About 1 year ago

Hello @tahir_javed_cs20d407

This happens when the evaluation times out (2 hours is the current timeout) or the evaluation takes up too much of memory.

Why is the evaluation failing at evaluation rollout phase?

About 1 year ago

Hello @utsav_dey_cs20s009

The issue should be fixed now. We are re-evaluating the effected submissions.

Evaluation failing at last stage

About 1 year ago

Hello @tahir_javed_cs20d407

The issue should be fixed now. We are re-evaluating the effected submissions.

Evaluation failing at last stage

About 1 year ago

Hello @tahir_javed_cs20d407

Thanks for pointing this out. We are looking into the issue and will soon fix it.

NeurIPS 2021 - The NetHack Challenge

Are the specs for the machine that the evaluations are run on available anywhere?

About 1 year ago

Hello @jon_grantham

is that without considering time spent in the agent?

Yes, it is without considering the time spent in the agent.

How much of that is network latency? I.e., if we have an agent capable of performing faster than that, does all of the agent compute time get hidden under the latency?

This includes the network latency, processing delay and everything that is needed to send a request and get the response. Any latency/compute time on the agent will be added to this.

For example, if you have something like

import aicrowd_gym
from tqdm import trange

env = aicrowd_gym.make("NetHackChallenge-v0")
env.reset()
for _ in trange(1000000):
    _, _, done, _ = env.step(1)
    if done:
        env.reset()

This should give you a throughput of 1500-2000 iterations per second during the evaluation.

Are the specs for the machine that the evaluations are run on available anywhere?

About 1 year ago

Hello @jon_grantham

The evaluations run on AWS EC2 instances. The resources available are as follows

GPU enabled flag in aicrowd.json vCPUs Memory GPU AWS instance type
true 4 16 GB NVIDIA T4 g4dn.xlarge
false 4 16 GB - m5.xlarge

Note:
During the evaluations, you will receive a proxy NetHack env object instead of the actual environment. This proxy object talks to the actual NetHack env over the network and returns the values as needed. We do this to prevent participants from tampering with the env. This also adds an overhead. Based on our benchmarks, a single env should roughly give a throughput of 1500-2000 steps/second. Using something like a batched env increases the throughput.

Hope this helps and please feel free to reach out to us for any help.

Iterating over Dict spaces in aicrowd-gym

Over 1 year ago

Hello @anssi

Thanks for sharing this with us.

  • Allow access to observation/action space variables that exist in normal Gym (e.g. spaces for Dict obs)

We will soon update the evaluator to allow access to a few more attributes. At the moment the following attributes/methods can be accessed.

On action space:

(
    "contains",
    "dtype",
    "sample",
    "shape",
    "n",
)

On observation space:

(
    "bounded_above",
    "bounded_below",
    "contains",
    "dtype",
    "high",
    "is_bounded",
    "low",
    "sample",
    "shape",
    "n",
)


  • This is perhaps result of bad programming from my end, but make sure variables that do not exist in the environment return appropriate exceptions. For example, I did hasattr("something", env). something was not a variable in the environment, but with aicrowd-gym it was set to None. This caused hasattr to return True and subsequently things failing.

I believe this happens with gym.Env objects as well. For example, if I do

import gym
env = gym.make("CartPolve-v0")
setattr(env, "something", "some value")

it doesn’t return any error. If possible, can you share with a simple example the expected behaviour?

Note: During the evaluation, the env object you have access to is a proxy object and not the actual gym.Env instance. So setting attributes from your code will not set any attributes on the actual env object. If you have any specific use case for doing this, please feel free to reach out to us and we can try to accommodate your use case.

Edit: Sorry, misread the last part. We will try to get the read access to all the env attributes in our future releases.

Music Demixing Challenge ISMIR 2021

Can I use GPU for prediction?

Over 1 year ago

Hello @woosung_choi @Li_Xue_Han

Sorry, my bad. GPUs are not available for this competition. Apologies again for the confusion.

Can I use GPU for prediction?

Over 1 year ago

Hello @Li_Xue_Han

Edit: GPUs are not available for this competition

You can use GPUs for inference. Please refer How to enable GPU for your submission? for more information on using GPUs for your submission.

jyotish has not provided any information yet.

Notebooks

Create Notebook