🧞 Requesting Feedback and Suggestions

Hey @dipam,

I just gave some feedback about tight time limits in this comment.

View post 0

Are the specs for the machine that the evaluations are run on available anywhere?

Hello @jyotish,

If the throughput is in 1500-2000 range, it indicates that the maximum average number of steps is only {1500--2000} * 0.5 * 3600 / 128 = 21k -- 28k per game.
Also a note, that the example you gave doesn’t really take into account the environment step delay, because env.step(1) (1 is CompassDirection.E) is a no-op after a few steps when the character hits the wall (the turn counter stops to tick after that).

28k steps per game is an extremely tight limit if one were to go for ascension (we are!). Assuming that we need 200k turns for ascension, which can be roughly equivalent to 400k steps, indicates that we have to early drop at least 93% of all games. We already have >40k steps in average, and have to do hacks to maximize the median score, like quitting after exceeding the median (in fact Panic team does the same judging from the leaderboard).

Of course the assumption that agent takes no time to execute is not realistic. In our case environment takes ~15% of the entire execution time (measured locally so there’s no communication delay).

The competition goal is to develop the best agent, but right now it’s more like an performance optimization problem instead. Citing the challenge motivation: “The only restriction is on the compute and runtime during evaluation, though these will be set to very generous limits to support a wide range of possible implementations”. Currently the compute limit is far from being “very generous” and to do this, we believe the time limit should be increased at least 10 times.

So, we encourage you to increase the time limit as much as possible to give participants a better chance to beat the game. With the current limit ascension is extremely unlikely, but it’s quite possible if the limit is increased.

View post 3

Evaluate rollouts timed out

Admins responded to me that the video recording timed out and they will release a fix for it soon.

View post 0

Evaluate rollouts timed out

I got a timeout on “Evaluate Rollouts” step. I’m not sure what the step is exactly doing, but I guess it doesn’t involve running the agent. If that’s the case, can you increase time limit for this step?

What does this step do and why does it take so long? Some of our episodes are quite long (>100k steps), maybe that’s the reason?

img762×434 35.1 KB

View post 1