Select your submissions for Test phase of the Nethack Challenge

17 October, perhaps?

View post 1

🧞 Requesting Feedback and Suggestions

I believe the finals will use 4096 runs.

I really like the stable distribution idea – maybe next year? One of the things I have considered doing is writing a function to normalize our local scores to an even role distribution, but it has never moved to the top of my priority list.

View post 0

Are the specs for the machine that the evaluations are run on available anywhere?

Thank you. That helps a lot.

When you say, “a single env should roughly give a throughput of 1500-2000 steps/second” – is that without considering time spent in the agent? How much of that is network latency? I.e., if we have an agent capable of performing faster than that, does all of the agent compute time get hidden under the latency?

View post 1

Are the specs for the machine that the evaluations are run on available anywhere?

It is hard to optimize the speed without knowing this. In particular, code changes that have given a 2x speedup on different local machines do not seem to have impacted the submission run time.

View post 0