Loading
0 Follower
0 Following
unixpickle
Alex Nichol

Location

US

Badges

2
0
0

Connect

Activity

Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Mon
Wed
Fri

Ratings Progression

Loading...

Challenge Categories

Loading...

Challenges Entered

A new benchmark for Artificial Intelligence (AI) research in Reinforcement Learning

Latest submissions

See All
graded 8719
graded 8714
graded 8711
Participant Rating
Participant Rating
unixpickle has not joined any teams yet...

Unity Obstacle Tower Challenge

Episode in evaluation ends prematurely

Over 5 years ago

My latest evaluation yielded this:

  • Episode-5
    • State: Episode Complete :tada:
    • Floor Number: 2
    • Reward: 2.400
    • Steps: 649

I find this rich, considering that there is no way to die on the third floor without running out of time, and an agent starts out with 3000 frames of time even if they pick up no time orbs. Clearly, there is some bug in the environment or the evaluation.

Anyway, I just resubmitted and will hopefully not hit this bug again.

Is the new v2.2 used for scoring?

Over 5 years ago

Same issue here, stuck on generating video. Maybe that feature should simply be removed until it can be implemented reliably.

UnityTimeOutException in evaluation

Over 5 years ago

Spoke too soon. Just did another submission and got another timeout after 998s seconds (my timeout was set to 900). Must be non-deterministic.

UnityTimeOutException in evaluation

Over 5 years ago

@mohanty looks like your suggestion worked! Submission runs now.

UnityTimeOutException in evaluation

Over 5 years ago

I see this stack trace when I try to submit an agent that worked previously:

2019-05-24T17:42:52.825782788Z root
2019-05-24T17:43:05.17870298Z INFO:mlagents_envs:Start training by pressing the Play button in the Unity Editor.
2019-05-24T17:43:35.184856349Z Traceback (most recent call last):
2019-05-24T17:43:35.184901058Z   File "run.py", line 56, in <module>
2019-05-24T17:43:35.184908019Z     env = create_single_env(args.environment_filename, docker_training=args.docker_training)
2019-05-24T17:43:35.184929282Z   File "/home/aicrowd/util.py", line 16, in create_single_env
2019-05-24T17:43:35.184932961Z     env = ObstacleTowerEnv(path, **kwargs)
2019-05-24T17:43:35.184935919Z   File "/srv/conda/lib/python3.6/site-packages/obstacle_tower_env.py", line 45, in __init__
2019-05-24T17:43:35.184939382Z     timeout_wait=timeout_wait)
2019-05-24T17:43:35.184942214Z   File "/srv/conda/lib/python3.6/site-packages/mlagents_envs/environment.py", line 69, in __init__
2019-05-24T17:43:35.184945802Z     aca_params = self.send_academy_parameters(rl_init_parameters_in)
2019-05-24T17:43:35.184948806Z   File "/srv/conda/lib/python3.6/site-packages/mlagents_envs/environment.py", line 491, in send_academy_parameters
2019-05-24T17:43:35.184952019Z     return self.communicator.initialize(inputs).rl_initialization_output
2019-05-24T17:43:35.184954878Z   File "/srv/conda/lib/python3.6/site-packages/mlagents_envs/rpc_communicator.py", line 80, in initialize
2019-05-24T17:43:35.184958142Z     "The Unity environment took too long to respond. Make sure that :\n"
2019-05-24T17:43:35.184963164Z mlagents_envs.exception.UnityTimeOutException: The Unity environment took too long to respond. Make sure that :
2019-05-24T17:43:35.184968225Z 	 The environment does not need user interaction to launch
2019-05-24T17:43:35.184972965Z 	 The Academy and the External Brain(s) are attached to objects in the Scene
2019-05-24T17:43:35.184977708Z 	 The environment and the Python interface have compatible versions.

I am definitely using v2.1 of the environment. Not sure what’s going on, but this happened several submissions in a row, and I see that nobody else has successfully submitted in a few days.

Track submissions to the leaderboard

Over 5 years ago

I created a script that tracks new submissions to the leaderboard and logs them to the console. Watch as people gradually all realize they can submit the Rainbow baseline. xD

import itertools
import time

from bs4 import BeautifulSoup
import requests

BASE_URL = 'https://www.aicrowd.com/challenges/unity-obstacle-tower-challenge/leaderboards'


def main():
    board = fetch_leaderboard()
    while True:
        time.sleep(60 * 5)
        new_board = fetch_leaderboard()
        print_diffs(board, new_board)
        board = new_board


def print_diffs(old, new):
    for k, v in new.items():
        if k not in old:
            print('new submission: %s -> %f' % (k, v))
        elif old[k] != v:
            print('new submission: %s -> %f (old: %f)' % (k, v, old[k]))
    for k, v in old.items():
        if k not in new:
            print('deleted submission: %s -> %f' % (k, v))


def fetch_leaderboard():
    result = {}
    for page in itertools.count():
        sub_result = fetch_leaderboard_page(page)
        if not len(sub_result):
            break
        result.update(sub_result)
    return result


def fetch_leaderboard_page(page):
    url = '%s?page=%d' % (BASE_URL, page)
    response = requests.get(url)
    data = response.text
    soup = BeautifulSoup(data, 'html.parser')
    table = soup.find('table', {'class': 'table-leaderboard'})
    result = {}
    for row in table.find('tbody').find_all('tr'):
        columns = row.find_all('td')
        result[columns[2].get_text().strip()] = float(columns[4].get_text().strip())
    return result


if __name__ == '__main__':
    main()

Programming is fun.