Loading
AIcrowd
Challenges
Organize a challenge FAQ Contact Jobs
Log In Sign Up

NeurIPS 2022 IGLU Challenge

Challenge Rules

COMPETITION OFFICIAL RULES

PLEASE READ THESE OFFICIAL RULES CAREFULLY. ENTRY INTO THIS COMPETITION CONSTITUTES YOUR ACCEPTANCE OF THESE OFFICIAL RULES. IF YOU DO NOT AGREE TO ANY PART OF THESE OFFICIAL RULES, PLEASE DO NOT ENTER THIS CHALLENGE.

NO PURCHASE IS NECESSARY TO ENTER OR WIN. A PURCHASE OF ANY KIND WILL NOT INCREASE YOUR CHANCES OF WINNING VOID WHERE PROHIBITED.

The following rules attempt to capture the spirit of the IGLU competition and any submissions found to be violating the rules may be deemed ineligible for participation by the organizers.

  • The final leaderboard will be determined by the weighted mean of the ranks of a participant/team across all the individual leaderboards. The organizers reserve the right to change the weight of a problem during the competition.
  • If a problem is not attempted by the participant, their rank in that problem is set to the total number of participants in the whole competition.
  • To help you "Get Started" we have a starter kit available for each of these problems. We hope they are helpful; if you find any bugs, typos,or improvements, please send us a pull request.
  • This Challenge will be run in accordance with these Official Rules ("Rules").

  1. Competition Description

    The general goal of the IGLU challenge is to facilitate research in the area of Human-AI collaboration through the natural language. The aim of this edition is to build interactive agents that learn to solve a task while provided with grounded natural language instructions in a collaborative environment. By interactive agent, we mean agents that can follow instructions in natural language and ask for clarification when needed. Ultimately, the agent should be able to quickly adapt newly acquired skills, just like humans do in collaborative interaction with each other. Despite all the recent progress in interactive problem solving, the task of interactive learning is far from solved. To facilitate research in this direction, we present IGLU – a voxel-based collaborative environment and set of tasks to study interactive grounded language understanding and learning.

  2. Sponsors

    The Competition is organized by University of Amsterdam referred to as “Organizers'' collectively from here on. Third parties, such as Micrsofot and Amazon, provided sponsorship to cover running costs, prizes and compute grants. These third parties will be known as “Sponsors”.

  3. Organizers Admins

    “Organizers Admins” are any companies or organizations authorized by Organizers to aid them with the administration, sponsorship or execution of this Challenge including but not limited to AICrowd SA.

  4. Entry The Entry in this competition refers to a git repository on [gitlab.aicrowd.com](AICrowd SA) which includes:

  • Source code for running the submitted model;
  • Any data locally required for running this source code, e.g., model weights, additional datasources; and
  • Specifications of the software runtime

  1. Eligibility

    You may enter this challenge individually or as a team of two or more members ("Team"). You are eligible to enter this Challenge if you (and each member of your Team) meet all of the following requirements as of the time and date of entry:

    • You are an individual;
    • You are 16 years of age or older but in no event less than the age of majority in your place of residence;
    • You have internet access, an email account, and access to a personal computer; and
    • You can be member of only one team

    The organizers will not be able to transfer the prize money to accounts of participants who reside in any of the following countries or regions. (Please note that residents of these countries or regions are still allowed to participate in the Challenge, but no prize money will be available.)

    • The Crimea region of Ukraine
    • Cuba
    • Iran
    • North Korea
    • Sudan
    • Syria
    • Quebec, Canada
    • Brazil
    • Italy

    Please Note: it is entirely your responsibility to review and understand your employer's and country's policies about your eligibility to participate in this Challenge. If you participate in violation of your employer's or country's policies, you and your entry may be disqualified from the Challenge. Dr. Derk Gym disclaims any and all liability or responsibility with respect to disputes arising between an employer and such employer's employee or between a country and its resident in relation to the Challenge.

  2. Competition Start and End Dates

    • Start date: 25th July 2022, 12:00 UTC
    • End date: 25th October 2022, 12:00 UTC

  3. Prizes

    • The competition has prizes of:

      • RL track:
        • 1st place: 4,000 USD
        • 2nd place: 1,500 USD
        • 3rd place: 500 USD
      • NLP track:
        • 1st place: 4,000 USD
        • 2nd place: 1,500 USD
        • 3rd place: 500 USD
      • Research prizes: 3,000 USD

    • Research prizes will be given out at the organizers’ discretion to submissions that the organizers think made a particularly interesting or valuable research contribution. If participants wish to be considered for a research prize, they should include a detailed description of their approach in the README for their submission. Organizers will award from 2 to 5 research prizes in total.

    • To be eligible for the prizes, participants will have to release the code to their solutions under an Open Source Foundation approved license of their choice. The submitted code is expected to be reproducible and should produce the same score on the leaderboard.

    • The winning participants will have to provide a valid and unexpired national ID card or passport which AIcrowd will use only for the purpose of verifying the individual's identity in case of transferring the cash prize and for internal record keeping.

    • Any and all prize(s) is(are) non transferable. All taxes, fees and expenses associated with participation in the Challenge or receipt and use of a prize are the sole responsibility of the Prize Winner(s). No substitution of prize or transfer/assignment of prize to others or request for the cash equivalent by winners is permitted. Acceptance of prize constitutes permission for AIcrowd to use winner’s name and entry for purposes of advertising and trade without further compensation unless prohibited by law.

    • The participation of AIcrowd employees and its affiliates, parents, agents, representatives, advertising and promotional agencies and members of the immediate family (parents, children or siblings) of these employees or any person with whom they are domiciled, are permitted to participate in the challenge but will not be eligible to win the cash prize.

  4. Is the Entry an Eligible Entry?

    To be eligible to be considered for a prize, as solely determined by the Organizers:

    The Entry MUST:

    • be compatible with the official submission format;
    • Be self-contained and function without a dependency on any external services and network access
    • be in English;
    • be the Team’s own original work;
    • not have been submitted previously in any promotion of any kind;
    • not contain material or content that: is inappropriate, indecent, obscene, offensive, sexually explicit, pornographic, hateful, tortious, defamatory, or slanderous or libelous; or promotes bigotry, racism, hatred or harm against any group or individual or promotes discrimination based on race, gender, ethnicity, religion, nationality, disability, sexual orientation, or age; or promotes alcohol, illegal drugs, or tobacco; or violates or infringes another’s rights, including but not limited to rights of privacy, publicity, or their intellectual property rights; or is inconsistent with the message, brand, or image of Organizers, is unlawful; or is in violation of or contrary to the laws or regulations of any jurisdiction in which the Entry is created; and

    The Team members MUST:

    • designate one person as the team leader who will be solely responsible for receiving communications from and communicating with Sponsor;
    • ensure the Team has obtained any and all consents, approvals, or licenses required for submission of the Entry;
    • obtain any consents necessary from all members of the Team with respect to the sharing of such member’s personal information as outlined herein;
    • obtain the agreement of all members of the Team to these Rules;
    • not generate the Entry by any means which violate these Rules, - the Organizers Terms of Service or the Organizers Privacy Policy;
    • not engage in false, fraudulent, or deceptive acts at any phase during participation in the Challenge; and
    • not tamper or abuse any aspect of this Challenge. In particular, it is prohibited to exploit the properties of the training environment by hard-following a copy of environment or obtain any parts of internal state of the game other that that are provided by the evaluation platform. It is allowed to predict internal game state given observations from the evaluation platform by means of machine learning models.

    The Team members MUST:

    • ensure the Team has obtained any and all consents, approvals, or licenses required for submission of the Entry;
    • obtain any consents necessary from all members of the Team with - respect to the sharing of such member’s personal information as outlined herein;
    • obtain the agreement of all members of the Team to these Rules;
    • not generate the Entry by any means which violate these Rules, - the Sponsor’s Terms of Service or the Sponsor’s Privacy Policy;
    • not engage in false, fraudulent, or deceptive acts at any phase during participation in the Challenge; and
    • not tamper or abuse any aspect of this Challenge.

  5. Disqualification

    If you, any Team member, or the Entry is found to be ineligible for any reason, including but not limited to conflicts within Teams and noncompliance with these Rules, Organizers and Organizers Affiliates reserve the right to disqualify the Entry and/or you and/or your Team members from this Challenge and any other contest or promotional activity sponsored or administered in any way by the Organizers. A participant is not allowed to create more than one account to participate in the challenge. Violating this will result in disqualification from the challenge. Participants should not attempt to get around the limited number of submissions during the test phase by entering several teams into the competition. Participants should only be associated with one Entry. If two teams have overlap in number of participants, or if the organizers deem two entries to be effectively similar modulo small changes, they reserve the right to disqualify both teams. If a participating team does not receive the Prizes for any of the above-mentioned reasons, the prizes will be offered to the next eligible team on the final leaderboard.

  6. How may the Entry potentially be used?

    The Entry may be used in a few different ways. Organizers do not claim to own your Team’s Entry, however, by submitting the Entry you and each member of your Team:

    hereby grants to Organizers a non-exclusive, irrevocable, royalty-free, world-wide right and license to review and analyze the Entry in relation to this Challenge; hereby grants to Organizers a non-exclusive, irrevocable, royalty-free, world-wide right and license to trajectory data generated from evaluation of the Entry, to be used in a report submitted to NeurIPS after the competition has ended, and possibly released as an open-source dataset; agrees that each member will execute any necessary paperwork for Organizers and Organizers Admins to use the rights and licenses granted hereunder; acknowledges and agrees that the Team will not be compensated and may not be credited (at Organizer’s sole discretion) for the use of the Entry as described in these Rules; acknowledges that the Organizers may have developed or commissioned materials similar to the Entry and waive any claims resulting from any similarities to the Entry; understand and acknowledge that, subject to provision of Prizes, Organizers are not obligated to use the Entry in any way, even if the Entry is selected as a winning Entry.

    Personal data you submit in relation to this Challenge will be used by Organizers in accordance with Section 20 of these Rules.

  7. How will Winners be Selected and Notified?

Entries will be judged via an algorithm that will generate a score based upon which Entries will be ranked and such ranking will be displayed on the AIcrowd Site’s Challenge specific leaderboard (“Leaderboard”). The algorithm will rank your Entry as follows:

To evaluate agent, we run several environment episodes for each subtask from the hidden part of the IGLU dataset. Each subtask initializes the world with some starting grid and sets and some target grid as target. The metric used is the F1 score where the ground truth is the signed difference between target structure and starting structure and where the prediction is the snapshot of the building zone at the end of the episode. The episode terminates either when the structure is completed, or when the time limit has been reached. We also let the agent decide when to end an episode (as a separate action). For each task in evaluation set, we run a specified number of episodes and calculate weighted average of task F1 scores, where weights are equal to the total number of blocks to add or remove.

  1. Allowed Entries

We allow Entries that represent either trained machine learning models, heuristic solutions, or combinations of these two. In RL track, Participants are allowed to choose either of two action spaces: walking or flying, these action spaces will be used in the evaluation of their Entries.

Submissions will be evaluated in IGLUGridworld-v0 gym environment, where only image, dialog, last instruction, compass, and inventory are exposed. The use of any other fields or other environment info is prohibited during the evaluation. Any information, including the agent position and the grid state, is allowed for use in training machine learning models.

The trained machine learning models can use either a public dataset from the Collaborative Dialog in Minecraft, public part of IGLU multiturn dataset, or public part of IGLU single-turn dataset. Other sources of data are allowed for the use, however, they should also be publicly available.

  1. When will prizes be awarded?

The prizes will be awarded within a commercially reasonable time frame. All members of a Team may be required to complete and sign additional documentation, such as non-disclosure agreements, representations and warranties, liability and publicity releases (unless prohibited by applicable law), tax documents, or other similar documentation in order for the potentially winning team to claim the prize. For Teams in excess of one member, the prize will be awarded to the Team Lead. Organizers will in no way be involved in any dispute with respect to receipt of a prize by any other members of a Team.

Only prizes claimed in accordance to these Rules will be awarded.

  1. Your Personal Data and Privacy

    Organizers may use cookies and/or collect IP addresses for the purpose of implementing or exercising its rights or obligations under the Rules, for information purposes, identifying your location, including without limitation for the purpose of redirecting you to the appropriate geographic website, if applicable, or for any other lawful purpose in accordance with the AIcrowd Privacy Policy.

    Organizers may use the personal data you provide via your participation in this Challenge:

    • to contact you in relation to the Challenge;
    • to confirm the details of your Entry;
    • to administer and execute this Challenge, including sharing it with Organizer Admins;
    • at Organizers’ discretion, to credit you and/or your Team for the Entry, identify you and/or your Team as a Winner, or other similar notice; and
    • as otherwise noted in these Rules or as necessary for Organizers to meet their obligations under these Rules or applicable law.

    Organizers only require name and email address to be submitted for you to participate in this Challenge for its uses as outlined in this Section 19. Please read the AIcrowd Terms and Conditions, Participation Terms carefully to understand how your data may be used by AIcrowd SA.

  2. Additional Terms and Conditions

    • The organizer reserves the right to amend the Rules and timeline without giving prior notification or any reasons thereof.
    • In case of conflicts, the decision of the Organizers will be final and binding.
    • Violation of the rules or other unfair activity may result in disqualification.
    • By participating in the Challenge, each Participant agrees that: (1) any and all disputes, claims, and causes of action arising out of or in connection with the Challenge, or any prize awarded, shall be resolved individually without resort to any form of class action; and under no circumstances will a Participant be permitted to obtain any award for, and each Participant hereby waives all rights to claim, punitive, incidental or consequential damages and any and all rights to have damages multiplied or otherwise increased and any other damages.
    • If there is a conflict between these Official rules and the AIcrowd Terms of use, the Aicrowd Terms of use will govern.