MEDIQA 2019 - Natural Language Inference (NLI)

The MEDIQA challenge is an ACL-BioNLP 2019 shared task aiming to attract further research efforts in Natural Language Inference (NLI), Recognizing Question Entailment (RQE), and their applications in medical Question Answering (QA).

Natural Language Inference Task (NLI)

The objective of this task is to identify three inference relations between two sentences: Entailment, Neutral and Contradiction.

Datasets

Training set: https://jgc128.github.io/mednli/ Participants will have to obtain access to MIMIC in order to access MedNLI and the test set.

Timeline

• April 15, 2019: Release of the test sets for the 3 tasks.
• April 30, 2019: Run submission deadline. Participants’ results will be available on AIcrowd.
• May 15, 2019: Paper submission deadline.
• August 1, 2019: BioNLP workshop, ACL 2019, Florence, Italy.

Evaluation criteria

For the NLI and RQE tasks:

• The evaluation will be based on Accuracy.
• For the result submission file, we expect a csv file with the following header: pair_id,label (cf. ValSet_Baseline.csv of the RQE task)

Rules

1) Each team is allowed to submit a maximum of 5 runs.

2) Please choose a username that represents your team, and update your profile with the following information: First name, Last nam, Affiliation, Address, City, Country.

3) For each run submission, it is mandatory to fill in the submission description field of the submission form with a short description of the methods, tools and resources used for that run.

4) The final results will not be considered official until a working notes paper with the full description of the methods is submitted.

Contact us

• We strongly encourage you to use our mailing list for communications between the participants and the organizers: https://groups.google.com/d/forum/bionlp-mediqa
• In extreme cases, if there are any queries or comments that you would like to make using a private communication channel, then you can send us an email at: asma.benabacha@nih.gov