MEDIQA 2019 - Recognizing Question Entailment (RQE)

The MEDIQA challenge is an ACL-BioNLP 2019 shared task aiming to attract further research efforts in Natural Language Inference (NLI), Recognizing Question Entailment (RQE), and their applications in medical Question Answering (QA).

Recognizing Question Entailment Task (RQE)

The objective of this task is to identify entailment between two questions in the context of QA. We use the following definition of question entailment: “a question A entails a question B if every answer to B is also a complete or partial answer to A” [1]

[1] A. Ben Abacha & D. Demner-Fushman. “Recognizing Question Entailment for Medical Question Answering”. AMIA 2016. Link

Datasets

Training set: RQE_Train_8588_AMIA2016.xml ( RQE_Data_AMIA2016

You can download the validation and test datasets in the Resources Section.

– Please note that the validation set is the same as AMIA2016_302_pairs_Test_Set .

– Training, validation and test sets are also available here: https://github.com/abachaa/MEDIQA2019/tree/master/MEDIQA_Task2_RQE

Timeline

• February 28, 2019: Release of the validation set for the RQE task.
• April 15, 2019: Release of the test sets for the 3 tasks.
• April 30, 2019: Run submission deadline. Participants’ results will be available on AIcrowd.
• May 15, 2019: Paper submission deadline.
• August 1, 2019: BioNLP workshop, ACL 2019, Florence, Italy.

Evaluation Criteria

For the RQE and NLI tasks:

• The evaluation will be based on Accuracy.
• For the result submission file, we expect a csv file with the following header:
  pair_id,label (cf. ValSet_Baseline.csv )

Rules

1) Each team is allowed to submit a maximum of 5 runs.

2) Please choose a username that represents your team, and update your profile with the following information: First name, Last nam, Affiliation, Address, City, Country.

3) For each run submission, it is mandatory to fill in the submission description field of the submission form with a short description of the methods, tools and resources used for that run.

4) The final results will not be considered official until a working notes paper with the full description of the methods is submitted.

Contact us

• We strongly encourage you to use our mailing list for communications between the participants and the organizers: https://groups.google.com/d/forum/bionlp-mediqa
• In extreme cases, if there are any queries or comments that you would like to make using a private communication channel, then you can send us an email at: asma.benabacha@nih.gov