(April 1st, 2024) 🚀 Submissions are now open.
And our 🚀Starter Kit is available to help you quickly onboard and make the first submission.

A RAG QA system takes a question Q as input and outputs an answer A; the answer is generated by LLMs according to information retrieved from external sources, or directly from the knowledge internalized in the model. The answer should provide useful information to answer the question, without adding any hallucination or harmful content such as profanity.

TASK #2: KNOWLEDGE GRAPH AND WEB AUGMENTATION

Taks #2 provides mock APIs to access information from underlying mock Knowledge Graphs (KGs). These mock KGs contain structured data relevant to the questions; however, the answers to the questions may or may not exist within the mock KGs. The mock APIs accept input parameters, often parsed from the question, and provide structured data from the mock KGs to support answer generation.

This task evaluates how effectively the RAG system:

• queries structured data sources
• synthesizes information from various sources.

To download the Mock API, please see: https://www.aicrowd.com/challenges/meta-comprehensive-rag-benchmark-kdd-cup-2024/problems/knowledge-graph-and-web-retrieval/dataset_files.

The Question Answering data used in Task #2 is the same as the one in Task #1. To download the data, please see: https://www.aicrowd.com/challenges/meta-comprehensive-rag-benchmark-kdd-cup-2024/problems/meta-kdd-cup-24-crag-retrieval-summarization/dataset_files

To know more about the CRAG challenge, please see: https://www.aicrowd.com/challenges/meta-comprehensive-rag-benchmark-kdd-cup-2024.