Loading
Warm-Up Round: 36 hours left Β· Ending 01 Dec 23:55 UTC #nlp #knowledge_graph #language_model #dialogue #llm
8122
125
14
8

Problem Statements

 

πŸ™‹β€β™€οΈ Don't know where to start? Read our Task 1 & Task 2 explainer to get started.

πŸ‘₯ Find teammates πŸ’¬ Share your queries 
πŸ’» Resources: Baseline Training Model πŸ“• Starter-kit  πŸ““ Baseline

Welcome to the Commonsense Persona-grounded Dialogue (CPD) Challenge !

This challenge is an opportunity for researchers and machine learning enthusiasts to test their skills on the challenging tasks of Commonsense Dialogue Response Generation (Task1) and Commonsense Persona Knowledge Linking (Task2) for persona-grounded dialogue.

πŸ•΅οΈ Introduction

Research on dialogue systems has been around for a long time, but thanks to Transformers and Large Language Models (LLM), conversational AI has come a long way in the last five years, becoming more human-like. On the other hand, it is still challenging to collect natural dialogue data for research and to benchmark which models ultimately perform the best because there is no definitive assessment data or metrics, and the comparisons are often within a limited amount of models.

We contribute to the research and development of current state-of-the-art dialogue systems, by crafting high quality human-human dialogues for model testing, and providing a common benchmarking venue by hosting this CPDC 2023 competition.

The competition aims to see the best approach among state-of-the-art participant models on an evaluation dataset of natural conversation. The submitted systems will be evaluated on a new Commonsense Persona-grounded Dialogue dataset. To this end, we first created several persona profiles, similar to ConvAI2, with a natural personality based on a commonsense persona-grounded knowledge graph (PeaCoK†) newly released on ACL 2023, and allowing us to obtain naturally related persona sentences. Furthermore, based on that persona, we created a natural dialogue between two people and prepared a sufficient amount of dialogue data for evaluation.

The Commonsense Persona-grounded Dialogue (CPD) Challenge hosts one track on Commonsense Dialogue Response Generation (Task 1) and one track on Commonsense Persona Knowledge Linking (Task 2). Independent leaderboards are set for the two tracks, each featuring a separate prize pool. In either case, participants may use any learning data. In Task 1, participants will submit dialogue response generation systems. We will evaluate them on the prepared persona-grounded dialogue dataset mentioned above. In Task 2, participants will submit systems linking knowledge to a dialogue. This task is designed in the similar spirit of ComFact, which is released along with the published paper in EMNLP 2022. We will evaluate them by checking if the linking of persona-grounded knowledge can be judged successfully on the persona-grounded dialogue dataset.

† PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives (ACL2023 Outstanding Paper Award)

 

πŸ“‘ The Task

Task 1: Commonsense Dialogue Response Generation
 

πŸ”— Task 1: Commonsense Dialogue Response Generation

Participants will submit dialogue response generation systems. We do not provide a training dataset, and participants may use any datasets which they want to use. We provide a baseline model, which can be tested on the ConvAI2 PERSONA-CHAT dataset, so that you can see what the problem of this task is. We will evaluate submitted systems on the persona-grounded dialogue dataset. The dialogues in the evaluation dataset have persona sentences similar to the PersonaChat dataset, but the number of persona sentences for a person is more than five sentences. The major part of the persona is derived from the PeaCoK knowledge graph.

Task 2: Commonsense Persona Knowledge Linking

πŸ”— Task 2: Commonsense Persona Knowledge Linking

Participants will submit systems for linking knowledge to a dialogue. We don’t provide a training dataset and participants may use any datasets which they want to use. We provide a baseline model, which can be tested on the ComFact benchmark, so that you can see what is the problem of this task. We will evaluate submitted systems by checking if the linking of persona-grounded knowledge (PeaCoK) can be judged successfully on the persona-grounded dialogue dataset. Note that the persona statements are prepared in advance when creating a dialogue and the persona knowledge selected as a candidate for linking are completely independent.

πŸ“… Timeline

The challenge will take place across in 3 Rounds which differ in the evaluation dataset used for ranking the systems.

  • Warm-up Round : 3rd November, 2023
  • Round 1 : 1st December, 2023
  • Round 2 : 15th January, 2024
  • Challenge End: 1st March, 2024

πŸ† Prizes

The prize pool is a total of 35,000 USD divided among the two tracks. Participating teams are eligible to win prizes in multiple leaderboards spread across both the tracks.

Task1: Commonsense Dialogue Response Generation

  • πŸ₯‡ First place 15,000 USD
  • πŸ₯ˆ Second place 7,000 USD
  • πŸ₯‰ Third place 3,000 USD

Task2: Commonsense Persona Knowledge Linking

  • πŸ₯‡ First place 5,000 USD
  • πŸ₯ˆ Second place 3,000 USD
  • πŸ₯‰ Third place 2,000 USD

Please refer to the Challenge Rules for more details about the Open Sourcing criteria for each of the leaderboards to be eligible for the associated prizes.

 

πŸ”— Quick Links

πŸ“– Citing the Dataset

If you are participating in this challenge or using the dataset please consider citing the following paper:

Task1 Dataset: PeaCoK

@inproceedings{gao-etal-2023-peacok,
title = "{P}ea{C}o{K}: Persona Commonsense Knowledge for Consistent and Engaging Narratives",
author = "Gao, Silin and
Borges, Beatriz and
Oh, Soyoung and
Bayazit, Deniz and
Kanno, Saya and
Wakaki, Hiromi and
Mitsufuji, Yuki and
Bosselut, Antoine",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
year = "2023",
pages = "6569--6591",
}

Task2 Dataset: ComFact

@inproceedings{gao-etal-2022-comfact,
title = "{C}om{F}act: A Benchmark for Linking Contextual Commonsense Knowledge",
author = "Gao, Silin and
Hwang, Jena D. and
Kanno, Saya and
Wakaki, Hiromi and
Mitsufuji, Yuki and
Bosselut, Antoine",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
year = "2022",
pages = "1656--1675",
}

πŸ“± Challenge Organizing Committee

  • Hiromi Wakaki (Sony)

  • Antoine Bosselut (EPFL)

  • Silin Gao (EPFL)

  • Yuki Mitsufuji (Sony)

  • Mengjie Zhao (Sony)

  • Yukiko Nishimura (Sony)

  • Yoshinori Maeda (Sony)

Have queries, feedback or looking for teammates, drop a message on AIcrowd Community. Don’t forget to hop onto the Discord channel to collaborate with fellow participants & connect directly with the organisers. Share your thoughts, spark collaborations and get your queries addressed promptly.

Participants

Leaderboard

01 mrd 2.000
01 mrd 2.000