About the Meta Comprehensive RAG Benchmark: KDD Cup 2-524854 category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the Meta Comprehensive RAG Benchmark: KDD Cup 2-35606c category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the Meta Comprehensive RAG Benchmark: KDD Cup 2-f8dd03 category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the Meta Comprehensive RAG Benchmark: KDD Cup 2-290ad7 category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the Meta Comprehensive RAG Benchmark: KDD Cup 2-b8c075 category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the Meta Comprehensive RAG Benchmark: KDD Cup 2-dcb737 category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the Meta Comprehensive RAG Benchmark: KDD Cup 2-68ef6d category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the Meta Comprehensive RAG Benchmark: KDD Cup 2-ac174e category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the Meta Comprehensive RAG Benchmark: KDD Cup 2-dd58fc category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the Meta Comprehensive RAG Benchmark: KDD Cup 2025 category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

About the meta-kdd-cup-24-staging category

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.)

Use the following paragraphs for a longer description, or to establish category guidelines or rules:

• Why should people use this category? What is it for?

• How exactly is this different than the other categories we already have?

• What should topics in this category generally contain?

• Do we need this category? Can we merge with another category, or subcategory?

View post 0

Updates to Task 1 Metrics

@saidinesh_pola I can’t share all the details about the GPT metric, but we do manage it for cases it’s not a valid score. For final submissions, each team will get to select any 2 successful submissions, doesn’t matter if it’s GPU track or not.

View post 1

Updates to Task 1 Metrics

@unnikrishnan.r I don’t see why gpt3.5 would naturally score prompt track submissions higher. Yes it is what is currently occuring but there is no natural reason for it. The metrics were decided based on actual human evaluations done on a blindly selected subset of Round 1 conversations, and gpt3.5 scores were the most correlated with the human evaluations.

View post 0

Updates to Task 1 Metrics

@saidinesh_pola , Yes GPT3.5 score is GPT-3.5-turbo generating a score based on a modified prompt similar to G-eval. It scores every utterance with the conversation history as context. And it generates a score.

View post 0

Updates to Task 1 Metrics

We are updating the metrics used to compute the leaderboards during the challenge.

Noting that the final prizes and standings will be decided based on the outcomes of the human evaluations, we explored how closely the current metrics correlated with overall score of human evaluation results.

After a thorough investigation, we noticed that World Level F1 and 4-gram BLEU did not accurately reflect the performance of the submitted models.

In light of the said observation, and with the motivation to provide a more accurate feedback to the teams about their performance, we have decided to incorporate three new metrics: CPDScore, USEScore, and BERTScore. CPDScore and USEScore have demonstrated superior accuracy compared to the previously used metrics. Moving forward, CPDScore will be the primary metric for leaderboard rankings, while BERTScore will serve as an additional metric for reference due to its widespread use in automatic evaluation benchmarks.

CPDScore is a LLM based metric that uses a prompt similar to G-EVAL. The metric focuses on “Humanness” whose criteria are described in the prompt. For Round 2, it employs GPT-3.5-turbo-0125, while for the final leaderboard we might use a stronger model.
USEScore calculates similarity using the Universal Sentence Encoder.
BERTScore is a commonly used benchmark in the evaluation of automatic metrics.

We have added the scores to all submissions of Round 2 - Task 1.

View post 2

How to know my submission was successful and what is the right way to make a successful submission?

Hi @kcy4 ,

Unfortunately your submission failed, you can see the logs here

Apologies for the confusion in the documentation, will fix them in the starter kit. You should change the model name in models/user_config.py and the function name should be generate_design, please ignore any other conflicting names in the starter kit.

View post 0

Can't open baseline starter kit

Thanks for informing @Camaro , it’s public now.

View post 1

Request for More Comprehensive Error Reports for Recent Submission Failures Task1

Hi @nazlicanto , unfortunately most these seem to be caused by intermittent issue in the OpenAI API while handling large number of API calls. I’ve made changes to try and fix this, have resubmitted your submission 248051 and will keep on eye on it so that it passes.

View post 0

Submissions getting failed

Okay I’ll create it in a few days

View post 0

Submissions getting failed

Hi @saidinesh_pola I’ve replied to got gitlab issue.

View post 0