Winner’s Solution Overview: KDD Cup 2024 - Team NVIDIA

Team NVIDIA, a group of data scientists and technologists, brought diverse skills to the KDD Cup 2024. Key members include Gilberto, a former #1 ranked Kaggle competitor with a background in Electrical Engineering; Chris, a Ph.D. holder in computational science and mathematics with experience across various professions; Benedikt Schifferer, a manager of Applied Research with expertise in recommender systems and LLMs; Ivan Sorokin, a Senior LLM Technologist; Ahmet Erdem, a Kaggle Grandmaster and open-source contributor; and Simon, a senior LLM technologist specializing in deep learning applications in computer vision and NLP.

Winning Strategy:

Team NVIDIA’s strategy for the KDD Cup 2024 involved the deployment of five fine-tuned Qwen2-72B LLM models, one for each of the competition’s tracks, leveraging cutting-edge techniques and substantial computational resources:

1. Data Transformation and Model Training:

• The team transformed data from six public datasets, including Amazon-M2 and MMLU, into 500k question-answer pairs across 40 tasks and 5 task types.

• They fine-tuned multiple Qwen2-72B models using QLoRA on NVIDIA’s powerful 8xA100 80GB GPUs, employing techniques like DeepSpeed and Axolotl for efficiency.

2. Model Optimization and Fine-Tuning:

• Fine-tuning involved adjusting LoRA parameters and experimenting with different weights for model adapters to optimize performance across various tasks.

• The models were trained with specific prompts tailored to simulate an online shopping assistant, enhancing task-specific performance.

3. Quantization and Inference Optimization:

• To meet the competition’s stringent hardware limitations and inference time constraints, Team NVIDIA employed 4-bit AWQ quantization and batch inference strategies using software vLLM, significantly reducing the model’s memory footprint.

• During inference, logits processors were added to the model’s predictions to ensure output accuracy, particularly in handling structured responses like numbers and commas.

4. Ensemble Techniques:

• The final submissions for each track involved sophisticated ensembles of base models and multiple LoRA adapters, fine-tuned to enhance the accuracy and robustness of the solutions.

Impact and Contributions:

Team NVIDIA’s comprehensive approach showcased their technical prowess and ability to innovate within constraints, leading to their first-place victory in all five competition tracks. Their work demonstrates the powerful capabilities of LLMs in handling diverse and complex real-world NLP tasks, particularly in a competitive setting with limited hardware resources.

View post 0

Change Team Name Request

Your team name has been updated to BMI_DLUT.

View post 0

Change Team Name Request

Hello, Your team name has been updated to AML_LabCityU

View post 0

Winner’s Solution Overview: Meta KDD Cup 2024 - Team db3

Team db3 consists of third-year PhD students from Peking University, mentored by Professor Gao Jun. Their research focuses on data mining for structured data, including community search, graph alignment, and table data integration. With a strong background in leveraging data mining for extracting insights in fields such as social networks and bioinformatics, the team’s expertise is especially pertinent to their work with large language models (LLMs) and Retrieval-Augmented Generation (RAG) systems.

Winning Strategy:

Team db3 excelled in all three tasks of the Meta KDD Cup 2024, securing first place with scores of 28.4%, 42.7%, and 47.8%, respectively. Their approach to creating a state-of-the-art RAG system involved several sophisticated techniques:

1. Task 1 - Web Retrieval and Answering:

The team developed a framework utilizing a combination of retrievers and rerankers to process and rank text chunks extracted from web pages. They employed BeautifulSoup for HTML parsing and LangChain for text splitting, alongside the bge-base-en-v1.5 retriever and a complementary reranker model to refine the selection of relevant text chunks.

2. Tasks 2 and 3 - Integration of Structured Data:

For the subsequent tasks, the team focused on integrating data from both web sources and mock Knowledge Graphs. They implemented a regularized API set and an API generation method using a tuned LLM. A Parent-Child Chunk Retriever system was crucial in managing the retrieval process, with the reranker further refining data selection to enhance accuracy and relevance.

3. Addressing Hallucination in LLMs:

A significant aspect of their strategy was tuning the models to reduce inaccuracies and improve groundedness in responses, thereby addressing the issue of hallucination commonly associated with LLMs.

Impact and Research Alignment:

Team db3’s method demonstrates the potential of RAG systems in providing accurate and reliable answers by effectively integrating and processing external information. Their strategy aligns seamlessly with their research interests in data mining and structured data analysis, particularly their focus on knowledge graphs, which represent information in a structured format essential for various applications like semantic search and intelligent personal assistants.

View post 0

[Issue resolved] We were asked to submit the revision but revision functionality is disabled

The deadline is now updated on OpenReview too. Please try to make a submission again.

View post 0

Final Evaluation Process & Team Scores

There are no manual scores, only auto-scores.

View post 0

🏆 Meta CRAG Challenge 2024 Winners Announcement

August 02, 2024 11:59PM UTC-0, updated in the post.

View post 0

🏆 Meta CRAG Challenge 2024 Winners Announcement

Hello Participants,

We are excited to announce the winners of the Meta CRAG Challenge 2024. We appreciate your patience as we completed the due diligence process for the annotations.

We want to thank all participants for their efforts and contributions to improve RAG solutions. Over the last four months, the Meta CRAG Challenge has seen over 2000 participants from various countries, making over 5500 submissions. Below are the winners for each task. The final evaluation process can be found here: Final Evaluation Process & Team Scores

Task 1: Retrieval Summarization

• Team db3
• Team md_dh
• Team ElectricSheep
• Simple_w_condition Question: Team dummy_model
• Set Question: Team dummy_model
• Comparison Question: Team dRAGonRAnGers
• Aggregation Question: Team dummy_model
• Multi-hop Question: Team bumblebee7
• Post-processing Question: Team dRAGonRAnGers
• False Premise Question: Team ETSLab

Task 2: Knowledge Graph and Web Retrieval

• Team db3
• Team APEX
• Team md_dh
• Simple_w_condition Question: Team ElectricSheep
• Set Question: Team ElectricSheep
• Comparison Question: Team dRAGonRAnGers
• Aggregation Question: Team ElectricSheep
• Multi-hop Question: Team ElectricSheep
• Post-processing Question: Team ElectricSheep
• False Premise Question: Team Future

Task 3: End-to-End Retrieval-Augmented Generation

• Team db3
• Team APEX
• Team vslyu-team
• Simple_w_condition Question: Team StarTeam
• Set Question: Team md_dh
• Comparison Question: Team dRAGonRAnGers
• Aggregation Question: Team md_dh
• Multi-hop Question: Team ETSLab
• Post-processing Question: Team md_dh
• False Premise Question: Team Riviera4

We welcome all teams to submit a technical report to the 2024 KDD Cup RAG Workshop. The report should follow the KDD submission format.

• Submission link: OpenReview
• Submission deadline: 11:59 UTC, Aug 2, 2024
• Submission format: Submissions are limited to 8 pages (excluding references), must be in PDF, and use the ACM Conference Proceeding template (two-column format). For more details, please refer to: KDD 2024 ADS Track Call for Papers

We extend our sincere gratitude to all participants who contributed to this event!

All the best,
Meta & AIcrowd

View post 2

Final Evaluation Process & Team Scores

Hello Participants,

We want to thank all participants for their efforts and contributions to improving RAG systems. This post provides information about the final evaluation process and shares the final scores.

Manual annotation:

For stage 2 in this challenge, we first conducted auto-evaluation for all the teams that provided submission numbers in Phase 2. We then selected the top 15 teams’ submissions according to the auto-eval scores for manual evaluation. The final scores are determined by the scores calculated from the manual grading labels. Automatic and manual evaluation details can be found in the paper.

Weighting:

We applied traffic weights to the questions to understand the solutions in real-world use cases. The traffic weights come from a real QA use case and are generated as follows. Within each domain, we first clustered the questions into question types with the exact definition of the CRAG questions. Then, we derived the weights for each type based on aggregated data reflective of user interactions. We applied the weight to each CRAG question to bridge the result to reflect user experience and reported the macro average scores across all domains (i.e., giving the same weight to all domains).

Code validation

We conducted a code review for the winning solutions to ensure the validity of the codes. For example, we checked whether there was any prompt attack to mislead the auto-evaluation and whether the solutions used too many hard-coded answers.

The scores from the winning teams are listed below.

The scores from the winning teams are listed below.

We extend our sincere gratitude to all participants who contributed to this event!

All the best,

Meta & AIcrowd

View post 4

Winner’s Solution Overview: Commonsense Persona-Grounded Challenge 2024

Task 1: Commonsense Dialogue Response Generation | Kaihua Ni

Kaihua Ni, an alumnus of the University of Leeds with a major in Artificial Intelligence, has extensive experience in AI and deep learning algorithms, having worked at major companies such as Augmentum and CareerBuilder. His expertise lies in natural language processing and the nuances of human conversation.

Winning Strategy:

Kaihua’s strategy for the Commonsense Persona-Grounded Challenge 2024 was rooted in a two-pronged approach focusing on fine-tuning a large language model (LLM) and expert, prompt engineering:

1. Fine-Tuning the LLM:

Utilizing transfer learning techniques, Kaihua adapted the pre-existing parameters of the LLM to the specific conversational style and knowledge domain of the persona being emulated. The model was trained on a curated dataset consisting of dialogues, written works, and other textual representations of the persona, significantly enhancing its ability to mimic the persona’s syntactic and semantic patterns.

2. Prompt Engineering:

Kaihua crafted optimized prompts that encapsulated the context of the conversation while embedding subtle cues that aligned with the personal characteristics of the persona. This optimization steered the model to generate responses that were contextually relevant and infused with the persona’s idiosyncratic communication style.

3. Advanced NLP Techniques:

Attention mechanisms and context window adjustments were employed to maintain coherence and context retention across multi-turn dialogues. Kaihua also developed a custom evaluation metric aligned with the challenge’s criteria to iteratively assess and refine the model’s performance.

4. Ethical Considerations:

Ethical aspects were critically addressed, ensuring that the AI’s mimicry respected the privacy and dignity of the individual. Kaihua implemented strict boundaries on the use of personal information and incorporated safeguards against generating inappropriate or harmful content.

Task 2: Commonsense Persona Knowledge Linking | Iris Lin

Iris Lin, an experienced Machine Learning Engineer, specializes in developing and optimizing large-scale machine learning models. With a robust background in deep learning, natural language processing, and recommendation systems, Iris is proficient in Scala, Python, Java, Spark, PyTorch, TensorFlow, and various machine learning libraries.

Winning Strategy:
Team biu_biu, managed by Iris Lin, employed a comprehensive approach to developing a commonsense persona knowledge linker for CPDC Task 2, leveraging cutting-edge techniques and tools:

1. Baseline Evaluation with ComFact Model:
The team initiated the project by testing the provided ComFact baseline model on a hidden test to pinpoint its limitations and identify areas for improvement in linking persona commonsense facts to dialogue contexts.

2. Dataset Curation and Enhancement:
The team merged the Conv2 and Peacock datasets to refine the training process, focusing on conversations that incorporated persona knowledge. This ensured the model was trained on highly relevant data. They also employed GPT-3.5-Turbo to create a synthetic dataset by labelling persona facts in 20,000 conversations, providing a diverse and extensive training foundation.

3. Model Fine-Tuning with Deberta-V3:
The team fine-tuned the Deberta-V3 model, renowned for its effectiveness in various NLP tasks. A rigorous hyperparameter search was conducted to optimize performance, particularly in capturing the nuances of persona knowledge linking.

4. Comprehensive Model Evaluation:
The model was rigorously evaluated under two settings: predicting head and tail facts separately and simultaneously. This dual-testing strategy allowed the team to thoroughly assess the model’s versatility and pinpoint any potential enhancements.

View post 0

💬 Feedback & Suggestions

View post 0

📹 Welcome to Sounding Video Generation Challenge 2024

View post 0

👥 Looking for teammates?

View post 0

👥 Looking for teammates?

View post 0

💬 Feedback & Suggestions

View post 0

📹 Welcome to Sounding Video Generation Challenge 2024

View post 0

📹 Welcome to Sounding Video Generation Challenge 2024

The Sounding Video Generation (SVG) Challenge 2024 is a competition to create AI models that make videos where the visuals match perfectly with sounds, like a dog barking in sync with the video. Participants will work to improve how well sounds and scenes align, with prizes for the best results.

ADVANCING AUDIO-VISUAL SYNCHRONISATION

Join the Sounding Video Generation (SVG) Challenge 2024, a groundbreaking competition at the intersection of video generation and audio-visual synchronisation. This innovative challenge invites participants to build state-of-the-art AI models that generate perfectly aligned and contextually accurate videos guided by audio.

Use your machine learning expertise to advance the frontier of audio-visual synchronisation, transforming datasets into dynamic, synchronised videos. With the right code and creativity, you’ll contribute to a rapidly evolving field that’s set to redefine how we generate and experience multi-modal content.

The Task
Develop models that generate videos with synchronised and contextually relevant audio in two specialised tracks:

• Temporal Alignment Track: Create videos where the audio is perfectly synchronised with the video content in time (e.g., a dog barking exactly when seen in the video). Find the starter kit here.
• Spatial Alignment Track: Develop models that produce videos with spatially aligned audio, creating a real sense of direction and space. Find the starter kit here.

Both tracks aim to push the boundaries of multi-modal AI, offering a unique platform to benchmark cutting-edge solutions in this underexplored domain.

Check out the starter-kits for Temporal Alignment Track and Spatial Alignment Track.
Download the resources for the challenge Temporal Alignment Track and Spatial Alignment Track.

Timeline

• Warmup Round: 29th Oct 2024
• Phase I: 2nd Dec 2024
• Phase II: 3rd Jan 2025
• Challenge End: 25th Mar 2025

Prizes
The challenge boasts a prize pool of USD 35,000 split across both tracks. The top three teams or participants in each track will be rewarded as follows:

• Track 1: Temporal Alignment ($17,500)
  • First place: USD 10,000
  • Second place: USD 5,000
  • Third place: USD 2,500
• Track 2: Spatial Alignment ($17,500)
  • First place: USD 10,000
  • Second place: USD 5,000
  • Third place: USD 2,500

Join a Thriving Community
Collaborate with like-minded researchers, practitioners, and AI enthusiasts eager to share ideas, team up, and drive innovation in this captivating challenge.

Sign up now for the SVG Challenge 2024 and start building your models in the warm-up round!

View post 1

💬 Feedback & Suggestions

We are constantly trying to improve this challenge for you and would appreciate any feedback you might have!

Please reply to this thread with your suggestions and feedback on making the challenge better for you!

• What have been your major pain points so far?
• What would you like to see improved?

All The Best!

View post 0

👥 Looking for teammates?

Competing is more fun with a team!

Introduce yourself here, and find others who are looking to team up!

Format:

• A short introduction about you and your background.
• What brings you to this challenge?
• Some ideas you wish to explore as a part of this challenge?

All The Best!

View post 0

About the Sounding Video Generation (SVG) Challenge 2024 category

View post 0