Activity
Ratings Progression
Challenge Categories
Challenges Entered
Improve RAG with Real-World Benchmarks
Latest submissions
Revolutionise E-Commerce with LLM!
Latest submissions
See Allgraded | 270741 | ||
graded | 270740 | ||
graded | 270655 |
Participant | Rating |
---|---|
mincheolyoon | 0 |
happystat | 0 |
unna97 | 0 |
linchia | 0 |
Karrich | 0 |
gaozhanfire | 0 |
pengbo_wang | 0 |
pp_ | 0 |
pengyue_jia3 | 0 |
GenpengXu | 0 |
xiaopeng_li | 0 |
Participant | Rating |
---|
-
NVIDIA-Merlin Amazon KDD Cup '23: Multilingual Recommendation ChallengeView
-
Team_NVIDIA Amazon KDD Cup 2024: Multi-Task Online Shopping Challenge for LLMsView
Amazon KDD Cup 2024: Multi-Task Online Shopping Ch
All Submissions Are Failing
5 months agoOur teamβs last 6 submissions failed. And when I look at the list of submissions from the other teams in the past 4 hours, all other teams failed too. Is there a problem with AIcrowd server?
Here are the links of our teamβs last two failures [here] and [here]
Can an admin please investigate? Thank you.
Push gitlab and cannot find issue
6 months agoThe same thing has just happened to me. I have have created 5 new tags. They all appear in my GitLab but none appear in my issues.
They are tags submission-200, submission-202, submission-203, submission-204, submission-205. Some code are duplicates of each other because I tried submitting the same thing twice without success.
All Submissions "Waiting In Queue" for 12 Hours
6 months agoFYI, all submissions (from all teams) have been βwaiting in queueβ for the past 12 hours. Perhaps an admin can investigate. Thanks.
Submission stuck on "evaluation initiated"
6 months agoThe following two submissions [here] and [here] are stuck with label βevaluation initiatedβ even though they have failed.
Can an admin switch the GitLab label to failed? Because as is, they are using 2 submission quotas. Thanks.
Submission Failed - Please Tell Us When Submission Works Again
6 months agoYes, this is not fixed. I just submitted and got
Submission failed : Failed to communicate with the grader. Please resubmit again in a few hours if this issue persists..
The GitLab issue is [here]
For the past 2 days, no team has been able to submit to track 5.
Please fix this issue and let us know when it is fixed. Thank you
Submissions fail
6 months agoI am also seeing weird submission behavior today. I posted a discussion describing the errors I have been seeing today [here]
Submission Failed - Please Tell Us When Submission Works Again
6 months agoHi, for the past 4 hours, I have been receiving " Submission failed : Failed to communicate with the grader. Please resubmit again in a few hours if this issue persists..
" when submitting to track 5. An example GitLab issue (for admins to review) is [here].
I have tried 3 times and received 3 βfailedβ submissions. I do not want to try anymore because I do not want to use up my failed submission quota. Can an admin tell us when submissions are working for track 5 again? Thanks.
Track 2 LB Doesn't Show Retrieval Score
6 months agoHi, Can admins @yilun_jin fix the track 2 leaderboard webpage to show each teamsβ retrieval score? Thank you.
Phase 2 launching!
7 months agoI notice that AIcrowd website says βRound 2: 21 days leftβ which implies that phase 2 ends on June 15th. It this the correct end of phase 2?
Another Frozen Evaluation
7 months agoThank you for fixing our previous frozen evaluation.
We have another evaluation here. The GitLab issue page shows that it failed but the AIcrowd website is still showing that the submission is being evaluated.
As such, we cannot submit again to this track because the AIcrowd website thinks that a submission is in progress. Can an admin @yilun_jin update the AIcrowd website to acknowledge that our submission failed thus allowing us to make a new submission?
Thank you.
Our Evaluation is Frozen
7 months agoHi. Our submission to track 1 [here] has frozen. It shows 97% completed and it appears to be within all time limits. There has been no update for the past 3 hours.
Can an admin @yilun_jin please unfreeze our submission and post the results? Thank you
Are some errors caused by AIcrowd server and not submission code?
8 months agoHi. Can an admin please solve the following problem and provide a score for our last submission?
We just submitted our other code that failed yesterday and this time the AIcrowd GitLab issue says the error is caused by AIcrowd server. The server could not load the sentence-transformers/all-MiniLM-L6-v2
to evaluate the score. Below is copy and paste from reason for failure.
Evaluate Scores: OSError: We couldnβt connect to βhttps://huggingface.coβ to load this file, couldnβt find it in the cached files and it looks like sentence-transformers/all-MiniLM-L6-v2 is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at βhttps://huggingface.co/docs/transformers/installation#offline-modeβ.
Our code successfully predicted all questions within all time limits and then AIcrowd server failed to load sentence transformer to compute our LB score
Are some errors caused by AIcrowd server and not submission code?
8 months agoI just submitted the exact same code to the exact same track that failed earlier today. This time it succeeded. Does this mean that we need to repeatedly submit our failed code to AIcrowd server?
What is causing the inconsistent behavior of AIcrowd server? Do fails that occur because of AIcrowd server count toward are weekly track 1-4 limit of 20 and weekly track 5 limit of 3?
Are some errors caused by AIcrowd server and not submission code?
8 months agoToday it took 10+ hours to receive the result of two submissions that we made. After submitting, we were watching the progress at AIcrowd GitLab issues. The submissions were successfully predicting questions. Then the updates froze and the submissions were labeled βevaluation_initatedβ for the next 8 hours. After 8 hours we saw that the submissions were labeled βfailedβ.
The debug logs do not show any errors. This is the same code that worked before (on a different track) and/or made more progress before (on same track).
Is it possible that the errors and/or inconsistent behavior is caused by AIcrowd servers? Should we just submit the same code again?
Has any team submitted the exact same code twice. And one time the code failed and one time the code was successful?
The maximum number of players per team.
8 months agoThere are currently two teams with 7 members. Do you mean to say βthe maximum is 7β?
We Cannot View Submissions
8 months agoThank you admins for adding the βSubmissionβ button to the individual track 1, 2, 3, 4, 5 webpages.
Can you also add the βSubmissionβ button to the main challenge page? Currently if we go to our profile page and click βSee Allβ referring to see all submission in this competition it does not work. This is because it attempts to use URL:
https://www.aicrowd.com/challenges/amazon-kdd-cup-2024-multi-task-online-shopping-challenge-for-llms/submissions
which is currently disabled by admins.
What Does 2 Submission Per Week Mean?
8 months agoWhat does 2 submissions per week mean? Is this based on a calendar week, like we can only make 2 submissions Monday thru Sunday? Or is this based on a sliding 7 day window, like we can only make maximum 2 submissions per 7 day sliding window?
We Cannot View Submissions
8 months agoAt the top of the track 1 page there are 8 buttons named βOverview, Leaderboard, Notebooks, Discussion, Insights, Resources, Submissions, Rulesβ.
If we click the button βSubmissionβ we see all submissions for track 1.
There is no way to see submissions for tracks 2, 3, 4, 5. Because neither the main page nor track 2 nor track 3 nor track 4 nor track 5 has a βSubmissionβ button.
Can an administrator, add βSubmissionβ button to each track? Thank you!
Why Doesn't Leaderboard Show Ranking Score?
8 months agoI noticed that the leaderboard for each track shows the score for the tasks: Multiple Choice, NER, Retrieval, and Generation. But no leaderboards show the results for Ranking?
Is this on purpose or a bug?
Note for our final evaluation
5 months ago@yilun_jin , Sometimes the exact same code will succeed one time and fail another time. For example we submitted the exact same code [here] and [here]. The first succeeded and the second failed. During re-run what happens if code fails that has previously succeeded? Will the admins run it a second time?
Also, can you tell us why the second link above failed?
When we select our final 2 submissions for each track, should we just select our best scoring submission twice in case it fails the first time it is re-run?