Are some errors caused by AIcrowd server and not submission code?

Hi. Can an admin please solve the following problem and provide a score for our last submission?

We just submitted our other code that failed yesterday and this time the AIcrowd GitLab issue says the error is caused by AIcrowd server. The server could not load the sentence-transformers/all-MiniLM-L6-v2 to evaluate the score. Below is copy and paste from reason for failure.

Evaluate Scores: OSError: We couldn’t connect to ‘https://huggingface.co’ to load this file, couldn’t find it in the cached files and it looks like sentence-transformers/all-MiniLM-L6-v2 is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at ‘https://huggingface.co/docs/transformers/installation#offline-mode’.

Our code successfully predicted all questions within all time limits and then AIcrowd server failed to load sentence transformer to compute our LB score

View post 0

Are some errors caused by AIcrowd server and not submission code?

I just submitted the exact same code to the exact same track that failed earlier today. This time it succeeded. Does this mean that we need to repeatedly submit our failed code to AIcrowd server?

What is causing the inconsistent behavior of AIcrowd server? Do fails that occur because of AIcrowd server count toward are weekly track 1-4 limit of 20 and weekly track 5 limit of 3?

View post 0

Are some errors caused by AIcrowd server and not submission code?

Today it took 10+ hours to receive the result of two submissions that we made. After submitting, we were watching the progress at AIcrowd GitLab issues. The submissions were successfully predicting questions. Then the updates froze and the submissions were labeled “evaluation_initated” for the next 8 hours. After 8 hours we saw that the submissions were labeled “failed”.

The debug logs do not show any errors. This is the same code that worked before (on a different track) and/or made more progress before (on same track).

Is it possible that the errors and/or inconsistent behavior is caused by AIcrowd servers? Should we just submit the same code again?

Has any team submitted the exact same code twice. And one time the code failed and one time the code was successful?

View post 0

The maximum number of players per team.

There are currently two teams with 7 members. Do you mean to say “the maximum is 7”?

View post 0

We Cannot View Submissions

Thank you admins for adding the “Submission” button to the individual track 1, 2, 3, 4, 5 webpages.

Can you also add the “Submission” button to the main challenge page? Currently if we go to our profile page and click “See All” referring to see all submission in this competition it does not work. This is because it attempts to use URL:
https://www.aicrowd.com/challenges/amazon-kdd-cup-2024-multi-task-online-shopping-challenge-for-llms/submissions
which is currently disabled by admins.

View post 0

What Does 2 Submission Per Week Mean?

What does 2 submissions per week mean? Is this based on a calendar week, like we can only make 2 submissions Monday thru Sunday? Or is this based on a sliding 7 day window, like we can only make maximum 2 submissions per 7 day sliding window?

View post 0

We Cannot View Submissions

At the top of the track 1 page there are 8 buttons named “Overview, Leaderboard, Notebooks, Discussion, Insights, Resources, Submissions, Rules”.

If we click the button “Submission” we see all submissions for track 1.

There is no way to see submissions for tracks 2, 3, 4, 5. Because neither the main page nor track 2 nor track 3 nor track 4 nor track 5 has a “Submission” button.

Can an administrator, add “Submission” button to each track? Thank you!

View post 0

Why Doesn't Leaderboard Show Ranking Score?

I noticed that the leaderboard for each track shows the score for the tasks: Multiple Choice, NER, Retrieval, and Generation. But no leaderboards show the results for Ranking?

Is this on purpose or a bug?

View post 0

Does this task have public & private leaderboard?

From the overall data description, it says

The dataset has been divided into three splits: train, phase-1 test, and phase-2 test

So it seems that there will be public and private LB. A safe guess is that private will be same size as public, but it would be good to get an official answer from admins.

View post 2

How do I view all my submission LB scores?

The button is still missing for task 1. Previously it was only be available for task 3. Now the submission button is available for task 3, task2, and the overall page. Using the overall page, we can view task 1 but it would be good to enable submission button on task 1 page too. Thanks!

View post 0

How do I view all my submission LB scores?

thanks. That’s helpful

View post 0

How do I view all my submission LB scores?

Is there a way to view all my previous submissions and their leaderboard scores? I cannot find this. I only see my best score displayed as my leaderboard rank placement but i do not know which submission generated this LB score.

View post 3

What order is prev items?

Based on my local validation, i believe that lower index in the provided prev_items list is older in time. And the larger index in prev_items is newer in time. (Using this assumption results in better CV score).

View post 5

!aicrowd dataset download --challenge task-1-next-product-recommendation

That is weird. Everything downloaded 100% for me.

View post 0