How to use Mock API in submission

Please refer to the example here:

You do not need to include the MockAPI in your submission, it will be automatically made available in your evaluation environment, and the correct URI will be populated in the environment variable referenced in the example above (CRAG_MOCK_API_URL)

View post 0

Function calling arguments in local_evaluation.py is mismatching with dummy_model's interface

@evelynintech : Can you please ensure that you are using git lfs pull to pull the example dataset files via git lfs ?
You can also verify the contents of the example data files in the example_data folder to ensure that they are locally available.

View post 1

Submit error

@yong_deng : Can you please try submitting again now ?
As the error says, we had a scheduled maintenance yesterday, so we were not accepting submissions when you made your submission.

View post 0

Track2 Validation failed:No module named 'aicrowd_gym'

@carpe : We have requeued the submission which failed with "No module named ‘aicrowd_gym’”, it was because of a transient error at our end. You are not expected to manually install aicrowd_gym.

View post 1

How to add SSH key to Gitlab?

@momi193 : Thanks for pointing it out, we have updated the links in the main message as well.

Best,
Mohanty

View post 0

Submission failure for Task 2

Hi Iris,

The issue reported seems specific to the submissions made by your account, as there are numerous successful submissions after the timestamp reported in your message.

We acknowledge that the error propagation still has some cluster specific issues that we are trying to resolve as we speak.
In the meantime, please find attached the error for your submission.

submission-248780-evaluation-4331-188153-bl8rn-1190196615: /home/aicrowd/.conda/lib/python3.9/site-packages/transformers/convert_slow_tokenizer.py:515: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   warnings.warn(
submission-248780-evaluation-4331-188153-bl8rn-1190196615: Traceback (most recent call last):
submission-248780-evaluation-4331-188153-bl8rn-1190196615: Evaluation started
submission-248780-evaluation-4331-188153-bl8rn-1190196615: Init wrapper
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/run.py", line 3, in <module>
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     start_test_client()
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/client_launcher.py", line 20, in start_test_client
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     client.run_agent()
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 193, in run_agent
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     raw_response, status, message = self.process_request(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 99, in process_request
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     "data": self.route_agent_request(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 129, in route_agent_request
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self.execute(target_attribute, *args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/aicrowd_gym/clients/base_oracle_client.py", line 142, in execute
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return method(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/aicrowd_wrapper.py", line 30, in classify_link
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self.agent.classify_link(test_data_batch)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/agents/deberta_v3_large_nli_comFact_agent.py", line 56, in classify_link
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     head_logits = self.model(**head_inputs).logits
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1300, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     outputs = self.deberta(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     encoder_outputs = self.encoder(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     output_states = layer_module(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     attention_output = self.attention(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     self_output = self.self(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return self._call_impl(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     return forward_call(*args, **kwargs)
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 721, in forward
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     rel_att = self.disentangled_attention_bias(
submission-248780-evaluation-4331-188153-bl8rn-1190196615:   File "/home/aicrowd/.conda/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 819, in disentangled_attention_bias
submission-248780-evaluation-4331-188153-bl8rn-1190196615:     p2c_att = torch.gather(
submission-248780-evaluation-4331-188153-bl8rn-1190196615: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.29 GiB. GPU 0 has a total capacty of 14.75 GiB of which 1.02 GiB is free. Process 38611 has 13.73 GiB memory in use. Of the allocated memory 11.06 GiB is allocated by PyTorch, and 2.54 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Best,
Mohanty

View post 0

🚨 Important Update: Challenge Deadline Extended & Update to Rules

Hi kevin,

There was indeed an issue with compute node assignment earlier today, but it has been fixed now. We can see that the image building for your latest submission has already started.

View post 0

Resources section

@soham17 : The starter kit for the challenge is available here: AIcrowd / Challenges / Generative Interior Design Challenge 2024 / Generative Interior Design 2024 starter kit · GitLab

It is also linked at the top of the Challenge Overview Page here: AIcrowd | Generative Interior Design Challenge 2024 | Challenges

Best of Luck,
Mohanty

View post 0

No Submission Slots Remaining

@crlandsc : The submission quotas have not changed. The submission quotas are checked against the number of submissions made by your (or any of your team members) in the last 24 hour window.
So for example, if you make 3 submissions at 23:55 on Day 1, then you can make only 2 more submissions until 23:55 on Day 2 (assuming the submission quota is 5 submissions / day).
The daily submission quotas are not automatically reset at midnight each day.

In the last few days, these are the only submissions we see from your account:

• April 28th 04:13 UTC
• April 28th 04:33 UTC
• April 28th 04:45 UTC
• April 28th 15:34 UTC
• April 29th 19:34 UTC

All of the said submissions failed to evaluate for unrelated reasons (for more details please refer to the relevant debug logs in the issues associated with the failed submissions), and not because of exceeding submission quotas.

Additionally, we are updating the quotas for the competition, to allow up to 5 failed submissions which would not count towards your daily submission quota of 5 submissions.

View post 1

Submissions are being stuck on "Preparing the cluster for you" for a long time

@kimberley_jensen : There is a spike in the number of submissions because of the approaching deadline, and all the submissions are queued on the evaluation servers, and are all being eventually evaluated. Some submissions are timing out because of being in the queue for too long, and we are manually re-queuing them when that happens.

We will keep a close eye on the submission queues, and intervene as required.
Thank you for your patience.

View post 1

Submission failed : Failed to communicate with the grader

This issue has been resolved now. Please do let us know if you continue to face this issue.

Best,
Mohanty

View post 0

Feedback and suggestion

@andrey1362010 : A person who is a citizen of Russia and is currently a resident of another eligible country, with an active bank account present in that eligible country, is eligible to receive prizes.

View post 1

Feedback and suggestion

Apologies for the relative radio silence due to the holiday season.

GPUs

We are still waiting for a response from the organizing team about the provision of GPUs, so we will have to hold off on answering the question until we hear back from the organizing team.

Throughputs

We are investigating the issue with differential throughputs across different evaluations. A new instance is instantiated for every evaluation on our cloud provider, and the instance type is exactly the same - and hence the resources available. We have confirmed that the exact same instance type is being made available to all the submissions as well. We will get back to you with more details on this soon as well.

Timeouts

The current timeouts are 1hr, or 60 mins.

Apologies on the slow response times due to the Holiday season. We will be providing support at full capacity again starting 2nd of January, 2023.

Best,
Mohanty

View post 1

Has anyone actually got the prize payment?

Dear All,

All the prizes here have been processed except one participant where the said participant is a Russian national with a bank that our banking partners do not support any transactions to.

We are working closely with the said participant and our financial partners to come to a resolution soon. In the meantime, we have the confirmation from the rest of the winners about the receipt of their prizes.

Best,
Mohanty

View post 0

About my submissions

@victorkras2008 : can you please try again. You should be able to access the submission lists on both the challenges.

View post 0

About my submissions

@victorkras2008 : You should be able to see all your submissions in the submissions tab, for ex: AIcrowd | Semantic Segmentation | Submissions

View post 0

💬 Feedback and suggestion

@ricardodeazambuja : Yes the observation is correct.

As described here:

The dataset contains 422 flights, 2056 total frames (5 frames per flight at different AGLs),  Full semantic segmentation annotations of all frames and depth estimations. The dataset has been split into training and (public) test datasets. While the challenge will be scored using a private test dataset, we considered it useful to have this split to allow teams to share their results even after the challenge ends.

Of the total frames available, a subset if used for the (public) test set that you are currently being scored on. There will be an additional (private) test set that the final leaderboards will be based on. And submissions are currently limited to 5 submissions / day.

Best,
Mohanty

View post 0

👥 Looking for teammates?

View post 0

About leaderboard sorting

@victorkras2008 : yes the sorting on the leaderboard is correct. In case of Scale invariant logarithmic error (SILog), a lower score is better.

View post 0

Bad link to "Baseline for Mono Depth Perception" 📅

@victorkras2008 : Thanks for pointing it out.The links are correct, we realized that theres a few more things that need to be updated in the repo before they can be made public. So sorry for the confusion, we will try to make the repositories public as soon as we can.

Thanks,
Mohanty

View post 0