Gitlab Dev Ops issue

The Gitlab issue page is not auto fetching by default. We have to refresh every time to see the new Evaluation Logs changes in the issue page

View post 0

Couldn't connect to gitlab

Nevermind, Its working after setting up ssh key again

View post 1

Couldn't connect to gitlab

git push
kex_exchange_identification: Connection closed by remote host
Connection closed by 52.72.8.30 port 22
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

View post 0

Can we assume the same websites in both the test and training datasets?

Can we assume these same websites in both the test and training datasets?

output_cloud1115×636 185 KB

View post 0

Are these evaluation qa values present in qa.json correct?

The evaluation dataset questions consists of stock prices, but none of the answers are accurate as of Feb 16th, which is the last date given in the dataset. I used Nasdaq and MarketWatch to test the prices on that day, but none of them matched.
you can check the closing price on https://www.nasdaq.com/market-activity/stocks/tfx
For example, the closing price on feb16th is 251.07$ but the answer is given as 249.07$
“interaction_id”: 0,
“query”: “what’s the current stock price of Teleflex Incorporated Common Stock in USD?”,
“answer”: “249.07 USD”

View post 0

Top teams solutions

Nice use of an external dataset and converting it to this challenge format. I did explore different variations at the time of inference using prompt engineering. I used a better segmentation model (swin-base-IN21k) and modified the control items with pillars as well for better geometry along with different prompt engineering techniques. Even though baseline gave me a better score, it is really inconsistent. Finally, I submitted a realistic vision model from Comfort UI, which gave stable and consistent results, and based on the human evaluations I did expect some kind of randomness in the leaderboard. I would like to express my gratitude to the organizers of this challenge. The challenge is new and exciting, but because there are only 40 images in the test dataset, the human evaluations are much worse and inconsistent. It was really fun exploring stable diffusion models and their adapters. When I have more processing power, I want to work on this in the near future.

View post 3

🏆 Generative Interior Design Challenge: Top 3 Teams

@lavanya_nemani you can not judge the test dataset’s performance based on 3 public images. As the scores are really close, the preference of annotators can change a little bit.

View post 0

🏆 Generative Interior Design Challenge: Top 3 Teams

Congratulations to the winners! Also post this in discord, we have no idea that this post exists.

View post 2

Build fail -- Ephemeral storage issue

@lavanya_nemani The maximum size your submission can have is 10GB. So keep only maximum of 10GB in submission tag. Check your models folder and delete unused files. The repo can have more than 10Gb and you don’t need to delete them.

View post 1

Submission stuck at intermediate state

My submission with Id #251530 evaluation was complete, but it got stuck and did not proceed to the human evaluation phase.

View post 2

Service Announcement: Delays in GPU Node Provisioning

Can we try new submissions now?

View post 0

Updates to Task 1 Metrics

Only problem is that the leader board is dominated by ChatGPT’s PE track, but the peacock paper’s human evaluation is not the same as ChatGPT/ChatGPT4. It is as if their experimentation can be made false simply by using prompt engineering. Could someone please clarify this?

In the human evaluation, we find that facts generated by COMET-BART receive a high acceptance rate by crowdworkers for plausibility, slightly beating fewshot GPT-3. We also find that zero-shot GPT-3.5 model, although more advanced than the GPT-3 baseline model, scores, on average, ∼15.3% and
∼9.3% lower than COMET-BART in terms of automatic metrics and human acceptance, respectively.

View post 0

Is anyone encountering SSL issue with image build caching API? I don't know if there is something I can do about this?

The same thing is happening for me in task 2 as well. The evaluation gets initialized, but it says the time-out error. @dipam Can you make the logs easier to understand?

View post 1

Is anyone encountering SSL issue with image build caching API? I don't know if there is something I can do about this?

Yeah, the resubmission of same commit

View post 1

Is anyone encountering SSL issue with image build caching API? I don't know if there is something I can do about this?

Try creating the submission again, It happened for me as well.

View post 1

Updates to Task 1 Metrics

@dipam Using chatGPT, you are producing a score between 0 and 5, but is there a backup pipeline? if the generated value is a random text string or some out-of-bounds number. For human evaluation, it is also preferable to select the best LB from the GPU and PE tracks.

View post 0

Updates to Task 1 Metrics

It’s not clear what this metric is? Is this embeddings similarity? Or just asking chatGPT to generate some floating point based on humanness.

View post 1

Can somebody please help us in creating submission with out any errors?

If anyone else is having the same problem, creating tags smaller than 10GB will work properly.

View post 0

Can somebody please help us in creating submission with out any errors?

I am creating new submissions, but it fails without any valid logs. If I fork the repo, it already crosses 12 GB, and then you can only fit 4GB to fit the 16GB limit. Every time I create a new one, it says evaluation time out, but the evaluation has not even started. The discord is inactive, so I am posting the problem here.

View post 0

Submissions getting failed

Thanks for the reply, Are you planning to create baseline for task2 from the comfact repo model?

View post 0