❗️Important Updates for Phase II of CityLearn Challenge 2023

View post 0

❗️Important Updates for Phase II of CityLearn Challenge 2023

TL;DR

• Fixed negative heat pump action bug in RBC.

• CityLearn updated to support power outage scenarios (see comparison to Phase I environment).

• Control track evaluation updated to consider performance during power outage.

• Online evaluator dataset to change in Phase II.

• Three buildings used in Phase II public leaderboard evaluation.

• Private leaderboard at the end of Phase II that uses six buildings will be used to select winners.

• From September 19, 2023, when Phase II commences, pull the latest control and forecast track starter-kits for your local evaluation and rerun pip install -r requirements.txt.

• There may be breaking changes in CityLearn and the starter-kit that affect your solutions from Phase I.

• Ensure that your submission to the the private leaderboard at the end of Phase II can be used on a larger number of buildings than trained on.

What has changed in Phase II?

Please take note of the following changes as we advance to Phase II of The CityLearn Challenge 2023:

Bug squashed!

There is a bug in CityLearn==2.0b9 used in Phase I where the RBC agents provide negative actions to the cooling_device. This causes the building’s cooling_demand and cooling_electricity_consumption to be negative. Since a large proportion of building loads are attributed to cooling, the building’s net_electricity_consumption becomes negative as well as KPIs that make use of it in their calculation. This bug has been fixed since CityLearn==2.1b1.

Lights out!

See comparison between Phase I and II environments.

CityLearn now supports power outage scenarios! During a power outage event, the grid is unable to provide the buildings with electricity and control solutions can only make use of the available distributed energy resources in buildings including domestic hot water storage, battery and PV system to satisfy cooling, domestic hot water and non-shiftable loads otherwise, risk thermal discomfort and unserved energy during the event. During normal operation i.e., when there is no power outage, there is unlimited supply from the grid.

The outage signals are either defined in the building data file or generated using some stochastic model that is defined in the schema for each building and constructed at runtime.

The CityLearn Challenge 2023 makes use of the stochastic model approach that is based on Reliability Metrics of U.S. Distribution System. The stochastic model generates time series of power outage signals using System Average Interruption Frequency Index (SAIFI) and Customer Average Interruption Duration Index (CAIDI) to find outage days, their start time and duration. The generated signal for a fixed SAIFI and CAIDI is controlled by changing the model’s random_seed. If interested, please, see the docs for how the model is implemented and feel free to suggest any improvements .

By the way, power outages will only be applied to the control track environment so, you do not need to worry about blackouts in your forecast track submissions and evaluations . Participants in the control track should expect outage events to happen at the same time in all buildings, at least one power outage event, and events that last at least, half a day during online evaluation.

How well did you control the dark?

A new score, Score_Control^Resilience has been added to the control track to evaluate the average building resilience during power outage events. This score is the average of two KPIs: 1 - thermal resilience (M), and normalized unserved energy (S). See the competition page for their definitions.

With the addition of Score_Control^Resilience, there are four control scores: thermal comfort score (Score_Control^Comfort), an emissions score (Score_Control^Emissions), a grid score (Score_Control^Grid), as well as the resilience score (Score_Control^Resilience) and their respective weights are defined here.

Who’s topping the leaderboard?

In Phase I, participants in both the forecast and control tracks were given the same dataset consisting of three buildings and one month worth of data. The online evaluator made use this same dataset to evaluate submissions and update the leaderboard. Thus, the score calculated during local evaluation were the same as those posted on the leaderboard (assuming the same CityLearn environment version in both local and online evaluation).

In Phase II, the same three-building dataset is provided for local evaluation in both tracks but with an updated schema.json that defines the stochastic power outage model for in each building as well as a random_seed that is used to generate outage signals. Participants can change this random_seed using the provided helper function in the control track starter kit (local_evaluation.update_power_outage_random_seed) to train their control agent(s). Although the forecast track uses this updated schema, power outages will be disabled in its environment.

On the online evaluator side in Phase II and prior to the end of the competition, a completely different dataset is used where there are three buildings (including the three in the local evaluation dataset) and three months worth of data based on a different weather file. Thus, submitted agents should be able to generalize to weather conditions, occupant behaviors, and buildings not seen during training. Also, in the control track, a public leaderboard will be reflective of the average scores from using three different private random_seed values to generate stochastic power outage signals in the environment. These three seeds will be kept constant throughout Phase II. In the case of the forecast track, the public leaderboard is reflective of just one environment’s score since power outages are not considered hence no stochastic influence on the environment by random_seed.

At the end of Phase II, submission acceptance will cease and a private leaderboard that is only visible to the challenge organizers will be launched. This leaderboard will be used to select winners for both tracks. This private leaderboard is reflective of a similar dataset as that used for online evaluation during Phase II except:

1. It is made up six buildings, three of which are used in Phase II public leaderboard evaluation.
2. The control track private leaderboard will be reflective of the average scores from using three private random_seed values that are different from those used in Phase II public leaderboard.

Hence, participants must make sure their final submission at the end of Phase II is coded in such a way that it is applicable to more buildings than it was trained on.

What you should do …

• From September 19, 2023, when Phase II commences, pull the latest control and forecast track starter-kits for your local evaluation and rerun pip install -r requirements.txt.

• There may be breaking changes in CityLearn and the starter-kit that affect your solutions from Phase I.

• Ensure that your submission to the the private leaderboard at the end of Phase II can be used on a larger number of buildings than trained on.

View post 0

💬 Feedback & Suggestions

View post 0

👥 Looking for teammates?

View post 0

👥 Looking for teammates?

View post 0

Welcome to CityLearn Challenge 2023

Hello Participant,

Welcome to NeurIPS 2022: CityLearn Challenge . In this challenge, participants have to coordinate the energy consumed by each of the buildings within a simulated micro-grid.

The CityLearn Challenge 2023 provides an avenue to address energy problems by leveraging CityLearn, an OpenAI Gym environment for the implementation of AI agents for demand response. In this challenge, participants must coordinate the energy consumed by each of the buildings within a simulated micro-grid.

View post 0

👥 Looking for teammates?

Competing is more fun with a team!

Introduce yourself here, and find others who are looking to team up!

Format:

• A short introduction about you and your background.
• What brings you to this challenge?
• Some ideas you wish to explore as a part of this challenge?

Cheers,
Team CityLearn 2023

View post 0

💬 Feedback & Suggestions

We are constantly trying to improve this challenge for you and would appreciate any feedback you might have!

Please reply to this thread with your suggestions and feedback on making the challenge better for you!

• What have been your major pain points so far?
• What would you like to see improved?

Cheers,
Team CityLearn 2023

View post 0

👥 Looking for teammates?

View post 0

👾 Welcome to MeltingPot Challenge 2023

View post 0

💬 Feedback & Suggestions

View post 0

👥 Looking for teammates?

View post 0

👾 Welcome to MeltingPot Challenge 2023

View post 0

📹 Highlights & Recording from the Mosquito Alert Challenge Townhall 🦟

View post 0

📹 Highlights & Recording from the Mosquito Alert Challenge Townhall 🦟

Hello all,

Our recent #MosquitoAlertTownhall was a whirlwind of insights, discussions, and revelations on mosquito surveillance in the digital age. If you missed it, here’s a quick recap:

• Frederic Bartumeus, Co-Director of MosquitoAlert, kickstarted the event by shedding light on the MosquitoAlert ecosystem. He emphasized the powerful potential of CitizenScience and smartphones in revolutionizing mosquito surveillance.

• Roger Eritja delved into the intricate challenge of species identification using mosquito photographs. He pointed out the nuances in mosquito morphology and how distinguishing genus is relatively straightforward, but discerning specific species can be a challenge.

• Monika Falk, AI Researcher, illustrated the intricate nature of mosquito identification. She highlighted how accurately identifying a mosquito is a challenging feat, even for experts. Detecting tiny objects poses a formidable obstacle in the realm of computer vision, a hurdle we’re actively trying to overcome.

• Joan Garriga introduced us to AIMA - Mosquito Alert’s AI. He highlighted its ability to swiftly identify mosquito images, ensuring prompt feedback for users and aiding experts in achieving faster, more precise mosquito identification.

For a complete experience, make sure to watch the full Townhall discussion on our YouTube channel: https://youtu.be/qSWJZUY-5DM.

Stay buzzing and keep contributing to this significant cause!

View post 1

🚨 Important Updates for Round 2

View post 0

📹 Join Mosquito Alert Challenge Townhall!

View post 0

📹 Join Mosquito Alert Challenge Townhall!

View post 0

📹 Join Mosquito Alert Challenge Townhall!

Desktop Banner4000×768 127 KB

Hello Participants!

We invite you to join the exclusive Mosquito Alert Townhall event just as Round 2 of the challenge kicks-off. This is a great opportunity to learn from domain specialists, acquire insights, dive into innovative strategies, and have your burning questions addressed by a panel of experts.

24th August, 2023 10:00 AM CEST
Join the Townhall on Zoom

For those who may not be able to attend, rest assured, a recording will be made available. Moreover, feel free to submit your questions directly on this post, and our panel will make it a point to address them.

Townhall Highlights

• Exclusive Insights: Engage with our expert panel, which includes eminent entomologists. Pose your questions related to Round 2 directly to the challenge organizers.

• Collaborative Discussions: Mingle with fellow participants, swap ideas, refine your strategies, and delve deeper into the AI realm.

• Round 1 Debrief: Reflect on the first round’s techniques and solutions to fortify your approach for the upcoming challenges.

• Stay Up-to-date: Receive the latest news about this challenge and be in the loop for upcoming AIcrowd rendezvous.

Meet the speakers

• Frederic Bartumeus: ICREA Research Professor and Head of the Computational and
  Theoretical Ecology Lab, he will shed light on the driving force and objectives behind this challenge.

• Roger Eritja: An experienced entomologist with a 40-year track on mosquito control, Roger will unravel the complexities faced in digital entomology.

• Monika Faik: Our AI Research Technician will provide an in-depth walkthrough of the challenge and the dataset.

• Joan Garriga: A renowned Data Analyst and Scientist, Joan will share insights on current solutions and offer a sneak peek into what the subsequent round holds.

Make the most of this townhall by directly interacting with our speakers. Should you be unable to attend, please ensure your questions are dropped in the comments below, and we will address them during our session.

Mark your calendars, prepare your questions, and get ready to immerse yourself in an inspiring even!

Hope to see you soon!
Team AIcrowd

View post 2

🚨 Important Updates for Round 2

Hello Participants!

We’re gearing up for Round 2 and have some exciting updates to share:

1. Code Submissions: We are now accepting code submissions for Round 2. Your model needs to predict of each image in 1 second. For compute, you will be provided a virutal machine with 2 CPU cores and 12 GB of RAM.

2. Dataset Enhancements:

• The test dataset from Round 1 has been integrated into the training set for Round 2, aiding in the further refinement of your models.
• The test set for the second round evaluation is curated with precision. It does not contain any images with multiple mosquitoes.
• The bounding boxes have undergone careful revision.
• The evaluation criteria remain consistent. We’ll be evaluating models based on the F1 score with a 0.75 IoU threshold, as initially defined.
• Despite rigorous quality checks, the training set might still include approximately 3-4% of images with multiple mosquitoes or suboptimal bounding boxes. If participants find such images detrimental to their training, they are welcome to exclude them.
• We remain committed to enhancing the dataset’s quality and will release updated versions if necessary.

3. Evaluation Criteria: We would like to reiterate that the evaluation metric remains the F1 score, employing a 0.75 IoU threshold, as was mentioned at the outset of this competition.

4. Leaderboards: Round 2 features both private and public leaderboards, each representing roughly half of the test set. During live evaluations, only the public leaderboard will be accessible. Post-challenge, private leaderboard scores will be unveiled. The final winners will be determined based on these private leaderboard scores.

5. Team Formation Deadline: A gentle reminder that the last date to finalize your teams is 5th September. Ensure your team members are confirmed by then.

Starter-kit for Phase 2

We’re looking forward to seeing your contributions in this round!

All the best!
Team Mosquito Alert

View post 0