Extreme Weather Bench

As AI weather forecasting models rapidly gain popularity and are being evaluated for operational use, it is critical that the models are evaluated on a standardized set of high-impact weather events. To achieve this, we propose a set of community-curated events and metrics to evaluate the models. The events are chosen to represent a wide variety of high-impact global extreme weather hazards.

This work builds on the successful work of WeatherBench (Rasp et al, 2024) by introducing Extreme WeatherBench (EWB). EWB provides a community-curated benchmark of high-impact weather events, data, and metrics for model evaluation, including code to compute metrics and plot results across the full set of test cases. EWB phenomena span across multiple spatiotemporal scales and different parts of the weather spectrum: it ranges from short-term and small-scale impacts, such as severe storms, to long-term and larger-scale impacts, such as drought. With the goal of developing models with global impact, we provide case studies and data for events around the world.

For each high-impact weather event, we provide standard metrics used to evaluate that type of event and place emphasis on the primary impacts of the event for evaluation. For example, when predicting a tropical cyclone (TC), we include metrics to not only predict the strength of the TC correctly but also to predict the speed at which it strengthens, the time and place it will impact communities by hitting land, and the associated rainfall and other severe human impacts that it will generate.

EWB is a community system and will iteratively add additional phenomena, test cases and metrics in collaboration with the worldwide weather and forecast verification community. By involving global stakeholders, our goal is for EWB to become a standard performance evaluation across teams in the private, public, and academic sectors developing AI models for human-oriented operational weather forecasting.