Perceptions and Performance of Global AI Models in the 2024 NOAA Hazardous Weather Testbed

Global artificial intelligence (AI) models for weather forecasting have rapidly advanced in recent years, often benchmarked against reputable models like the European Centre for Medium-Range Weather Forecasting Integrated Forecasting System. While several studies have evaluated these AI models within the context of extreme events and underlying physical processes, there is a need for user-focused evaluation within an operational forecasting environment. In the 2024 NOAA Hazardous Weather Testbed (HWT), we measured user perception and objective performance of several global AI models, including GraphCast, FourCastNet v2, and Pangu-Weather, alongside the NOAA Global Forecast System (GFS).

Participants, composed of operational forecasters, model developers, and researchers from academia and government agencies, subjectively evaluated day-7 forecasts of the synoptic pattern and quantitative precipitation forecasts to assess whether these AI models provided value at medium-range lead times. These subjective ratings are compared against an objective verification of model forecasts to measure accuracy, reliability, and gaps in model performance versus perception. While the GFS exhibited a higher subjective rating overall, GraphCast had the highest subjective rating for AI models. However, the top-ranked AI model shifted considerably throughout the experiment, consistent with more variability observed in the objective performance metrics. This study is one of the first comprehensive user-based evaluations of global AI models, providing valuable insights into the specific attributes that users find most important. These insights can help guide future global AI model development, making them more reliable and tailored to meet the needs of end users, while also informing training and introduction into true operational environments.