A Comparison of Pure AI and Physics-Based Weather Prediction Precipitation Forecasts

Pure AI-based Weather Prediction (AIWP) models have made waves within the scientific community and the media, with claims of superior performance and utility to dynamical models. These claims are at times based on limited verification metrics and observational datasets and neglect key considerations for real-world applications, such as comprehensiveness of the output variables. Google DeepMind's GraphCast model became the first AIWP model to predict precipitation, but performed only limited verification. We present an analysis of GraphCast precipitation forecasts over the contiguous United States and compare to results from the Global Forecast System (GFS) model using 1) grid-based, 2) neighborhood, and 3) object-oriented verification verified against the ECMWF Reanalysis v5 (ERA5) and the Multi-Radar Multi-Sensor (MRMS) datasets. We affirmed that GraphCast performs better than the GFS in many basic metrics, but the GFS's precipitation distribution more closely matches the ERA5 and MRMS distributions. The GraphCast advantage over the GFS for these metrics is also smaller when both are compared against MRMS instead of ERA5. Equitable Threat Score (ETS) at varying thresholds shows the GFS gains on or exceeds GraphCast forecast performance for higher precipitation accumulations and fractions skill score (FSS) for increasing window sizes shows greater gains for the GFS than GraphCast, implying the GFS has a better handle on intensity but perhaps struggles with precipitation position. GraphCast showed a smaller bias in the Southeastern United States, a finding worthy of further investigation. Lastly, object-oriented verification for GraphCast showed negative area and intensity biases at higher intensity thresholds. One major caveat to these results is that GraphCast was initialized with GFS initial conditions. We are now testing GraphCast initialized with ECMWF's Integrated Forecast System (IFS) initial conditions, which may further improve GraphCast forecasts.