Generating a Flexible Verification System for Precipitation by Assessing Precipitation Skill in the Unified Forecast System (UFS) and North American Multi-Model Ensemble (NMME) via Model Evaluation Tools (METplus)

Integration of the NOAA Unified Forecast System (UFS) Seasonal Forecast System (SFS) into the seasonal research and forecasting communities relies on assessment of skill and biases of precipitation over North America in both hindcasts and realtime forecasts, as well as comparison to existing seasonal model forecasts, particularly multi-model ensembles, such as the North American Multi Model Ensemble (NMME). The National Center for Atmospheric Research (NCAR)’s enhanced Model Evaluation Tools (METplus) verification framework is intended to be used to verify the UFS and has a large library of verification metrics and community support approach, and is thus a natural choice for creation of a verification system that allows easy comparison of individual models and multi-model ensembles. We aim to create a flexible verification framework utilizing METplus to allow streamlined assessment of single and multi-model probabilistic seasonal precipitation forecast skill, including deterministic and probabilistic hindcast, realtime, and conditional skill related to the key drivers of precipitation such as the El Nino Southern Oscillation (ENSO) and soil moisture that can be easily expanded to any climate model ensemble. Via METplus, we are able to calculate a variety of metrics including bias, error, anomaly correlation, Brier Skill Score, Rank Probability Skill Score, and more. These metrics are available without the need to code them in a preferred language such as Python, which can minimize code bugs in verification and allow for more consistency when calculating skill scores. The development, documentation, and demonstration of these process-based model capabilities can provide valuable feedback to the model development teams and community, with the potential to improve the key modes of variability that impact seasonal precipitation forecasts.