Comparing Surrogate Severe Forecasting Skill and Storm Object Properties of the Experimental NSSL MPAS, RRFS, C-SHiELD, and HRRR Models

NSSL is testing NCAR’s Model Prediction Across Scales (MPAS) as a potential future dynamical core for both the Warn-on-Forecast System (WoFS) and eventually NOAA’s Unified Forecasting System (UFS) in support of NSSL’s goal to improve the lead time and accuracy of severe weather warnings and forecasts. NSSL began running experimental versions of MPAS in January of 2023. Additionally, NSSL performed daily realtime CONUS MPAS forecasts during the 2023 Spring Forecasting Experiment (SFE) with the goal of comparing MPAS core performance against that of the Finite Volume Cubed Sphere (FV3) core used in the Rapid Refresh Forecasting System (RRFS) and GFDL’s C-SHiELD, as well as the Advanced Research Weather Research and Forecasting (WRF-ARW) core used in the High Resolution Rapid Refresh (HRRR) model. The RRFS is tentatively slated to replace the HRRR in the operational forecasting suite in 2024, hence the comparisons presented here will help assess the strengths and weaknesses of the current and future operational model cores alongside those of the experimental MPAS.

In total, three sets of MPAS forecasts were produced. Two MPAS forecasts used the same physics configuration as HRRR and RRFS with varying initial conditions: MPAS-RT-NSSL (initialized from RRFS) and MPAS-HT-NSSL (initialized from HRRR). Varying the initial conditions this way enables a fair test of model core performance without the potential implications of initial condition bias. Additionally, to test the hypothesis that forecasted thunderstorm property differences between the NSSL and Thompson microphysics schemes when used in WRF remain similar in MPAS, an additional MPAS forecast was produced using HRRR initial conditions and NSSL microphysics (MPAS-HN-NSSL).

We will compare simulated reflectivity and updraft helicity against Multi-Radar / Multi-Sensor (MRMS) composite reflectivity and observe storm reports, respectively, to analyze reflectivity and surrogate severe forecasting skill using traditional metrics such as bias, ROC curves, FSS and Briar Score reliability. Accumulated forecast precipitation will also be compared against Stage 4 observed precipitation to compute forecast bias ratios for various precipitation rate thresholds. This analysis is important as the current largest weakness of the FV3-based RRFS is significant overprediction of precipitation, particularly for larger amounts/rates. Additionally, storm objects will be identified, classified, and tracked using the iterative storm segmentation and classification technique developed by Potvin et al. (2022). Object properties such as area, strength, depth, and various shape parameters will be compared against each other and those identified in MRMS data to assess how these relationships change across model cores and physics parameterizations.