Development of Verification Techniques for the NSSL Experimental Warn-on-Forecast System for Ensembles (NEWS-e)

The NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e) provides probabilistic, short-term (0-6 hr) guidance for thunderstorm hazards such as tornadoes, large hail, damaging straight-line winds, and flash flooding. NEWS-e is designed to forecast hazards within individual convective storms rather than for regional threats, which presents several challenges for forecast verification. Primarily, forecasts are issued for relatively rare events, such as mesocyclones, that are not fully observed by standard observations. Object-based verification provides a method for overcoming these challenges as it allows matching between different, non-traditional forecast and verification objects (i.e. updraft helicity swaths and radar-derived rotation tracks) and provides extensive diagnostic information on specific errors between each matched object pair.

An object-based verification strategy for NEWS-e has been developed that matches composite reflectivity and updraft helicity objects, used as proxies for thunderstorm and mesocyclone occurrence, to corresponding objects in Multi-Radar Multi-Sensor (MRMS) composite reflectivity and rotation track observations. This verification technique has been applied to over 50 NEWS-e cases between 2016 and 2018 to establish a baseline for NEWS-e skill. However, despite being valuable for providing bulk measures of forecast skill, there are several limitations to the current methodology. Specifically, the object-based verification technique provides deterministic measures of forecast skill for an ensemble system, scores are sensitive to several tunable parameters in the object identification and matching process, and verification metrics are not weighted by storm impact (e.g. a brief, nontornadic mesocyclone affects skill scores as much as one producing a violent tornado).

The goal of this study is to expand the current NEWS-e verification framework to provide improved measures of skill for probabilistic forecasts, quantify sensitivities to tunable parameters in verification metrics, and develop methods for weighting bulk verification metrics by storm impact. Probabilistic forecasts will be evaluated by inclusion of verification metrics such as the Brier Skill Score and Continuous Ranked Probability Score. Additionally, verification metrics will be calculated using a parameter space of varying tunable parameters for a subset of NEWS-e forecasts subjectively determined to be relatively ‘poor’ or ‘good’ in order to identify thresholds that best match the subjective interpretation. Finally, MRMS rotation track objects will be matched with tornado reports and National Weather Service tornado warnings in order to quantify NEWS-e guidance for tornadic and nontornadic supercells.