An object-based verification strategy for NEWS-e has been developed that matches composite reflectivity and updraft helicity objects, used as proxies for thunderstorm and mesocyclone occurrence, to corresponding objects in Multi-Radar Multi-Sensor (MRMS) composite reflectivity and rotation track observations. This verification technique has been applied to over 50 NEWS-e cases between 2016 and 2018 to establish a baseline for NEWS-e skill. However, despite being valuable for providing bulk measures of forecast skill, there are several limitations to the current methodology. Specifically, the object-based verification technique provides deterministic measures of forecast skill for an ensemble system, scores are sensitive to several tunable parameters in the object identification and matching process, and verification metrics are not weighted by storm impact (e.g. a brief, nontornadic mesocyclone affects skill scores as much as one producing a violent tornado).
The goal of this study is to expand the current NEWS-e verification framework to provide improved measures of skill for probabilistic forecasts, quantify sensitivities to tunable parameters in verification metrics, and develop methods for weighting bulk verification metrics by storm impact. Probabilistic forecasts will be evaluated by inclusion of verification metrics such as the Brier Skill Score and Continuous Ranked Probability Score. Additionally, verification metrics will be calculated using a parameter space of varying tunable parameters for a subset of NEWS-e forecasts subjectively determined to be relatively ‘poor’ or ‘good’ in order to identify thresholds that best match the subjective interpretation. Finally, MRMS rotation track objects will be matched with tornado reports and National Weather Service tornado warnings in order to quantify NEWS-e guidance for tornadic and nontornadic supercells.