Object-based verification of short-term, storm-scale probabilistic mesocyclone guidance from an experimental Warn-on-Forecast system

An object-based verification method for short-term, storm-scale probabilistic forecasts was developed and applied to mesocyclone guidance produced by the experimental Warn-on-Forecast System (WoFS) in 63 cases from 2017 and 2018. The probabilistic mesocyclone guidance was generated by calculating grid-scale ensemble probabilities from WoFS forecasts of updraft helicity (UH) in layers 2-5 km (mid-level) and 0-2 km (low-level) above ground level (AGL) aggregated over 60-min periods. The resulting ensemble probability swaths are associated with individual thunderstorms and treated as objects with a single, representative probability value prescribed. A mesocyclone probability object, conceptually, is a region bounded by the ensemble forecast envelope of a mesocyclone track for a given thunderstorm over 1-h. The mesocyclone probability objects were matched against rotation track objects in Multi-Radar Multi-Sensor data using the total interest score, but with the maximum displacement varied between 0, 9, 15, and 30 km. Forecast accuracy and reliability were assessed at four different forecast lead time periods: 0-60 min, 30-90 min, 60-120 min, and 90-150 min. In the 0-60 minute forecast period, the low-level UH probabilistic forecasts had a POD, FAR, and CSI of 0.53, 0.54, and 0.33, respectively, with a probability threshold of 22.2%. In the 90-150 minute forecast period, the POD and CSI dropped to 0.42 and 0.29 while FAR remained relatively unchanged. Forecast probabilities >30% overpredicted the likelihood of observed mesocyclones in the 0-60 min period; however, reliability improved when allowing larger maximum displacements for object matching and longer lead times.