Verification of Automated Hail Forecasts from the 2016 Hazardous Weather Testbed Spring Experiment

Every spring, the Storm Prediction Center (SPC) and the National Severe Storms Laboratory (NSSL) run an experiment to improve the prediction of severe weather called the Hazardous Weather Testbed. One of the major goals of the experiment is to forecast individual hazards, such as hail. These hail forecasts are run on the Center for Analysis and Prediction of Storms (CAPS) mixed physics ensemble. This ensemble is run using the Advanced Research Weather Research and Forecasting (WRF-ARW) numerical weather prediction model with 9 ensemble members and horizontal grid-spacing of 3 km. Automated hail forecasts are run for a 24 hour period using three different methods: HAILCAST, the Thompson Hail Size Method, and the Gagne Machine Learning Method.

To verify the three hail forecasting methods, neighborhood ensemble probabilities are calculated for a 24-hour period for both 25 mm and 50 mm hail.  These hail forecasting methods are verified against data from the NSSL Multi-Radar Multi-Sensor (MRMS) radar mosaic using the Maximum Expected Size of Hail (MESH) method. Relative Operating Characteristic (ROC) curves as well as Attribute Diagrams were created along with calculating the ROC Area Under the Curve (ROC AUC) and Brier Skill Score. A case study of May 26, 2016 was performed; on this day a large complex of storms moved over Nebraska, Kansas, Oklahoma, and Texas, producing 204 reports of severe hail, 183 reports of severe wind, and 21 tornado reports.

Overall, the Gagne Machine Learning Method has greater skill, in terms of the Brier Skill Score, than the other two hail forecasting methods. The Gagne Machine Learning Method also exhibits better discrimination for 25 mm hail in terms of the ROC AUC score. Lastly, the Gagne Machine Learning Method consistently performs well across all microphysics schemes because it is calibrated on each microphysics scheme. For the May 26, 2016 case study, the Gagne Machine Learning method exhibited greater capability to predict hail exceeding 25 mm in diameter while producing relatively few false alarms.