When Do Machine Learning Forecasts Succeed and Fail? Evaluating Synoptic Regimes Associated With a Random Forest's Good and Bad Severe Weather Predictions

The Colorado State University Machine Learning Probabilities (CSU-MLP) system produces real-time daily random forest-based forecasts for severe weather at short- to medium-range timescales. Probabilistic predictions are made for individual severe weather hazards (tornadoes, wind, and hail) out to lead times of three days, and aggregate severe hazards out to eight days, similar to the formatting of the convective outlooks issued by the Storm Prediction Center (SPC). These forecast guidance products began to be regularly considered in operations by the SPC as well as local National Weather Service Weather Forecast Offices since at least early 2022. With the operational use of the products, recent research efforts by our group have focused on improving trustworthiness and transparency of the CSU-MLP products through explainability methods and performance evaluations.

To support these goals, this work aims to examine various synoptic regimes, which are identified using an unsupervised machine learning technique, across well- and poor-performing CSU-MLP forecasts. Self-organizing maps (SOM) are harnessed to cluster daily synoptic patterns. Environmental fields from the ERA-5 reanalysis over a two-year period (January 2021 through December 2022) are used as inputs to the SOM. SOM nodes are optimally grouped using k-means clustering. From there, the spatiotemporal characteristics and composited environmental fields associated with each cluster of SOM-identified regimes are examined. The clustered regimes are then used to stratify daily CSU-MLP forecast cases and associated performance metrics across varying forecast lead times in an effort to evaluate how various environmental set-ups might be related to successful versus unsuccessful forecasts. Identifying regimes that are associated with high and low forecast performance may help forecasters who use the CSU-MLP products better delineate when they should or should not trust its guidance in their real-time forecasting operations.