A Distributions-Oriented Approach to Forecast Verification of SPC

The history of forecast verification of Storm Prediction Center convective outlooks has been characterized by a “measures-oriented” approach over the last 30 years as first detailed in Weiss et al. (1980). For over 25 years, SPC outlooks were strictly categorical in their description of expected severe threat (i.e., Slight, Moderate and High Risk) over an outlined region. In order to better quantify the uncertainty of the forecasts and to provide better resolution of the type of severe weather expected (severe wind, hail and tornado) probabilistic convective outlooks have been utilized at the SPC since the year 2000. However, probabilistic forecasts carry with them a higher dimensionality (Murphy 1991) of information available for verification than the standard 2X2 contingency table statistics (i.e., CSI, FAR, POD) that are part of the traditional “measures-oriented” approach to forecast verification. Moreover a “measures-oriented” approach fails to identify the situations in which forecast performance might be weak or strong or when a correct null forecast is actually a “good” forecast.

Through various examples an alternative approach to forecast verification of SPC probability forecasts involving the use of the joint distribution of forecasts and observations will be presented. The results of this “distributions-oriented” approach to forecast verification (Murphy and Winkler, 1987) for a 6 year period (2001-2006) are shown graphically utilizing both reliability and Relative Operating Characteristics (ROC) diagrams and through the breakdown of the Brier Score into the 3 terms of reliability, resolution and uncertainty. Some examples including yearly and daily trends in forecast performance, forecast reliability and discrimination will be presented and discussed. The results indicate a marked improvement of forecast reliability between 2001 and 2006 as well a decrease in severe weather forecast discrimination and accuracy from the late spring into the mid-late summer months. In addition to the value of a distributions oriented verification methodology for probabilistic outlooks, the ROC diagram can also be utilized to ascertain forecast discrimination and accuracy prior to the advent of probabilistic outlooks providing for a robust historical forecast verification archive. The intention of this newer verification methodology is to allow forecasters to better understand the strengths and weaknesses of their forecasts in a multi-dimensional fashion and to identify ways in which their forecasts may be improved and re-calibrated.