Using traditional and spatial verification methods to evaluate real-time model forecasts of convection

As higher resolution numerical models, and their output of discontinuous fields like simulated reflectivity and 1-h accumulated precipitation, continue to be developed and used operationally, there is an increasing need in the research community to use newer verification techniques. Traditional metrics, such as the Equitable Threat Score (ETS), might penalize a model twice, once for observed-but-not forecasted and once for forecasted-but-not-observed, even though the forecast might be subjectively judged a good quality forecast. Conversely, the ETS may overestimate the forecast accuracy and value in severe-weather forecasting applications, depending on the forecast bias and spatial scales in question. A goal of the newer verification metrics, that attempt to account for the spatial scale of the event, is to objectively compare model output data with observations much like forecasters and scientists would do through subjective analysis.

This study will use both traditional and spatial verification techniques to evaluate two WRF model forecasts, one that assimilates radar data in the initial condition using a 3DVAR technique (CN) and one that is identical, but uses the 12-km North American Mesoscale (NAM) model analysis as the initial condition (C0). These forecasts were run in real-time by the Center for Analysis and Prediction of Storms (CAPS) during the Spring of 2009 and 2010 in support of the NOAA Hazardous Weather Testbed (HWT) annual Spring Experiment and the Verification of the Origins of Rotation in Tornadoes Experiment (VORTEX-2) field program. The Spring Experiment participants noted, subjectively, that the skill that was suggested by the traditional objective metrics often seemed large compared to the skill than was perceived by the participants. For this study, both traditional and newer verification metrics that take the bias and spatial scales of the forecasts into account will be computed to characterize the spatial scales and lead times at which skill diminishes for forecasts of convection. This study will also examine the performance of the real-time 3-km High Resolution Rapid Refresh (HRRR) forecasts over the same time period in a similar manner.