Application of object-oriented verification techniques to ensemble precipitation forecasts

Both the Method for Diagnostic Evaluation (MODE) and Contiguous Rain Area (CRA) object-oriented verification techniques have been used to analyze precipitation forecasts from two sets of ensembles to determine if spread-skill behavior observed using traditional measures can be seen in the object parameters, and to examine several methods of obtaining forecast guidance from the object parameters. One set of ensembles consisted of two 8 member Weather Research and Forecasting (WRF) model ensembles, one having mixed physics and dynamics with common initial and lateral boundary conditions (Phys) and another using common physics and dynamic core but with perturbed initial and lateral boundary conditions (IC/LBC). Traditional measures had found that spread grows much faster in IC/LBC than in Phys such that although skill and spread initially are as large or larger in Phys than in IC/LBC, after roughly 24 hours, better skill and spread are found in IC/LBC. These measures also reflected the strong diurnal signal of precipitation dominating the central United States during the warm season. The other set of ensembles included 5 members of a 4 km grid spacing WRF ensemble (ENS4) and 5 members of a 20 km WRF ensemble (ENS20). Traditional measures applied to these ensembles suggested that the diurnal signal was better in ENS4 and spread increased more rapidly than in ENS20.

Standard deviations (SDs) of four object parameters computed for the first set of ensembles showed the trend of enhanced spread growth in IC/LBC compared to Phys that had been observed in traditional measures, with areal coverage of precipitation exhibiting the greatest growth in spread with time. The two techniques did not produce identical results, although they did show the same general trends. CRA better showed differences between Phys and IC/LBC for SDs of rain rate, while MODE showed more of a difference for SDs of rain volume. A diurnal cycle had some influence on the SDs of all parameters, especially rain rate, volume, and areal coverage. MODE results also found evidence of a diurnal signal and faster growth of spread in object parameters in ENS4 than in ENS20, although the increases in spread were confined to the last 6 hours of a 33 hour forecast. Rain rate appeared to be the best forecasted parameter examined, with relatively little spread among members. Errors were larger for rain volume, area and location, and for these parameters a spread-skill relationship existed where cases with the largest spread in forecasted values had the least skillful forecasts.

Some forecasting approaches based on MODE and CRA output also have been demonstrated. It was found that forecasts based on averages of object parameters from each ensemble member might be more skillful than forecasts based on MODE or CRA applied to an ensemble mean computed using the probability matching technique for some parameters, but not all. The use of a probability threshold to define objects was also shown to be a valid forecasting approach with MODE.