Evaluation of probabilistic in-flight icing forecasts

During the winters of 1996-97 and 1997-98, two forecasters from NCAR formulated experimental probabilistic forecasts of icing conditions in the regions around several large cities in the midwestern United States. The purposes of this forecasting exercise were (1) to determine the capabilities of the forecasters to make probabilistic forecasts of this weather phenomenon and (2) to investigate the characteristics of a forecast verification method based on icing observations in the regions around large cities. In the middle of the experiment, feedback was provided to the forecasters feedback regarding the quality of their forecasts.

Pilot reports (PIREPs) of icing are used to verify the forecasts. To compensate for known problems with using PIREPs for verification, a "cities-based" approach is used to evaluate the probabilistic forecasts. The cities-based verification approach assumes that if icing conditions exist in the regions around larger cities/airports (i.e., where there is a lot of air traffic), the conditions will be reported. Thus, systematic observations of icing conditions are obtained.

Several types of forecasts were included in the study, including outlooks and short-term forecasts, and forecasts of several categories of icing severity. A variety of approaches are used to evaluate the forecasts, including "standard" diagnostic methods (i.e., based on reliability diagrams and Brier skill scores); signal-detection-theory approaches; methods designed to evaluate multiple forecast categories; and newly-developed methods that incorporate information about forecast value as well as forecast quality. Results obtained through application of these various approaches are compared and contrasted.

General results of the evaluation indicate that the forecasters were able to create reliable and reasonably accurate probabilistic icing forecasts for all categories of forecasts, with the best results for short-term forecasts of general icing conditions. However, the forecasts are most reliable when the region around the cities is extended to a larger area than was originally specified in the experimental plan. The initial results suggest that it is possible to formulate skillful probability forecasts of icing conditions.