5.2
Confidence intervals for some verification measures - a survey of several methods
Tressa L. Kane, NCAR, Boulder, CO; and B. G. Brown
Frequently, the goal of a forecast verification study is to compare the quality of different forecasts or algorithms. Point estimates of forecast quality are inadequate for determining if one type of forecast or algorithm performs significantly better than another or whether forecasts have been improved. Confidence intervals provide an effective way to make these comparisons. Additionally, confidence intervals explicitly communicate the uncertainty in a measure of forecast quality in a way that point estimates cannot. However, these methods have been infrequently applied in forecast verification, due to the characteristics of the verification data and measures.
Among the commonly used measures of forecast quality are the probability of detection (POD) and the probability of false detection (POFD). Confidence intervals for the POD and POFD for aircraft icing and turbulence forecasts are constructed and examined. Intervals are estimated using several methods, including Gaussian, binomial, and bootstrap. The confidence intervals are compared, and the assumptions made by the different methods are discussed along with possible violations of these assumptions by the forecast and verification data. Finally, applicability of these methods to other measures of forecast quality is discussed.
Session 5, Forecast evaluation
Thursday, 11 May 2000, 1:30 PM-5:00 PM
Previous paper Next paper