An approach for calibration of probabilistic forecasts with limited observational data
Two approaches for the calibration process are described. One approach is based on a limited-area analysis in small regions around large airports where the observation of icing conditions can be assumed to be relatively systematic. From these observations, a calibration function can be directly estimated. However, this method suffers from the limited numbers of observations available even in regions with a lot of aircraft activity, so that the assumptions often fail. In addition, this method only allows estimation of overall calibration values (i.e., typically, it is not possible to estimate values for individual locations, regions, or altitude ranges).
The second calibration approach relies on analytical estimation of the base rate and subsequently the calibration function, using verification results in the form of the Relative Operating Characteristic (ROC) curve. Both estimated values and assumed values of the base rate can be utilized in this approach. The approach also allows stratification of the results by region, altitude, and other factors.
The two approaches are applied to in-flight icing forecasts and the two sets of calibration results are compared. In general, the analytical approach seems to provide more reasonable results. These results generally suggest that all values of icing potential generally over-forecast the true icing probability. Regional and altitude variations in the calibration results are also evaluated, as are alternative approaches to estimating the base rate.
Although the main application of the approach described here is currently for in-flight icing algorithms, the approach potentially has application for other types of forecasts. For example, severe weather forecasts and turbulence forecasts suffer from many of the same types of observation limitations.