An approach for calibration of probabilistic forecasts with limited observational data

Some types of automated forecasting algorithms are able to estimate the likelihood or potential for the occurrence of an event (e.g., thunderstorms, turbulence, icing conditions), but the forecast values are not appropriately calibrated to represent true probabilities. In particular, the reliability of the forecast values cannot be demonstrated due to the limitations of observations of the phenomenon. For example, forecasts of in-flight icing potential (with forecast values ranging from 0 to 1) are formulated by an automated algorithm; as the forecast value increases, the likelihood of icing conditions also increases. However, the probability values are un-calibrated because observations of icing conditions are not systematic (i.e., it is not possible to consistently evaluate the forecasts at any given forecast point); to be useful, the icing forecasts must be transformed to calibrated probability values. In this example, the observations are based on pilot reports (PIREPs) of icing conditions, which are sporadic in both time and space. Underlying the calibration issue is the fundamental limitation that we do not know and cannot directly estimate (at least with PIREPs) the base rate or climatology associated with the occurrence of icing conditions.

Two approaches for the calibration process are described. One approach is based on a limited-area analysis in small regions around large airports where the observation of icing conditions can be assumed to be relatively systematic. From these observations, a calibration function can be directly estimated. However, this method suffers from the limited numbers of observations available even in regions with a lot of aircraft activity, so that the assumptions often fail. In addition, this method only allows estimation of overall calibration values (i.e., typically, it is not possible to estimate values for individual locations, regions, or altitude ranges).

The second calibration approach relies on analytical estimation of the base rate and subsequently the calibration function, using verification results in the form of the Relative Operating Characteristic (ROC) curve. Both estimated values and assumed values of the base rate can be utilized in this approach. The approach also allows stratification of the results by region, altitude, and other factors.

The two approaches are applied to in-flight icing forecasts and the two sets of calibration results are compared. In general, the analytical approach seems to provide more reasonable results. These results generally suggest that all values of icing potential generally over-forecast the true icing probability. Regional and altitude variations in the calibration results are also evaluated, as are alternative approaches to estimating the base rate.

Although the main application of the approach described here is currently for in-flight icing algorithms, the approach potentially has application for other types of forecasts. For example, severe weather forecasts and turbulence forecasts suffer from many of the same types of observation limitations.