Airline Impact-based Verification of Winter Weather Precipitation Forecasts

Verification of forecast guidance is important to gauge its quality and to compare the skill of different prediction methodologies, forecasters or providers. Traditional skill metrics that assess forecasts’ meteorological correctness can be useful for this purpose. However, such metrics may not provide insight into the trustworthiness and usefulness of the forecast guidance in the context of a user’s specific weather-impacted decision. Indeed, consumers of forecast guidance may have very different information needs and may value tradeoffs between different aspects of forecast skill differently.

In this work we present impact-based verification as a paradigm for assessing airport winter weather precipitation type and intensity forecasts in the context of their operational use in guiding airline decisions related to aircraft deicing. In this use case, reliable forecast guidance could help anticipate a mismatch between the demand and throughput of available aircraft deicing facilities, allowing an airline to proactively push, delay or even cancel departing flights to minimize schedule disruptions and keep passengers informed. On the other hand, trusting a poor forecast could result in an unanticipated need for deicing if conditions turned out to be worse than predicted, forcing last-minute cancellations or delays. Or it could lead to making costly and unnecessary schedule adjustments when the forecast overwarns. The impact-based verification approach was co-developed between subject matter experts at American Airlines and forecasters and scientists at The Weather Company. Precipitation type and intensity combinations are segmented into 13 categories, each of which has a different potential impact on aircraft de-icing requirements. ˚A 13x13 penalty table for all possible forecast & observation pairs provides a score for each combination. Matching forecast and observation categories are assigned a penalty of zero, while mismatched categories are penalized by an amount between 0 and 1 depending on the potential operational significance of the mismatch (e.g., different de-icing requirements).

After testing during the 2022-23 winter season, the penalty-based verification methodology is being run routinely at The Weather Company and applied to both automated and forecaster-issued forecasts for selected airports. This presentation will illustrate the verification results via daily evaluation tables that display forecasts, observations, and verification scores, as well as daily and seasonal aggregations. We will discuss how the verification results provide feedback to forecasters, their use by the airline to gain insight on how best to utilize the winter weather forecast guidance, and potential opportunities to improve the approach.