Impact-Based Aviation Forecast Verification

The evaluation of weather forecast quality, as related to air traffic management performance, is of primary interest to Federal Aviation Administration (FAA) Systems Operations. One common forecast product used within the aviation community is the Terminal Aerodrome Forecasts (TAFs). National Weather Service (NWS) TAFs are used by a variety of aviation users, including domestic and international airlines, for important flight planning considerations such as fuel-load. TAFs are verified by the NWS with Probability of Detection (POD) and False Alarm Ratio (FAR) metrics. While POD and FAR can be used to evaluate forecast quality and consistency, they cannot alone be used to assess the incremental economic and safety benefits of a forecast—the value of a forecast—without some incorporated measure of impact or benefit realized by decision-makers.

In late 2011, AvMet Applications, Inc. (AvMet) at the request of the FAA created an automated, web-based method of TAF and air traffic information retrieval and review. This FAA-funded web tool enables next-day reviews of NWS TAFs during periods when Traffic Management Initiatives (TMI), specifically Ground Delay Programs and Ground Stops, are implemented. The tool identifies differences between Meteorological Aerodrome Report (METAR) observations and NWS TAFs for periods when TMIs were in effect on a daily basis. The differences between observation and TAF are measured in terms of critical operating thresholds, as defined by the FAA, and verify 4-hour lead-time. The tool highlights distinct instances of weather forecast accuracy during significant impact periods when TMIs were valid. Such information permits focused research on various weather parameters and forecast performance at airports throughout the National Airspace System (NAS). The NWS Aviation Services Branch recognizes the AvMet tool links forecast quality to system impacts and has conducted preliminary evaluations of its use in verification.

This paper and presentation will describe the significance of the tool, present results from the verification scheme, summarize the FAA and NWS perspective on the tool's potential use and discuss further enhancements to be developed that will expand the amount air traffic information for analysis, provide immediate access to real-time information, and further resolve relationships between forecast information and air traffic performance/impact.