Operational Scoring of Forecasts of Convective Weather Impacts in the Route Availability Planning Tool (RAPT)

The ability to evaluate forecast accuracy is crucial in automated air traffic management tools. The Route Availability Planning Tool (RAPT) uses convective weather forecasts from the Corridor Integrated Weather System (CIWS) to help air traffic controllers determine the operational impact of convective weather on specific departure times and routes. An evaluation of forecast accuracy may be performed by creating dual data sets of RAPT departure status timelines: the operational “predictive” blockage set calculated from CIWS weather forecasts, and a retrospective “true” blockage set using the actual weather. The comparison of predictive blockages and true blockages produces a forecast accuracy score, which can be used to evaluate the ability to forecast weather impacts.

An evaluation of forecasts accuracy addresses two crucial and related issues. The first issue is the sensitivity of RAPT to small details in the spatial organization and intensity of the weather forecast. If RAPT is overly sensitive to small changes in the weather, RAPT guidance may show excessive volatility as forecasts are updated, greatly increasing uncertainty about route blockage and reducing the value of RAPT guidance. The second issue is the predictability of RAPT performance. For RAPT, forecast validation requires a significant time lag (75 minutes) to collect the observed weather needed to calculate “true” forecast accuracy scores. Can such time-lagged scores be used to predict future performance? If not, what can be done to create a reasonable score approximation that does not require the time lag needed for “true” forecast validation?

Starting in 2009, RAPT began providing a heuristic measurement of forecast accuracy to help traffic managers assess the quality of RAPT route blockage forecasts. The algorithm generates a Modified Critical Success Index (mod-CSI) score via the comparison of “predictive” versus “true” grids of convective weather blockage. These grids are separated by a 40 minute lag, and are comprised of segmented blockage relative to the route direction. In the New York airspace, scores are calculated relative to the of the North, West, and South departure gate orientations. Multiple variations on the mod-CSI algorithm exist, and may potentially create an operational forecast score approximation.

In this paper, we will analyze RAPT blockage data from the 2009 convective season to answer these questions. The “true” RAPT blockage data will be created to determine the robustness of the RAPT blockage algorithm. Using this baseline, questions of RAPT stability and predictability relative to forecast accuracy will be addressed. Variations on the mod-CSI algorithm will be tested as means to provide an alternative to “true” forecast validation. Results of the analysis will be presented, and potential operational usage will be discussed.