Toward understanding how well skill scores represent the utility of convective storm forecasts to aviation planners

For many years pixel-based skill scores such as correlation, CSI, FAR, POD, RMSE have been used to quantify forecast skill. Unfortunately, these pixel-based metrics do not necessarily convey information that can be easily interpreted by the users of the forecasts (in this case aviation planners) who would like to know the performance of a forecast system with respect to particular attributes of storm that impact aviation (e.g., those that cause a pilot to deviate). This is particularly true when trying to assess the utility of a forecast for strategic planning (i.e., planning routes for air traffic across the country with a lead time of more than 5 hours). Attempts at quantifying skill for this application include up-scaling the data (in space and or time) to give credit for “near” misses, using fuzzy verification approaches to account for displacement errors (e.g., Fractions Skill Score) and object-based verification techniques, for assessing the ability of a forecast system to represent the properties of storms (e.g., storm number density, shape, organization (linear squall line with or without breaks), orientation, etc). Only the latter of these approaches allows for exploration of a forecast system's skill in predicting the storm attributes that are relevant for aviation planning. In this paper, we explore the relationships between pixel-based, up-scaling, fuzzy and object-based verification approaches in order to assess which skill scores provide the most telling information on the skill of forecasts used by aviation planners. Forecast and observation used for this study are from the summer 2010 Operational Evaluation (OpEval) of CoSPA (short term forecasts of storm intensity and echo tops for use by aviation planners developed under FAA Aviation Weather Research Program by MIT-LL, NOAA-GSD and NCAR-RAL) which was sponsored by FAA Reduce Weather Impact (RWI) program. Results will be shown in the context of how aviation planners used deterministic forecast information during the summer 2010 OpEval. Areas where this research can help improve the calculation of weights need by the blending algorithm of CoSPA will be discussed.

This research is in response to requirements and funding by the Federal Aviation Administration (FAA). The views expressed are those of the authors and do not necessarily represent the official policy or position of the FAA.