Assessment and Evaluation from the AWC Summer Experiment

A variety of activities take place during the summer experiment as part of the Aviation Weather Center Aviation Weather Testbed including subjective and objective assessments for both experimental forecast products and the forecasting process. The primary goal of the summer experiment is to expose forecasters and industry representatives to experimental forecast products and procedures aiding in aviation forecasting and decision making. Providing participants with the performance of experimental products and, in turn, acquiring user feedback of those products from a forecasting perspective add valuable information to the experiment. The assessment portion of the experiment includes a variety of quantitative and qualitative verification activities as well as interpreting user survey responses. The combination of product verification and user feedback is critical in determining the potential success of operational transition. A forecast may have high quality but may not add value to the forecast process. Quantitative evaluations consist of a variety of standard verification sets such as: reliability of probabilistic forecasts, categorical skill scores for high resolution forecasts, and some neighborhood approaches for a comparison of both probabilistic and deterministic forecasts. Diagnostic climatologies were also used to qualitatively assess some of the experimental forecasts. Further, numerous observations sets were used as ‘truth' depending on the variable of interest, for example multiple ground-sensed lightning datasets were used for decision support in concert with experimental radar and satellite products. A summary of the methodology and findings will be presented.