Reliability and Skill of the Air Force Weather Agency's Ensemble Prediction Systems

Deterministic weather models are limited by the fact that they only depict one of the many possible evolutions of the atmosphere. Inevitably, weather models will always be prone to error, especially since sparse observations make it impossible to represent the true initial state of the atmosphere. Ensemble weather models that represent multiple plausible forecasts are the next progression of numerical weather prediction and need to be operationally tested. Ensembles are capable of assessing the probability of certain weather forecast outcomes which is especially valuable to decision makers who apply risk management to operational decisions. The Ensemble Prediction Systems (EPS) at Air Force Weather (AFW) provide probability based forecasts to hundreds of worldwide locations. EPS products known as Point Ensemble Probabilities (PEP) are tailored specifically to the United States military and its criteria for operationally significant weather thresholds. In 2013 from April to October, a validation study was performed on the PEP products of AFW's Global EPS, and the 20km and 4km resolution Mesoscale EPS across 10 geographically diverse locations. In general, they found that the PEP products were over forecasting lightning and that precipitation and wind forecasts improved with higher resolution EPS. Since then, significant changes have been made to how the EPS generate products. This current study will assess additional weather parameters and compare AFW global and mesoscale EPS at 12 new Continental United States (CONUS) locations. The PEP bulletins will be compared to climatology, METARs, and national lightning detection network data to generate reliability diagrams and Brier Skill Scores (BSS). Initial results from April to July of 2015 show that each EPS is under forecasting ceiling and visibility for most forecast hours at several locations. The under forecasting is most severe at Vandenburg AFB, an area prone to frequent marine layer fog. The MEPS 4km also shows a significant improvement to skill for forecasting lightning over the other EPS grid scales with skill remaining higher for more forecast hours. Future research will expand the data through October 2015 to determine additional trends with the comparative results.