More Than Just a Feeling: Statistical Analysis of Numerical Weather Prediction Models to Inform WxChallenge Forecasts

Many undergraduate students in atmospheric science participate in WxChallenge by submitting daily forecasts of temperature, wind, and precipitation for the following day. As an outcome of this exercise, participants develop a feel for various numerical weather prediction (NWP) models by comparing their predictions to the next day’s recorded observations. Many have noticed that some models’ forecasts appear consistently inaccurate under certain conditions. For example, one model may seem to consistently overestimate daily high temperatures, while another seems to underestimate accumulated precipitation. Thus, my goal for this project has been to investigate whether any statistically significant trends in model inaccuracy underlie these feelings. Characterizing the conditions under which a model is systematically biased would assist WxChallenge participants by informing them how to weigh that model in their overall forecast.
My method begins with data acquisition. Daily aggregate observational data is retrieved from Automated Surface Observing Systems (ASOS). Gridded forecast data is retrieved from a variety of models, subset, then aggregated to match the locations and time periods of observational data. Statistical analysis begins with spatial and temporal data visualization of model accuracies. Hypothesis testing is then performed to compare the difference of means between regions of the United States and between seasons of the year. Confidence intervals of significant trends in model inaccuracies are produced and described in the context of the conditions under which the biases appear.
This project has been an enriching experience in developing computational methods and statistical analysis in atmospheric science. While WxChallenge was the motivation behind this project, the results could be useful in the broader field of operational forecasting. Looking forward, I want to improve the accessibility of the results of this project by developing a product with a web-based graphical user interface. This would allow many others to visualize, interact with, and build upon the findings of this analysis.