Towards Comprehensive Evaluation of Data-Driven Numerical Weather Prediction Models

In the past two years, large machine learning (ML) models have shown remarkable progress in their ability to generate medium-range weather forecasts that have competitive skill with state-of-the-art physical process-based models. To provide mature weather forecasting guidance, ML models should be evaluated on metrics that measure the physical consistency of their outputs across medium-range, s2s, and seasonal timescales. Furthermore, for probabilistic forecasting, ML models need to generate ensemble forecasts that correctly capture the distribution of possible outcomes, including extreme events. We share some of our work on developing metrics and benchmarks as well as our progress in building the next generation of data-driven weather prediction models.