Assessing Space Weather Predictions Using METplus - A Community Verification and Diagnostic Package

Verification and validation activities are critical for the success of modelling and prediction efforts at organizations around the world, including the National Oceanic and Atmospheric Administration (NOAA) and the National Aeronautics and Space Administration (NASA) in the U.S., the Met Office in the UK, and others across the globe. Over many decades, methods have been developed, tested and applied to evaluate tropospheric forecasts and to track their performance over time. Because different methods are required for different types of forecasts, and users have different requirements for verification information, the methods cover a broad spectrum of tools and approaches. For example, categorical methods are used to evaluate event-based forecasts and probabilistic methods are used for ensemble and probability forecasts. A wide variety of display and summary approaches – including line graphs, box plots, reliability diagrams, performance diagrams, and score cards – are used to track and compare performance across time, model, and region. Advanced spatial methods developed and applied in recent years provide enhanced information about forecast performance beyond point-by-point comparisons.

There is a community verification and diagnostic package available to assist with evaluating model performance as well as help diagnose why a forecast failed. This package is based on the Model Evaluation Tools (MET) developed by the National Center for Atmospheric Research (NCAR) and supported to the the Development Testbed Center. The system has recently been expanded to include Python wrappers to streamline complex verification workflows and the ability to call Python scripts to customize data ingest. The enhanced system is called METplus. While METplus were originally developed for verification and evaluation of atmospheric numerical weather prediction models, METplus is continually being developed and expanded into new prediction domains including Space Weather. Through collaborative projects with the National Oceanic and Atmospheric Administration (NOAA) and National Aeronautic and Space Administration (NASA), NCAR has expanded its support to Ionospheric and Geospace evaluation. This conference paper summarizes progress toward this goal and provides examples of how METplus could be used to help answer the question of why a forecast fails and how to improve it. It will also highlight future METplus development for space weather evaluation.