Going Beyond the Terrestrial: Space Weather Verification with METplus

NOAA’s Space Weather Prediction Center (SWPC) and the National Center for Atmospheric Research (NCAR) have been working together since 2018 to advance verification capabilities for space weather. The capabilities are being built using the Model Evaluation Tools (MET) and METplus. MET is an efficient, configurable, state-of-the-art suite of verification tools developed by NCAR. METplus is a corresponding suite of python wrappers and other supporting capabilities which allow for complex real-time and retrospective verification workflows to be simplified and codified for robustness and reproducibility. This abstract provides an overview of the capabilities which have been developed or are under active development.

The evaluation of space weather prediction techniques presents unique challenges and opportunities. One challenge is the use of parameters and forecast outputs that are not used in terrestrial weather prediction (e.g., Total Electron Count, TEC; electron density profiles, magnetic field direction and strength, height of ionospheric reflectivity layers, ENLIL spiral), yet many of these parameters have similarities with terrestrial weather (e.g., TEC is somewhat analogous to precipitable water, the heights of ionospheric reflectivity layers are analogous to cloud ceiling heights). Another challenge is that many of the observational and model formats used in space weather prediction are quite different from those used in terrestrial prediction. These are surmountable through METplus’s powerful python embedding features, which allow the system to provide a generalized capability to read any model data source with user-written python codes. Like terrestrial weather, space weather has distinct areas of interest, such as the location of the auroral oval, areas of high TEC values, or areas of scintillation. MET’s object-based MODE tool is well suited for identifying objects and comparing the resulting object attributes such as displacement error and intensity, offering more insights beyond what traditional verification metrics provide.

The capabilities being developed through this effort are implemented in the Space Weather Prediction Center Real-Time (SWPC-RT), a platform-independent verification system which we have developed to apply advanced methods and techniques to space weather verification. The system is containerized, allowing for easy installation on Windows Subsystem for Linux (WSL-2), MacOSX, and Linux platforms without the complexities of installing and compiling MET. One main configuration file allows users to set up and configure multiple workflows and select which capabilities to run, which models to compare, and which graphical outputs to be generated. Users can use the system as is or develop their own use case by patterning after any of the more than 10 distinct use cases which currently are supported in the system. The system is being structured to allow both real-time and retrospective evaluation workflows.

SWPC-RT currently offers the capability to conduct gridded comparisons of ionospheric TEC between various analyses and models and stratify the resulting verification statistics by complex mask regions and time-varying quality flags. SWPC has extensively used these capabilities to evaluate the efficacy of commercial space weather radio occultation observations. Additional use cases allow object-oriented comparisons, illustrate verification of SWPC’s human forecasts (e.g., Kp index), perform gridded comparisons of gradients, and most recently, conduct comparisons using Global Ionospheric Map (GIM) IONosphere Map EXchange (IONEX) formatted files. New development is slated to expand SWPC-RT’s capabilities to conduct point-wise comparisons of scintillation observations to gradients in TEC fields. We are also working to add point-wise comparisons of ground-based observations, such as magnetic field direction, with the outputs from geospace models. Ultimately, SWPC-RT aims to provide routine and robust statistics, comparisons, and diagnostics to assist SWPC forecasters in evaluating model guidance (e.g., WAM-IPE, GloTEC, CTIPe) and SWPC’s official forecast products.