A Demonstration of Using a Suite of Tropospheric Verification Tools to Evaluate Predictions of Ionospheric Total Electron Count

Verification and validation activities are critical for the success of modelling and prediction efforts at organizations around the world, including the National Oceanic and Atmospheric Administration (NOAA) and the National Aeronautics and Space Administration (NASA) in the U.S., the Met Office in the UK, and others across the globe. Over many decades, methods have been developed, tested and applied to evaluate tropospheric forecasts and to track their performance over time. Because different methods are required for different types of forecasts, and users have different requirements for verification information, the methods cover a broad spectrum of tools and approaches. For example, categorical methods are used to evaluate event-based forecasts and probabilistic methods are used for ensemble and probability forecasts. A wide variety of display and summary approaches – including line graphs, box plots, reliability diagrams, performance diagrams, and score cards – are used to track and compare performance across time, model, and region. Advanced spatial methods developed and applied in recent years provide enhanced information about forecast performance beyond point-by-point comparisons.

More recently, comprehensive community tools have been developed and applied to provide consistent evaluations and to track terrestrial forecast performance. In particular, the Model Evaluation Tools (MET) developed at NCAR is being applied under the auspices of the U.S.’s Next Generation Global Prediction System (NGGPS) program to create an advanced unified terrestrial verification system (METplus). The METplus system consists of several components, including a C++/Fortran code-base (MET), for the computation of verification statistics based on gridded forecasts and either a gridded analysis or point-based observations. The system also incorporates a database and display system (METviewer) for aggregating statistics and plotting graphical results. These tools are designed to be highly flexible to allow for quick adaption to meet additional evaluation and diagnostic needs. A suite of python wrappers have been implemented to facilitate a quick set-up and implementation of the system, and to enhance the pre-existing plotting capabilities.

It is envisioned that this effort will expand to development of a unified verification system that will encompass a variety of spatial scales and provide a basis for evaluation of individual earth system component models (including atmosphere, atmospheric composition, land, ocean, ice and waves) and of the entire earth system model including coupling among system components and the evaluation of critical space weather fields and forecast products. This presentation will provide an overview of the current state-of-the-art in terrestrial forecast verification and the METplus unified verification system. It will highlight the application of METplus to a example evaluation of the Space Weather Prediction Center (SWPC) Whole Atmopheric Model-Ionosphere Plasmasphere Electrodynamics (WAM-IPE) prediction of Total Electron Count. It will also briefly address plans for continued development of METplus capability for space weather applications.