Initial Development of GSL’s Interactive Verification Scorecard Tool

The Model Analysis Tool Suite (MATS) is a software product developed in-house by the National Oceanic and Atmospheric Administration (NOAA)’s Global Systems Laboratory (GSL) to assess the forecast skill of regional and global weather models. MATS consists of individual applications, which are each designed to verify a particular meteorological facet (e.g., ceiling heights, radar reflectivities, precipitation accumulations, etc.). Using MATS, model developers can produce interactive, publication-quality plots to quickly and easily compare the skill of individual models’ forecasts as they occur, as well as assess the performance of models under development. In the past year, GSL added a new feature to MATS: interactive verification scorecards that provide a high-level overview of the relative performance of two modeling systems (e.g., the operational HRRR vs. the experimental RRFS).

In general, scorecards consist of a matrix of cells, with each cell showing which model performs better for a set of user-selected parameters (e.g., time period, domain, variable, skill score, etc.), and whether that performance difference is statistically significant. MATS interactive scorecards comprise three components. The first is an interface that allows the user to select all parameters relevant to the model comparison. The second is a data processing service that allows for timely calculations of all the individual cell’s model data. The third is a visualization application that provides an interactive display of the scorecard results. allowing the user to click on any of its cells. When the user clicks on a scorecard cell, the data that was used to generate the score for that particular cell will be passed to the appropriate MATS application, thereby allowing the user to interactively plot the data using the range of MATS plot types, and understand the characteristics of the data. We envision that this interactive capability to visualize verification statistics in a number of different variables, forecast hours, and regions simultaneously (as well as the ability to automatically examine the data traces behind each cell in the scorecard) will greatly aid the understanding of model behavior and biases, improving the speed of model development and increasing GSL’s ability to contribute to R2O efforts.