Development of GSD's Model Analysis Tool Suite (MATS)

The Model Analysis Tool Suite (MATS) is a software product developed in-house by the National Oceanic and Atmospheric Administration (NOAA)’s Global Systems Division (GSD) to assess the forecast skill of regional and global weather models. MATS contains fourteen individual applications, each designed to verify a particular meteorological facet, such as 24 hour precipitation, cloud ceiling height, or surface radiation flux. The goal of MATS is to produce interactive graphs in a web-based framework that makes it easy for GSD developers to quickly compare the skill of individual models and ensembles, as well as assess the performance of models under development. MATS also allows developers to produce publication-quality interactive graphs in a web-based framework, and to perform a focused analysis of interesting weather features.

Previously, GSD developers relied either on a legacy Java-based graphing suite, which posed a network security risk, or pre-generated static images of a few chosen fields. MATS development was motivated by the need to create a fast and flexible verification system to replace the legacy Java code, and allow users to generate plots from a wide selection of parameters and date ranges. Each MATS application has its own customized collection of parameter and date range selectors embedded within the MATS framework, which consists of a JavaScript Meteor app coupled to a MySQL database. When a user requests a particular graph through the MATS interface, the JavaScript routines construct a database query, execute it, process the returned data, and produce an interactive, easily-customizable plot. MATS supports a variety of plot types, such as timeseries, vertical profiles, diurnal averages, dieoff curves, threshold graphs, scatter plots, and 2D map displays, each of which display a variety of verification statistics, like skill score, root mean squared error, bias, and mean absolute error. These statistics are calculated using numerous observation data sources, such as METARs and mesonet data.

The MATS build and deploy process includes browser-based automated testing, which provides significant quality control, ensuring that new updates don’t unintentionally break other parts of MATS. The MATS automated testing is done via the software package chimp.js, which uses WebdriverIO to iterate through a growing library of scenarios in a browser window, each trying to produce a specific graph with a specific set of parameters. Chimp.js executes individual scenarios, using WebdriverIO to load an app in a browser, click specified buttons and set specified parameters, and confirm that the desired graph is produced. If the app behaves as expected, chimp.js returns a pass. Otherwise, it returns a fail and warns against publishing apps that have software bugs. Scenarios to check for these specific bugs in the future are subsequently added to the testing library.

Ultimately, we hope that MATS can enhance model development at GSD by allowing programmers to easily and efficiently incorporate real-time and retro verification into the development process.