Scores: A New Python Package for Verifying and Evaluating Forecasts with Xarray

Verification and evaluation of weather forecasts is essential for understanding their quality. Over the last decade, several verification methods with a focus on evaluating performance of predicting extremes or key decision thresholds have been developed across the statistics, economics and meteorological communities (e.g., threshold weighted scoring rules, Murphy diagrams, and the Fixed Risk Multicategorical (FiRM) score). Previously, there were no publicly available Python implementations of these recently developed methods. In response to these needs, a new open-source Python package for verifying and evaluating forecasts called "scores" was developed (available at https://github.com/nci/scores). In addition to these novel verification methods, scores also contains commonly-used verification metrics as well as data pre-processing tools and the ability to calculate confidence intervals.

Scores is a lightweight Python package that allows anyone to use it standalone or import it into their Python-based system. It is flexible and easy to use, working with multidimensional labelled data. This allows it to work with both point-based data (e.g., from weather stations), area-based data (e.g., weather warnings), and gridded data (e.g., numerical weather prediction ensembles). It leverages xarray's ability to handle most map projections and reads commonly used data formats including NetCDF, GRIB, and Zarr. Additionally, by exploiting xarray's ability to use Dask, it allows both in and out of memory calculation and scales from working on a laptop through to working with big datasets on HPC. All code in scores has been reviewed by scientists and software developers with all code covered by automated testing. Currently there is limited support for verification of pandas objects and we expect this to expand in the future.

Finally, there are Jupyter Notebook tutorials for every verification metric implemented in scores. These interactive tutorials not only demonstrate how to use the metric in scores with real or synthetic data but give an overview of theory behind the method, making it an ideal tool for learning verification science.

Documentation for scores is available at https://scores.readthedocs.io/.