Towards Operational Standards: A Proposed Procedure for ML Training and Validation for Gridded Data in Meteorology

Overview

We propose a procedure to generate training-validation-test split and truth datasets for operational gridded forecast post-processing. We also present preliminary results of a convolutional network model trained using the proposed procedures. Standardizing these processes is essential to avoid variability in dataset selection by individual model architects, which can significantly impact model performance. Inconsistent training and truth data selection also complicates performance comparisons across different models. Best practices in ML are difficult to transfer to geosciences due to factors such as autocorrelation in multiple dimensions, varying data structure and resolutions, and sparse observations. Due to the complexity of meteorological data, there are no well-established standards to adhere to the principle that training, validation, and test data should be chosen such that they are as independent as possible, thereby avoiding overfitting to temporal or spatial characteristics. Developing such standards for geoscience ML with operational constraints in mind is essential to reducing barriers to the adoption of machine learning models.

Training-Validation-Test Split:

We propose training on the entire model domain while selectively excluding recent observations from certain stations. A typical approach of spatially splitting training and validation sets into smaller subdomains increases the likelihood of overfitting to a small number of stations. Furthermore, improper spatial splitting may result in the model failing to learn all terrain features, especially in convolutional models. A constant geographical domain also removes complexity of choosing consistent spatial dimensions between training and testing. In summary, the inclusion of the entire prediction domain removes variability in model performance due to initial model design decisions.

In order to introduce some spatial independence between training and testing, we exclude certain stations from training. This allows us to evaluate model performance when previously unseen station data is included. The stations that are included during training and validation are sampled by weight of their local predictability. By sampling more heavily at stations with lower predictability, we ensure that the model can predict well in more complex spaces without overfitting, due to the increased general prediction difficulty in those regions. The end goal is for the difficulty of prediction in training to be similar in testing, and that the prediction difficulty is generally consistent across the entire grid.

Ground truth

To evaluate performance, we propose using an augmented ERA-5 grid as ground truth. To create an augmented grid, we interpolate our selected stations with ERA-5 data, excluding stations already included in the reanalysis. While ERA-5 reanalysis product provides consistent grid coverage across many years, it is still an imperfect representation of ground truth. In an operational setting, more observational data is available than what is included in ERA-5 reanalyses. Therefore, we can use ERA-5 to fill observational gaps, rather than to provide the ground truth entirely.

Summary

This work mainly focuses on the data preparation phase, which is a critical and time-consuming portion of machine learning model development. It is part of a larger effort to develop proof-of-concepts for proposed standards in ML implementation for operational meteorology, which will include criteria for computational requirements, model evaluation, and explainable AI (XAI).