A Multi-modal, AI-ready Dataset of Atmospheric Observations for ML Applications

A major factor driving the contemporary revolution in data-driven weather modeling (“AI-NWP”) is the availability of a high-quality, complete reanalysis dataset (the ECMWF Reanalysis v5, or ERA5) of weather over the past several decades. Despite its large volume, ERA5 is composed of, effectively, a single, gridded dataset of relevant atmospheric variables for a large variety of ML tasks. Furthermore, ERA5 is readily available – both in legacy, GRIB format from the Copernicus Data Store, as well as in cloud-optimized versions, such as the ARCO-ERA5 published by Google Research. All of these details lower the bar necessary for outsiders to the meteorology research community to adopt it for model development, evaluation, and a variety of other technical applications.

However, ERA5 has limitations, and a growing research community is turning to work directly with observational datasets to bolster their model development and evaluation activities. These observations include a wide variety of different sensing types of modalities, ranging from in situ surface stations, radiosondes, and aircraft data, to remotely sensed data across the electromagnetic spectrum, retrieved from terrestrial systems to spaceborne platforms in both low-Earth and geosynchronous orbit. In contrast with ERA5, these observations come in many different formats, shapes, and sizes, and come from as many disparate sources as there are observation systems. Consolidating a useful cross-section of these observations is a significant task, and the lack of such a comprehensive dataset is a major impediment towards leveraging the significant utility of observations in the development of data-driven weather models. The contribution of a common observational dataset – and related models and metrics built on top of it – will catalyze innovation in the data-driven weather modeling space.

Here, we present a new, multi-modal, “AI-ready” dataset consolidating a diverse selection of atmospheric observations in an archive tailored for interfacing with machine learning applications. The archive is based on the NOAA-NASA Joint Archive (NNJA) of Observations for Earth System Reanalysis, which bolsters its suitability for these tasks. Importantly, we have re-processed this archive in a modern, cloud-native format, established infrastructure for easily and efficiently interfacing and interacting with its contents, and taken care to abide by the recommendations of the ESIP Data Readiness Cluster’s recommendations for Open Environmental Datasets. The archive is backed by a large, structured collection of Parquet files, and features both data lake and cataloging tools to present a single, unified interface for downstream users.

To demonstrate the opportunities to which this new archive may cater, we present both simple demo applications involving exploratory data analysis as well as pipelines which interface with AI training and inference systems. We further discuss opportunities to develop a real-time complement to our archival system, which we hope could serve as the foundation for a class of operational, next-generation, observation-based, data-driven weather models.