Anemoi: an Open-Source framework for developing data-driven weather forecasts

With the active collaboration and leadership of its Member States and Cooperating States, ECMWF is developing an open-source framework designed to build, train, and run data-driven weather forecasts. The framework is called Anemoi after the collective names of the gods of winds in Ancient Greece. The main components are:

• anemoi-datasets: code to build training datasets from various sources and formats, such as GRIB, NetCDF and any formats supported by Xarray;
• anemoi-models: code for creating modular neural network architectures, with a focus on graph and transformer designs. This features gradient checkpointing and model sharding for memory efficient and scalable training;
• anemoi-graphs: code to build graphs from any grids, to be used by graph neural networks;
• anemoi-training: code to train models, in multi-nodes multi-GPU environments, including deterministic and probabilistic training, with callbacks for evaluating, plotting and logging intermediate results during training;
• anemoi-inference: code for run the trained models in inference mode, using the output of physics-based NPW analyses or ensembles, as initial conditions;
• anemoi-registry: code to manage a catalogue of training datasets, model configurations, weights, results of model evaluations, etc, with a strong emphasis on recording lineage dependencies.

The goal of the project is to support the development of global weather forecasting systems, such as AIFS, as well as high-resolution limited area models. Training is currently mostly done on meteorological fields, but developments are ongoing to support observations. Furthermore, support for ocean, land and hydrology models (funded by the Destination Earth programme of the European Commission) is in progress.

The project is monitored by a technical group composed of experts from the Centre’s Members States and Cooperating States.

The code is developed in Python, available under the Apache 2.0 Open-Source licence.