ECMWF Software & Services for Producing and Serving Peta-byte Earth System Data Sets – a Journey of Bytes from HPC to End-Users

Numerical Weather Prediction (NWP) produces massive volumes of output data. ECMWF's current operational forecasts generate 320 TiB per day in 4 one-hour time-critical bursts. The size of these datasets is only expected to grow in the future, with anticipated increases in resolution as well as an increased range of parameters being output. Within the European Union Destination Earth initiative, we are planning for handling 1 PiB of data output per day. Upcoming ML-based forecasts have the potential to output even more data in shorter time windows, increasing the challenges associated with I/O.

Building on decades of experience in handling high-density meteorological datasets, ECMWF has developed a holistic approach to managing, processing, and delivering Earth system data at scale. This presentation will provide an overview of ECMWF’s software ecosystem, which includes production pipelines, data access services, and data analysis tools that collectively ensure users have seamless access to high-quality, high-volume Earth system data. This ecosystem is at the basis of ECMWF’s data services provided in the European Union Destination Earth (DestinE) initiative, for which ECMWF needed to further develop its software components to cope with the diversity of users and the high-fidelity nature of the data.

We call this collection of components 'ECMWF Software Engine' and at the core of this ecosystem is ECMWF’s approach to scientific data management, centred on semantically driven datacubes that enable efficient, context-aware data retrieval. By integrating these datacubes into our in-memory, high-efficiency pipelines, we significantly reduce data redundancy and latency, enabling faster and more precise data delivery. Crucially, our data services also provide interfaces compliant with international standards, such as OGC APIs, ensuring broad interoperability and ease of integration into existing systems and workflows.

An integral part of the ECMWF's Software Engine is Earthkit, an open-source comprehensive software stack designed to lower the barriers to accessing, analysing and visualising Earth system data. Earthkit not only simplifies data access but also supports ECMWF’s machine learning developments, such as the Anemoi framework, by facilitating data preparation for the learning process. Users can seamlessly extract and transform the data needed to train models, enabling advanced predictive capabilities and more sophisticated analysis. The latest implementation of the Copernicus Data Store is powered by Earthkit.

This presentation will demonstrate how these innovations, alongside the feature extraction capabilities introduced last year through the Polytope concept, are improving the way scientists and stakeholders interact with meteorological data. Attendees will gain insights into how ECMWF’s Software Engine is supporting users through improved data accessibility, efficiency, and scientific accuracy in the Earth system sciences.