Earthkit - ECMWF’s Next-Generation Community Python Software Stack

ECMWF has over 30 years of experience in developing and providing software tools to enable users and services to access, process and visualize meteorological and climate data. A recent initiative to refactor ECMWF’s software stack for the purposes of performance and maintenance, plus a continued drive for open development has led to the creation of a new Python-based software toolset called Earthkit.

Earthkit is part of a collection of software components 'ECMWF Software Engine' and at the core of this ecosystem is ECMWF’s approach to scientific data management, centred on semantically driven datacubes that enable efficient, context-aware data access, processing and visualisation capabilities. The 'ECMWF Software Engine' is the software stack that powers the new Copernicus Data Store (hosting ERA5), the Destination Earth Digital Twins and data provision and the core data services of ECMWF NWP mission.

This presentation will provide an overview of the rationale for Earthkit, its architecture, and its applications to data from the fields of numerical weather prediction and climate science. Attendees will learn about the ways that Earthkit can provide efficient access to data, harmonize the representation of different data formats and enable weather and climate scientists to work seamlessly with and visualize diverse data sets.

With Earthkit we aim to (1) reuse existing community software stack where appropriate (e.g. xarray, matplotlib, pandas); (2) provide users with familiar and interoperable interfaces with community standards (OGC EDR, CovJson, ...); (3) provide scalable solutions that match the needs of large-scale operational environments, with ability to operate with Petabyte datasets; (4) engage with the community of ECMWF end-users and Member and Cooperating states to improve user experience and finally (5) provide a modern software stack that integrates well with Machine Learning stacks and helps to drive innovation.

Details on selected Earthkit components will be given, plus examples of their use in activities such as ECMWF’s core post-processing operations, Machine Learning, the Copernicus Climate Data Store and Destination Earth. Various additional benefits will be emphasized, including those of Open Development and the value of using the same software toolkit to quickly transition from research and development to operations.