Since their public opening back in 2018, the Climate and Atmosphere Data Stores demonstrated as a sound concept with an expanding catalogue of data and services, a growing community of users and a continuous functional evolution driven by incoming user requirements and technical adaptations. The key factors that have fostered this sustained evolution were 1) the capacity of all the different layers of the infrastructure to growth organically, from the underlaying In-house Cloud to the Software Components, 2) its capacity to accommodate incoming requirements due to a plug-in architecture and open technologies and 3) the centralized and automatic management and deployment of configuration and catalogued content.
After years in operation, the DSs reached a mature status and embarked on a deep modernization process all-across the different layers of the infrastructure. This transversal modernization aimed to capitalize experience and lesson-learned, overcome the obsolescence of former components, ensure compatibility with state-of-the-art solutions, integrate cutting-edge technologies, embrace a cloud oriented and open-development approach and set up a FAIRest system able to strengthen synergies with other platforms and user communities. Modernization encompassed:
- Cloud Layer: A newly In-House Common Cloud Infrastructure (CCI) provided by ECMWF as part of its IT Service provision strategy which, together with other services as the European Weather Cloud (EWC), host and run the DSs. This In-House Cloud provides elasticity of computing and storage resources to scale up the Services and to allocate resources to 3rd parties working closely with the DSs. Moving into CCI simplified the technical landscape fostering the synergies, federation and distributed computing with other Cloud Infrastructures.
- Data layer: The foundational base of the Data Stores, this layer split on:
-
- External Data (hosted outside the CCI), ranging from federated and brokered datasets supported by third party organizations to the ECMWF MARS Archive, where flagship datasets, such as Global Reanalysis, are primarily hosted.
-
- Internal Data (hosted within CCI). Managed as part of the DSs. The most representative is the on-line MARS where a subset of the most requested variables from the core MARS archive are regularly uploaded. It also includes an Observations repository, other small on-disk datasets, and an experimental ARCO (Analysis Ready, Cloud Optimized) data lake where data is stored on data cubes (Zarr, STAC) chunked by Geographical and Temporal dimensions.
- System layer (core engine): Software components which deploy and run together to full fill the wide range of functional capabilities of the DSs for a seamless user journey for searching, discovering, sub-setting, retrieving, processing and visualizing data via interactive and programmatic interfaces. Components of this layer:
-
- Oversees the full life-cycle of incoming data requests.
- Host the catalogue of resources, including datasets metadata and configuration.
- Implement Quality of Service rules (QoS) to administer limits, priorities and user preferences when accessing catalogued resources.
- Dispatch requests to the Data Layer via dedicated adaptors which transform incoming requests into the different formats and protocols understood by data sources.
- Exposes interactive and programmatic interfaces (APIs) highly configurable and compliant with standards (OGC API, WMTS, CSW, STAC).
- Support the Evaluation and Quality Control (EQC) function.
- Integrate a JupyterHub that provides a Python development environment preconfigured to work with DSs (CDSAPI, Eartkit).
- Business Layer: This is a technically thin but visually prominent layer that results from the combination of operational deployments of the System Layer with tailored catalogued content and configuration. This layer split in two: Operational Capabilities and “Branded by” Interfaces (eg. Climate DS, Atmosphere DS)
The detachment between the System Layer and the Business Layer provides great flexibility to instantiate for multiple purposes, deploy in different platforms or run as backend engine “powering” other platforms.
Catalogue content and configuration are populated via automatic deployment tools following strict integration and stewardship procedures where simplicity and consistency are key factors.
On the periphery of these core layers, but closely interlinked with them, there are other components which are “powered by” or “complementary to” the DSs. Of special relevance are the following:
- earthkit ( https://earthkit.readthedocs.io/en/latest/ ): The open evolution of the CDS Tools. ECMWF open-source Python code repository offering a broad set of expert libraries optimised to deploy and run on the CCI-DSs Jupyter Environment and to work with ECMWF and Data Stores data resources.
- Visual Interactive Content (VICs): this includes a broad set of user-oriented applications and training material that showcase or make use of the full range of data resources and capabilities of the Data Stores and earthkit. VICs can deploy and run anywhere but preferable on CCI-DS to optimize resources.
- External platforms: Platforms and infrastructures “powered by” the Data Stores. These platforms may interact with or consume data resources via the exposed interfaces, embed technical components or integrate VICs as part of their portfolio. The Copernicus WEkEO ( https://wekeo.eu/ ) DIAS Platform, a partnership of ECMWF, EUMETSAT, Mercator Ocean and EEA, can be cited as a representative example of an external platform “powered by” the Data Stores.
In a nutshell the aim of this abstract is to introduce the modernized Data Stores, the driven principles behind the modernization and to describe how modernization has encompassed across different layers and components, highlighting the new features that leverage DSs to “power” other platforms.
https://cds-beta.climate.copernicus.eu/
- Indicates paper has been withdrawn from meeting
- Indicates an Award Winner