Towards a FAIRest Copernicus Climate and Atmosphere Data Store

The modernized Copernicus Climate and Atmosphere Data Store (CADS) is the backbone software infrastructure supporting the implementation of the Copernicus Services entrusted to ECMWF by the European Commission. The infrastructure exposes to the public two different instances, the Climate Data Store (CDS) as support of the Climate Change Service (C3S) and the Atmosphere Data Store (ADS) for the Atmosphere Monitoring Service (CAMS) when sharing all the backend components. CADS Infrastructure also provides dedicated instances for the implementation of the Copernicus Thematic Hubs on Energy (implemented by C3S) and Health (implemented by CAMS) in the form of catalogues of references for data and services across Copernicus which are of relevance for the respective domains. Combined CDS and ADS provide service to more than 250k registered users, delivering over 120 TBs of data on daily average in the form of more than 700k processed requests.

CADS Software Infrastructure is designed as a distributed system and an open framework providing improved access and a single-entry point to a wide catalogue of datasets, applications and other digital information throughout harmonized web-based and programmatic interfaces which rely on a variety of adaptors, standards and protocols. Available datasets are distributed and located at different data suppliers. Currently CDS catalogue provides access to 141 datasets including reanalysis, seasonal, projections and satellite ECVs when ADS catalogue offers 13 entries about Atmosphere Composition, Air Quality Reanalysis and Forecast among others. CDS catalogue portfolio also includes dataset from the Copernicus Emergency Management Service (CEMS). Dedicated resources for an evaluation and quality control function (EQC) provide users with the assurances required to use the data products available in operational systems.

Simplicity and consistency are key pillars of the CADS architecture to facilitate a seamless experience on the user journey to explore, discover, retrieve and use available resources. To assure that, a well-defined, rigorous and iterative content integration and management methodology has been set up. This is supported by automatic deployment tools and a set of configuration files stored in a centralized version control repository. Configuration files cover different aspects of the resource definition from descriptive and extended metadata to interactive forms, dynamic constraints, documentation, adaptors and quality control. This versatile content management and deployment mechanism provides huge flexibility to accommodate changes, adapt for compliance with different standards and to facilitate interoperability with third party platforms and services.

Versatility is essential for CADS to continuously evolve and adapt on a changing landscape in where it has became a main player in the global ecosystem of environmental data and services. CADS already has strong interdependencies with other European Platforms as the WEKEO Data and Information Access Platform (DIAS), Climate-Adapt or ClimSA Station and it is condition to easily integrate with others being a main building block for the implementation of the coming Green Deal Data Space as support of the EU Green Deal Objectives.

At the core of this versatility and adaptability capabilities is the adoption of FAIR principles (Findable, Accessible, Interoperable, Reusable) across different components of the infrastructure since the conceptual definition of the architecture. CADS adherence to FAIR principles follows a gradual and systematic approach which keeps the platform and its content under continuous evolution and improvement.

Furthermore, looking to engage with a broader user community and assure compatibility with state-of-the-art solutions, the modernized CADS has embraced an open-source approach to post-processing and visualisation via an easy-to-use online JupyterHub based environment and contributions to python packages dedicated to climate and meteorological purposes. This allows users to perform computationally expensive analysis and produce demonstrative visualisations without having to download large quantities of data. In addition, the JupyterHub environment provides an educational platform via a large library of training material which has been developed by experts in the field and aims to inform users how to access and handle the data effectively.

Here we describe the journey to the CADS modernized infrastructure, its functional and technical improvements and overall the objectives that have driven this modernization effort towards a more Open, Flexible and FAIRest Copernicus Climate and Atmosphere Data Store Infrastructure.