Research Repository for Data and Diagnostics (R2D2): A Standardized Online Data Management Service for JEDI-Skylab

To harness the full potential of ever-increasing volumes of data from new and evolving Earth observation systems, a new online database software system has been developed and deployed by the Joint Center for Satellite Data Assimilation (JCSDA). As a core component of the JCSDA’s flagship application, JEDI-Skylab, the Research Repository for Data and Diagnostics (R2D2) performs data registration, management, and configuration services for high performance computing and cloud-based data assimilation (DA) computational workflows. R2D2 satisfies these requirements by offering a centralized, model agnostic platform for handling diverse DA datasets and computational experiment registrations and associated metadata. Implemented in python as a multi-tier, client-server system, R2D2 integrates a series of distributed, independently administered data hubs by indexing holdings according to an SQL database schema and enabling robust end user access through an intuitive client API. R2D2’s data server utilizes the OpenAPI Specification to expose a standardized REST HTTP API, uniquely define this interface to the consumer, validate incoming requests, and automatically generate much of the client API codebase. Deployed using the Flask micro web framework, Docker, a MySQL database, and Amazon’s API Gateway, the server is manageable, maintainable, scalable, and portable. We present an overview of R2D2’s architecture, deployment, and design, as well as utilization in JEDI-Skylab.