Building Infrastructure to Support the Next-generation Joint Effort for Data Assimilation Integration (JEDI) System for NOAA, NASA, U.S. Air Force, U.S. Navy, and UK Met Office

The Joint Effort for Data Assimilation Integration (JEDI) provides a data assimilation framework for Earth system prediction. JEDI is a collaborative project run by the Joint Center for Satellite Data Assimilation (JCSDA) supported by NOAA, NASA, U.S. Air Force, U.S. Navy, and UK Met Office. The interagency partnership is key to transition data assimilation research to operational modeling systems and academic communities. This transition is possible through robust computational infrastructure, comprehensive testing, open-source software, and agile development.

JEDI requires a large number of software packages to build and run experiments using several forecast models such as the Unified Forecast System (UFS), the Goddard Earth Observing System (GEOS), the Modular Ocean Model (MOM6), the Model for Prediction across Scales (MPAS), the Navy Environmental Prediction sysTem Utilizing the NUMA corE (NEPTUNE), and the Met Office LFRic. In order to support multiple configurations of the JEDI software on High Performance Computing systems, commercial clouds, workstations, and laptops, a package management software stack (spack-stack) developed by JCSDA, the NOAA Environmental Modeling Center (EMC) and the U.S. Earth Prediction Innovation Center (EPIC) is used.

Built on top of spack-stack, JCSDA’s Skylab Earth System Data Assimilation is an end-to-end system which features the following JEDI components: the Experiments and Workflow Orchestration Kit (EWOK), a data store for observation and model data called the Research Repository for Data and Diagnostics (R2D2), and an Interface for Observation Data Assimilation (IODA) to handle observational data. Skylab utilizes JEDI to model the atmosphere, ocean, sea-ice, soil moisture, snow, aerosols, and trace gasses. With a system this advanced, continuous integration and automated testing is key to rapid and effective code development at the research and production levels.

This presentation covers the JEDI infrastructure team’s development efforts for the next generation data assimilation system by leveraging cloud computing environments for research, development, and near real-time applications of JEDI. The development of a Continuous Integration/Continuous Delivery (CI/CD) pipeline using tools such as GitHub, Docker containers, various Amazon Web Services (AWS), and CodeCov enables the rapid testing and implementation of emerging technologies. The future of data assimilation lies in the ability to support new software environments and integrate new datasets in a ready to use format for research and operations in a matter of days.