Adapting to a Changing Environment: Automating the Transition to Operations for Scientific Applications

In the dynamic realm of environmental science, the National Environmental Satellite, Data, and Information Service (NESDIS) has embarked on a transformative journey in response to the evolving landscape. Over the past several years, NESDIS has witnessed a substantial IT transition, particularly within the NESDIS Ground Enterprise (NGE) by embracing cloud technologies and beginning its migration into the NESDIS Common Cloud Framework.

In response to this paradigm, NESDIS recognizes the imperative to revolutionize its business operations strategy and is moving away from siloed systems towards a centralized processing system specifically designed to run scientific applications with complex dependency trees.

This pivotal solution is the Algorithm Orchestration (AO) System—a collaborative effort between NESDIS and the Algorithm Support for Science, Integration, and Technical Transition Team (ASSISTT). This innovative system is being developed as a scalable, fault-tolerant, and data-agnostic service that empowers scientists to schedule and execute complex science applications. These applications, rife with intricate dependencies and production rules, require adept orchestration and management to yield accurate science products.

The AO system will provide scientists with the ability to test and validate their algorithms using near real-time and batch processing in a operation-like environment. AO will expand the Scientists research to operations (R2O) capabilities to compare one version of their application versus another to determine if they have made desired improvements.

Furthermore, the AO System catalyzes Continuous Integration/Continuous Deployment (CI/CD), enabling on-demand deployment of applications post rigorous testing. This ensures that applications maintain a deployable state through rigorous performance, integration, and additional tests, safeguarding operational readiness. In a desired CI/CD pipeline, the science application is registered to AO in CI pipeline, then the CD testing is followed using AO with its convenient deployment capabilities. This guarantees science applications are always in a deployable state through CI/CD pipelines.

As the AO System evolves and matures, NESDIS is committed to extending access to the NESDIS Consolidated Cloud Framework and the AO system. This strategic move aims to accommodate a broader spectrum of users, both within and beyond the National Oceanic and Atmospheric Administration (NOAA) community. By doing so, NESDIS envisages a scalable infrastructure that adeptly addresses present and future requirements for generating products that underpin environmental science.