Using an AI-based Open-Architecture and Cloud-Agnostic Framework to Deliver Weather Data Faster

Meteorological data will continue to proliferate as the number of sensors and satellites grows, leading to an exponential increase in data volumes processed over the next 25 years. The volume and increasing complexity of this data requires higher agility in data processing infrastructure to gain insights at reduced costs, including rapid adoption of new technologies like scalable Cloud infrastructure, artificial intelligence (AI) and machine learning (ML), and deep learning (DL) in combination with modern IT engineering techniques and open-source tools.

NOAA NESDIS is progressing the use of these advanced technologies through a technical AI-enabled Cloud data processing and dissemination prototype. This prototype explores the infrastructure, tools, IT security, science infusion, and services needed to facilitate data processing and product generation, collaborative science development and its integration into NESDIS operational enterprise services. The solution demonstrates cloud infrastructure scalability, including processing via low-latency data pipelines; the ability to collaboratively implement and execute algorithms and AI/ML models; and improved accessibility to datasets and models through a modular, open architecture framework, and compatibility with and augmentation of the NESDIS Common Cloud Framework (NCCF).

The prototype solution consists of open-source tools and a full platform solution with hosting and security services with multi-cloud capabilities. The Kubernetes-based hosting environment supports containerized workloads and provides a highly available, scalable, and resilient architecture. The AI Operations framework provides executable architecture for implementing dataflows and AI/ML solutions. Highlighted performance benchmarks including latency, simultaneous users, and running simultaneous algorithms in both exploratory and operational contexts will showcase techniques for achieving rapid scalability and highlight the potential for an open architecture to improve science workflows and support scientific algorithm innovation. Use cases will include the streamlining of satellite and High-Resolution Rapid Refresh model data among others for the prototype demonstration.

This presentation will explore the modern technologies and frameworks that support meteorological data ingestion, processing, and dissemination, including:

• Cloud-agnostic, secure infrastructure frameworks that demonstrate scalability for data ingestion and algorithm processing for data product generation
• Cloud-based algorithm development and data processing solutions that accelerate data product generation to increase and democratize innovation