Building Scalable, Cloud-Native Systems for Low-Latency Weather Forecasting at Zeus AI

Advancements in AI have revolutionized many aspects of weather modeling, paving the way for more accurate and timely delivery of environmental intelligence. Deploying AI-based tools on commercial cloud platforms presents a set of challenges in terms of data management and scalability, which come in addition to the ongoing challenge of demonstrating the trustworthiness of data-driven solutions. At Zeus AI, a NASA SBIR-funded startup, we are learning from vast amounts of open scientific data on the cloud to deliver data assimilation and weather forecast products via our Low-latency Environmental prediction from Neural Systems (LENS) platform. We use cloud-native data formats and a variety of cloud services for data management, model training and validation, real-time production, and distribution to end users. Our pipeline on the cloud demonstrates how an efficient system can be built to scale up high spatiotemporal resolution (2km / 15 minute), high refresh rate (4x per hour), and low latency (<5 minutes) weather forecasting to near-global coverage. We seek to establish trustworthiness by continuous verification using feeds of ground-based observations as well as validation in specific use cases related to energy, aviation, and more. Finally, challenges will be discussed around the use and production of petabyte-scale datasets and validation of sparsely observed weather variables.