Prototyping Diversified Exascale Numerical Weather Prediction Workflows

Robust automation of complex numerical weather prediction (NWP) workflows is essential for both conducting earth system modeling research and for the reliable delivery of operational forecasts that serve the public and protect life and property. The workflows used in those two closely related, but distinct, pursuits are similar in their structure and share many of the same constituent components and requirements. However, the research and operational workflow regimes also have important differences arising from their different missions and goals. In the case of workflows used to research application of new high performance computing (HPC) tools and technologies for NWP, many of these differences manifest as extreme computational scales and use of emerging technologies unsuitable for an operational environment. Workflow efforts at NOAA have primarily focused on the needs of the NWP research to operations (R2O) pipeline and are subject to the rigid constraints of the existing operational framework. To address gaps and enable pioneering HPC research for NWP, we have established a collaborative project to develop a next generation NWP workflow system responsive to the advanced research needs of exascale and beyond. This work is conducted under the auspices of the NOAA Software Engineering for Novel Architectures (SENA) project in collaboration with the Department of Energy’s Exascale Computing Project (ECP) and Parallel Works.

This presentation will discuss our vision, progress, and challenges towards development of a diversified workflow system that can straddle on-premise and cloud-based computing resources and accommodate both high-performance and high-throughput oriented tasks while managing the associated data flows.