Chiltepin: A Federated Workflow System For Numerical Weather Prediction

Robust automation of complex numerical weather prediction (NWP) workflows is essential for both conducting earth system modeling research and for the reliable delivery of operational forecasts that serve the public and protect life and property. The workflows used in those two closely related, but distinct, pursuits are similar in their structure and share many of the same constituent components and requirements. However, the research and operational workflow regimes also have important differences arising from their different missions and goals. In the case of workflows used to research application of new high performance computing (HPC) tools and technologies for NWP, many of these differences manifest as extreme computational scales and use of emerging technologies unsuitable for an operational environment. The problems of scale and diversity naturally lead to a requirement for federated workflows that effectively harness the computational power of a diverse set of resources distributed both geographically and across multiple administrative domains.

To address gaps and enable pioneering HPC research for NWP, we have established a collaborative project to develop a next generation federated NWP workflow system called Chiltepin. This presentation will provide an update on our progress, goals, and challenges with the development of Chiltepin.

This work is conducted under the auspices of the NOAA Software Engineering for Novel Architectures (SENA) project in collaboration with the Parsl and Globus teams at the University of Chicago and Parallel Works.