CROW: A Common Toolchain for Developing Environmental Forecast Workflows in Research and Operational Environments

Within NCEP, and among our collaborators, there is an overabundance of workflows. By and large, people are reimplementing the same functionality many different ways. This leads to scientists losing much of their time maintaining workflow scripts rather than doing actual science. It also leads to less-reliable, operational, workflows because these ad-hoc systems are frequently filled with flaws, and are maintained by few people, or even one person (or fewer). Striving to eliminate this problem, we created a common toolchain that makes it easier to create workflows in a common manner.

This system, the Community Research and Operational Workflow (CROW), makes use of industry standard technology and modern software design principles to create a highly configurable system that is easy to use. We will demonstrate functionality of critical subsystems that allow tracking of dataflow, cross-platform means to execute programs and batch jobs, a workflow definition system that can support multiple batch systems and metaschedulers, and a powerful, YAML-based, configuration system. The internal implementation is in Python, but the system is accessible via APIs from other languages. Our reference implementation is the FV3 GFS, the next-generation global model in NCEP, and will be demonstrated in this presentation.