Handout (1.2 MB)
Development of GEFS v12 faces numerous challenges including new capabilities, coupling with wave and chemistry models, extended scope of products, requirements for 30-year reforecasting and 3-year retrospective forecast. With these challenges and the limitation of computational resources, the development has to be carried out on different platforms, including NCEP operational machines, NOAA research machines and even the Cloud. To ensure timely progress of the project against these challenges, simplifying and optimizing workflow management is one of the critical tasks in the development and implementation of GEFS v12. We tackle this task by employing PYTHON and taking a practical, operation-oriented and step-by-step approach. A "workflow" module has been developed in the GEFS code repository with high portability, flexible expandability and easy accessibility by users. With minimum human resources, this module grew up from infancy to maturity and supported GEFS v12 development at the various stages, including configuration, scientific validation, computational optimization, as well as the reforecast and retrospective sub-projects. It will continue to grow and support the inclusion of the wave ensemble and the chemistry model, and the code delivery to NCO for implementation.
In this article, the structure of this PYTHON based workflow management module will be described and some of its basic capabilities, including rocoto and ecflow workflow, as well as visualization of job flow will be demonstrated. We believe it will provide useful information and reference for other similar complex forecasting and analysis systems.
- Indicates paper has been withdrawn from meeting
- Indicates an Award Winner