Leveraging Cloud Computing and Software Container Technologies to Create a Portable End-to-End Numerical Weather Prediction System

Numerical Weather Prediction (NWP) is at the core of weather research, modeling, and forecasting. Yet, major hurdles can exist in the pursuit of running full end-to-end NWP systems. First, building the software systems necessary to perform NWP requires extensive effort and knowledge to ensure all dependencies, libraries, and codes are properly installed. This is non-trivial, and procedures and dependencies vary across different platforms, often requiring substantial time to accomplish the desired setup. Assuming all components are successfully built, the second obstacle is making sure adequate computer resources (processors, memory, storage, etc.) are available to perform the desired simulations.

Two technological innovations are becoming increasingly popular to help overcome these hurdles in running NWP systems: containerized software and cloud computing. Containerized software allows for all the components (including the operating system, source code, library dependencies, and executables) required to build and run a software system to be bundled together and deployed on any platform. Cloud computing provides computing resources, big or small, to be available to anyone, anywhere. Combining the accessibility of cloud computing platforms with the portability of software containers offers an NWP solution extendable to a wide range of audiences and promotes cross-community research collaborations and interactions.

Motivated by the advantages these two technologies afford, the Developmental Testbed Center (DTC) has extended the previous work of the Big Weather Web project that established containers for the Weather Research and Forecasting (WRF) Pre-processing System (WPS), WRF model, and NCAR Command Language (NCL) by adding several new containerized components, including the Gridpoint Statistical Interpolation system (GSI), Unified Post Processor (UPP), Model Evaluation Tools (MET), and the METviewer database and display software systems. The end result is a portable end-to-end NWP system comprised of essential software containers that can be deployed on any platform, including the cloud, without the typical setup time and frustration of building the software packages from the ground up. In addition, the DTC has created a comprehensive online tutorial to guide users through building and running example case studies on both local machines and the Amazon Web Service (AWS), as well as instructions for adapting the containers for customized research needs.

This presentation will provide the motivation for using NWP containers and cloud resources, an overview and update on the current DTC containerized end-to-end NWP system, and highlights from several usage examples.