Containerization of Weather Forecasting Platforms: Benefits and Challenges

IBM's Deep Thunder is a high spatial- and temporal-resolution weather forecasting system, part of which is based on the ARW core of the Weather Research and Forecasting (WRF) model. WRF, its dependencies, and the other components that form the Deep Thunder system are non-trivial to build, install, maintain and deploy. To alleviate these problems we have produced a development operations (DevOps) pipeline to automate the deployment of containerised versions of Deep Thunder.  This is particularly important because enterprise processing and data storage are increasingly moving to the cloud. For our research we have begun performing large scale multi-year hind-casts and simulations for diverse applications on cloud computing resources. Given the scale and complexity of such large runs in the cloud, it is essential to rely on reliable tools and a process design that is robust and scales well with the number of jobs deployed.  In this work we describe our implementation, use of emerging technologies (such as Docker, Object Storage, and cloud computing), and the benefits and constraints that this solution brings in both a research and commercial environment.

The pipeline presented here takes advantage of modern agile and DevOps tools and methodologies to allow us to automate, distribute, control and communicate more effectively.  WRF and its dependencies comprise a rich set of tools that are challenging to build in a consistent manner given their interdependencies and associated libraries.  It is therefore important to be able to build the toolset in a well-defined manner that is reproducible and has no hidden dependencies on a local workstation that could unknowingly alter its run-time behavior.  Containers such as Docker allow the build and test of the pipeline to be automated and operate in a completely controlled environment with no contamination from local system state, such as libraries or customization.  Working in a clean container alleviates the need for developers to take care to ensure the build/run environment is exactly as required for correct and optimal performance. Containers also speed up the process of development by reducing the time it took when adding a new feature and testing its results.   Distribution of the final runtime environment is enabled through a docker registry for effortless updates of existing deployments to the most recent version, with integrity enforced through the communication of version-stamped images.

It is now possible for researchers not only to present or record the results of an experiment, but also deliver the exact version of the application they used and the exact version of the data they relied upon to create those results.  Existing deployments are often in physically diverse locations with heterogenous hardware, base operating systems, and configurations.   Utilisation of a common registry enables control through deploying identical versions en-mass to diverse systems with a single command.  We found that the use of continuous integration coupled with automated testing, containerisation and container distribution technology has transformed the distribution of both applications and data - and is essential for large scale deployments to heterogeneous environments.