The pipeline presented here takes advantage of modern agile and DevOps tools and methodologies to allow us to automate, distribute, control and communicate more effectively. WRF and its dependencies comprise a rich set of tools that are challenging to build in a consistent manner given their interdependencies and associated libraries. It is therefore important to be able to build the toolset in a well-defined manner that is reproducible and has no hidden dependencies on a local workstation that could unknowingly alter its run-time behavior. Containers such as Docker allow the build and test of the pipeline to be automated and operate in a completely controlled environment with no contamination from local system state, such as libraries or customization. Working in a clean container alleviates the need for developers to take care to ensure the build/run environment is exactly as required for correct and optimal performance. Containers also speed up the process of development by reducing the time it took when adding a new feature and testing its results. Distribution of the final runtime environment is enabled through a docker registry for effortless updates of existing deployments to the most recent version, with integrity enforced through the communication of version-stamped images.
It is now possible for researchers not only to present or record the results of an experiment, but also deliver the exact version of the application they used and the exact version of the data they relied upon to create those results. Existing deployments are often in physically diverse locations with heterogenous hardware, base operating systems, and configurations. Utilisation of a common registry enables control through deploying identical versions en-mass to diverse systems with a single command. We found that the use of continuous integration coupled with automated testing, containerisation and container distribution technology has transformed the distribution of both applications and data - and is essential for large scale deployments to heterogeneous environments.