Leveraging Container Virtualization to Improve Algorithm and Model Processing Portability

Compute-intensive weather processing is often very sensitive to its running environment, which makes transitioning from one environment to another very time consuming.  Often, applications are required to be rebuilt from scratch, which also can be time consuming.  Different environments can include variations in OS, library versions, etc., which may lead to changes in output values as well.  Virtualization provides the promise of portability by including the complete environment in a virtual space, but at the cost of performance overhead.  Containers offer a lightweight virtualization solution that enables applications to execute in their own virtual space while reducing memory footprint, increasing startup times, and reducing processing overhead over hypervisor-based virtualization.  Our research evaluates this technology to find the advantages and challenges in using containers in compute-intensive processing, from a single node system to distributed compute environments found in High Performance Computing (HPC) and cloud based systems.