Climate and Weather Forecasting Models in a Virtual Scalable High Performance Computing Environment

 Geophysical Fluid Dynamics (GFD) simulations such as large scale climate modelling or weather forecasting simulations are usually run on High Performance Computing Clusters (HPC) in Academic (University based) or Governmental Organizations. These platforms usually have a rigid structure where competition for resources are in high demand. Jobs are submitted to a scheduler which run jobs based on funding priority and resource allocation. Recent interest has turned to running HPC as a Service (HPCaaS or Haas) similar to the notion of Software as Service (SaaS). SaaS has spawned a commercial market where software is provided on demand so that a software application can be run on a cloud based service such as Amazon Web Services (AWS) and accessed through a thin client such as a web interface. Likewise, commercial services that provide HaaS have begun to appear to fill a niche market. Recent interest in virtualization through containerization has created huge interest from the public and private sector in container technologies such as Docker. The idea of compartmentalizing a separate application that forms part of a software stack and scaling these applications out on an “as demand basis” has led to the development of orchestration technologies (e.g. Docker Swarm, Kubernates, Nomad, etc.). Orchestration has evolved to enable technology companies to address consumer needs on an on demand basis, instantly scaling up to meet heavy loads. More traditional schedulers like SLURM, HTCondor and TORQUE are used in HPC environments to delegate jobs in an efficient manner. GFD codes are often based on older codes which use the highly venerable Message Passing Interface (MPI) and OpenMP libraries to allow communication between different shared memory and distributed memory tasks assigned to large GFD simulations which require 1000s of cores. In this talk I will present an application which combines these technologies to run a climate model simulation and numerical weather prediction simulation in an on demand scalable manner. A test case will be presented through containerization of the code and an orchestrated deployment of the application on a virtual HPC environment running on the AWS cloud computing platform using the technologies described above.