A Python-Driven Workflow to Automate and Process Many Numerical Simulations

Research in the atmospheric and oceanic sciences can often require the creation and processing of large volumes of data — and big data can mean big problems. Repetitive manual submission of HPC jobs is not only tiresome and prone to error, but also inefficient in taking advantage of resources during a cluster’s quiet overnight hours. Furthermore, large output datafiles may need to move to separate storage facilities, adding another repetitive task. Here, a workflow is presented whereby Python scripts start, monitor, and clean up after hundreds of Weather Research and Forecast (WRF) model simulations, as an extension of the WRF Ensemble Management (WEM) open-source package. These methods use the Python Standard Library to maximize portability.  Other WEM features include the automation of the WRF pre-processing system (WPS), and a suite of methods to evaluate numerical output that utilise the additional numpy, netcdf4-python, and matplotlib/basemap Python packages.