Schedule WRF Model executions in parallel computing enviroments using Python

WRF, Weather Research and Forecasting (About the Weather Research & Forecasting Model), is an non-hydrostatic atmospheric simulation model for a limited area, sensitive to the characteristics of the terrain, and designed to forecast atmospheric circulation on synoptic, mesoscale, and regional scales. It was developed in collaboration with the NOAA (National Oceanic and Atmospheric Administration), the NCAR (National Centers for Atmospheric Research), and other organizations.

The implementation of the model is prepared to work in parallel computing environments with shared memory, via OpenMP, and with distributed memory, via MPI. Additionally, the model has the capacity to combine both technologies.

The WRF model is composed of a series of modules (User's Guide for the Advanced Research WRF (ARW) Modeling System). Each module corresponds to a different function: GEOGRID allows for the configuration of the geographic area of study. UNGRIB prepares the data for the initialization of the model, which is normally established by the output of another model with greater spatial coverage. METGRID prepares the boundary conditions, adapting itself to the characteristics of the domains defined in GEOGRID. REAL does a vertical interpolation from the pressure levels to a system of normalized Sigma coordinates. The WRF model does the physical forecasting and diagnostic equations that allow for a forecast with a predetermined temporal horizon.

The WRF model is designed so that each module must be executed independently. This provides many advantages, especially if it is only used for case studies. However, if the objective is to plan its execution in order to obtain forecasts with a specific time period, it can be inconvenient. The Group for Atmospheric Physics (GAP) at the University of León (ULE), uses the model every day in order to get a 48-hour forecast.

It is important to point out that there are no tools that allow for planning executions. This makes centers that want to use the model periodically define their own solutions ad hoc. In order to do so, it might be necessary to have at least some basic knowledge of programming, especially if the objective is to make use of its capacity to run parallel computations.

The GAP has implemented a tool, using Python, which resolves this problem and that can be configured by any researcher without previous programming knowledge. This tool guarantees the optimization of resources in a parallel computing environment. This can be very important, since the access to this type of environment is usually limited, and often, it can incur high economic costs.

Normally, access to these environments is controlled by a line administrator, who is usually in charge of sending jobs and reserving the necessary nodes to run them in a parallel computing cluster. The GAP carries out its forecasts in a cluster from the Central Supercomputing Foundation of Castilla y León (CSFCL), using up to 360 nodes in some cases. The CSFCL uses SGE (Sun Grid Engine) to manage sending their jobs to Caléndula, its parallel computing cluster.

On the other hand, the most common way—which is also the least efficient—of executing the model in a parallel computing environment is to send only one job to the line administrator. This model does not guarantee a complete optimization of resources, since not all of the modules of the WRF model are prepared to be executed simultaneously. Usually, only the REAL and WRF models are done this way, since executing both of these modules implies the largest part of overall time that the model is run. As such, it is necessary to point out that during the execution of the other models, only one of the reserved nodes is used. This means that during a period of time, some nodes in the cluster are unoccupied. This period of time can be relatively large, especially if the post-processing is complex. For example, in the executions programmed by the GAP, post-processing has an average execution time of more than 40 minutes, because multivariable meteorological graphics in different instances are generated.

The tool developed by the GAP optimizes the use of parallel resources. Thus, instead of sending one job to the line administrator, a job for each module is sent, plus an extra job for the post-processing of output. In this way, it is possible to indicate that the line administrator should only reserve the exact number of nodes needed in order to carry out each job, which presents important advantages, as previously shown.

References

The Weather Research & Forecasting Model: About the Weather Research & Forecasting Model. [Available online at http://www.wrf-model.org/.]

WRF users page: User's Guide for the Advanced Research WRF (ARW) Modeling System Version 3.4. [Available online at http://www.mmm.ucar.edu/wrf/users/docs/user_guide_V3/contents.html.]

Acknowledgements

The authors would like to thank the Junta de Castilla y León for their economic support via the LE176A11-2 Project.