Collaborations for the Exploration of the Use of GPU Accelerated Computing for Weather and Climate Community

How to move to GPUs and take advantage of their compute power to achieve maximum performance on current and future High-Performance Computing (HPC) systems for weather and climate simulation while keeping portability and maintainability of the code? This question is common as the proportion of the supercomputer equipped with GPU is growing because this type of accelerator allows reaching the highest peak FLOPS on LINPACK Benchmark used to rank the fastest supercomputer of the world. But, are these accelerators useful for the weather and climate community, which largely depend on complex, legacy memory-bandwidth-bound legacy Fortran codes? Is it worth porting or re-coding applications? What is the best approach, the best programming language?

We present a set of collaborations between domain scientists and HPC experts that address these questions. In these collaborative projects, set up with some of the most important weather and climate centers in the EU, we explored different approaches on real weather and climate applications. One is porting a legacy code from scratch with directives or with source-to-source transformations by hand or with tools to automate part or full of the process. Another is to optimize GPU code already ported with directives or GPU languages. A third is porting code from OpenACC to OpenMP offload to unlock vendor-agnostic portability.

We describe some selected collaborative projects from different frameworks: the objectives, the work done, the obtained results and the lesson learnt.

First, in the Centre of Excellence (CoE) in Simulation of Weather and Climate in Europe, named ESiWACE, a service is proposed where Research Software Engineers (RSEs) offer their expertise to climate and earth system modeling groups across Europe. Within 6-month collaborative projects, the RSEs intend to provide guidance and advice regarding the performance, portability to new architectures, and scalability of selected applications including for GPU accelerated computing. In this context, the ocean models FESOM2, OGSTM,the Regional Climate Model RegCM and the atmospheric large-eddy model DALES have been ported to GPU with OpenACC directives. In addition, EMAC, RTE-RRTMGP-C++ and MicroHH GPU codes have been optimized, the latter with the help of Kernel Tuner auto-tuning tool. In addition, in this CoE, NEMO ocean model is ported to GPUs using a source to source approach with PSyclone, a code generation system developed to support domain-specific languages (DSLs) for finite element, finite volume and finite difference codes extended to support the processing of existing finite difference codes such as NEMO.

Second, in direct collaborations between weather and climate centers and vendor’s performance programming experts. Here, for example, the Center of Excellence in Weather & Climate Modelling created jointly by ECMWF and Eviden which looks among other subjects to develop a CPU-GPU-based version of ECMWF’s Integrated Forecasting System (IFS). In this context, we worked on the ecRad radiation kernel from IFS to study which loop transformation can make the code more GPU friendly to then develop a source-to-source transformation with Loki. We also port the dwarf-p-cloudsc kernel which is intended to test the CLOUDSC cloud microphysics scheme of the IFS from OpenACC to OpenMP.

We show that these collaborations fosters the exchange of expertise, enabling researchers and industry to be more productive and paving the way to scientific excellence. This is why collaboration is key.