CLUBB GPUization and Performance Portability

The Community Earth System Model (CESM) and the Energy Exascale Earth System Model (E3SM) utilize a cloud and turbulence parameterization called CLUBB (Cloud Layers Unified by Binormals). CLUBB operates by taking a set of fields, discretized over the vertical dimension of the atmosphere, and producing time-advanced predictions of those fields. To leverage the emerging GPU power employed by modern computing systems, CLUBB has undergone a significant refactoring process. As a result, the entire CLUBB codebase is now capable of running on GPUs, and this new version will be available in the next release of CESM. The refactoring process involved several design decisions that influenced the required effort, verification methods, the final outcomes, and the subsequent steps.

CLUBB is written in Fortran, so the first design consideration was the choice of directives. To accelerate computations on GPUs, we opted for OpenACC directives instead of OpenMP target offloading directives. However, with the assistance of an Intel-developed tool, we can convert the entire codebase from OpenACC to OpenMP in seconds. Another major consideration before initiating the refactoring process was effectively distinguishing small numerical differences from refactoring errors. Instead of relying on traditional debugging methods when issues arose, we developed a quick test which proved capable of identifying nearly every bug.

Timing results were gathered using a standalone model for development, where CLUBB was run with non-interactive columns, as well as in global CESM runs with various configurations. The CESM timing results provide insight into the speedup achievable when running on GPUs with OpenACC compared to CPUs. However, the speedup is highly dependent on the choice of model resolution and the ratio of CPU cores to GPUs, necessitating careful configuration. Additionally, it is challenging to find a balanced configuration that allows the GPU-ported code to execute efficiently without degrading the performance of sections not yet ported.

The simplicity of the standalone model enabled us to test the code more extensively on multiple systems, including DOE's Frontier. Beyond CPU versus GPU comparisons, we also gathered timing results to compare the GPU-accelerated runtime when using OpenACC offloading versus OpenMP offloading, the Cray Fortran compiler versus the Nvidia Fortran compiler, and the Nvidia A100 versus the AMD MI250X.