Performance Portable Fortran Code Using Kokkos: A Case Study with the YSU PBL Scheme

Traditionally, multiple source code implementations of a single algorithm or software package for CPUs and each individual vendor-dependent framework for GPUs is created, with each implementation tuned for maximum performance using different methods tailored to the hardware architecture. While good for performance, this method is costly in terms of development time as developers must become experts on each execution target on which they wish to implement their algorithm, costly in terms of long-term support as the number of potential bugs increases for each new implementation, and costly in terms of future improvements which must be implemented not only once, but across all implementations for each execution target. Realizing these costs, many developers choose not to embark on this path, but instead, choose to develop only for what they consider to be the most “popular” execution targets. However, this decision – if made by enough large and influential software development teams – can result in a different problem termed as “vendor-lock”, where a particular product or line of products from a specific manufacturer becomes the de-facto standard for the large swath of people who are users of those packages, potentially resulting in higher costs to obtain the necessary hardware due to inflated demand and reduced competition.

Performance portable code has the potential to alleviate these issues as GPUs become ever more present in new and planned HPC systems of all sizes. Performance portability aims to facilitate single-source code implementations that can be execution hardware independent. Rather than requiring the use of vendor-proprietary libraries directly in the code of an algorithm, performance portable libraries can be used which can later be targeted to be built for a specific execution architecture at compile time. Therefore, there would be no need to retool the algorithm at the code level to achieve best performance on the underlying execution target, whether that be a CPU, a GPU, or some combination of the two.

As a simple first step to explore the potential of this for the future of numerical weather prediction models which heavily depend upon high performance computing resources and are gradually increasing their utilization of GPUs, we attempt to implement the YSU boundary layer parameterization scheme in a performance portable manner. For this effort, we use Kokkos, a C++ performance portability library developed under the DOE Exascale Compute Project umbrella along with the Kokkos Fortran Language Compatibility Layer (FLCL). This way, we hope to maintain the YSU Fortran code’s overall structure while offloading the computation-heavy portions to the GPU in a performance-portable manner.

This presentation will not only provide a brief introduction to Kokkos and consider the final performance results of this effort, but will also look at the many challenges encountered along the way so as to provide developers considering this option for their code with useful insight.