Evaluation of Runtime Performance and Fine-scale Forecasting Potential of the NCAR FastEddy® GPU-Resident LES Model on NOAA's Hera GPU Cluster

Large Eddy Simulation (LES) of the atmosphere has traditionally been reserved for idealized academic explorations of fine-scale turbulence characteristics or as a high-resolution standard comparison for the development of parameterization schemes for models with much coarser grid scales. Real-case simulations were generally considered to be extremely difficult to attain due to the large amount of computational resources needed, a lack of fine-scale initial conditions with turbulent statistical information, and the need for significant amounts of patience due to the long runtimes generally required even on large HPC systems. Traditional CPU-based clusters were unable to make significant strides in carrying out simulations at LES resolutions on the order of tens of meters or less at a rate quick enough to be useful for short-term forecasting while also preserving the fine-scale turbulent structures desired. Users oftentimes opted to sacrifice turbulent flow detail by choosing lower grid resolutions so as to attain results more quickly.

To address these challenges and take advantage of the immense computing power of modern GPU accelerators, a team at NCAR developed the FastEddy® model for running LES on Nvidia GPUs. The increase in computing performance has been cited as 6X or more versus the previous CPU models. This has opened an opportunity for LES to be used as a limited-domain forecasting model for the first time over areas where high turbulent detail would be beneficial for understanding complex flow features. Such domains could include urban centers, wind farms, complex terrain, airports, wildfires, and more.

Realizing this potential, FastEddy® was recently evaluated at NOAA’s Global Systems Laboratory simply on the basis of finding configurations which could be used for short-term forecasting. Evaluations were done using the Hera GPU cluster with configurations of up to 256 Nvidia P100 GPUs at once. Results showed that for certain domain configurations up to 5*10⁷ gridpoints at 50m resolution, FastEddy® was able to perform calculations at a rate supportive of short-term forecasting, even with the P100 GPUs and interconnect systems that are several compute hardware generations old at this point. The authors consider these results to be very encouraging, especially as Hera is expected to be refreshed with more modern accelerators in the near future. The results presented here from the evaluation could be used as a baseline for future evaluations on the newer hardware, and projections of expected performance after the Hera refresh will be presented as well.