Refinement of Horizontal Resolution in Dynamical Downscaling of Climate Information Using WRF: Costs, Benefits, and Lessons Learned

Dynamical downscaling techniques have previously been developed by the U.S. Environmental Protection Agency (EPA) using a nested WRF at 108- and 36-km. Subsequent work extended one-way nesting down to 12-km resolution. Recently, the EPA Office of Research and Development used computing facilities at NASA's National Center for Climate Simulation (NCCS) to perform a 108/36/12-km two-way nested dynamical downscaling application of WRF based on a 2025-2035 dataset from NASA's Model-E2 climate model (Schmidt et al., 2014). As a base case, a 108/36-km simulation was also performed to measure the effect of feedback from the 12-km nest. However, the primary intent was to investigate changes in the magnitude of simulated extreme events when finer horizontal resolution is used.

Refinement of the WRF horizontal modeling grid from 36 to 12 km required a significant amount of additional CPU time and data storage capacity. Whereas previous 36-km simulations were performed using no more than 128 Intel E5-2670 (Sandy Bridge) compute cores on EPA's computing system, this 12-km simulation was expected to be calculated on 1000 or more similar cores. However, it was discovered that our WRF configuration using spectral nudging toward the ModelE-2 data did not scale well beyond approximately 560 cores at NASA's NCCS where Intel E5-2697v3 (Haswell) processors were used. The 11-year continuous simulation required almost 2 months to complete and generated almost 33 TB of output data.

Results show that the 12-km resolution did not affect the magnitude of extremes in any general sense, but there were a variety of locations sampled where the 36-km resolution did not accurately represent the true terrain height and/or surface type, but the 12-km grid did, and the results were significantly different. And of course the 12-km simulation exhibited finer texture in all climate variables examined. In addition to standard WRF output (wrfout files), climate diagnostic "wrfxtrm" files were also produced. These showed excessively high surface skin temperatures and 2-m air temperatures for individual grid cells in a sporadic nature. This behavior had not been previously noticed in standard hourly outputs. It appears that extremely high surface skin temperatures are occurring in the simulation that last only one time step and are only noticeable in hourly output if that time step is at the top of the hour. These indications point to a possible numerical instability in the calculation of these variables in the Noah land surface model that was employed for this study. Investigate of these positive temperature excursions is ongoing.