A highly parallel algorithm for turbulence simulations in planetary boundary layers: Results with meshes up to 20483

Large-scale parallel computing has altered the landscape of turbulence simulations in planetary boundary layers. Increased computer power using O(100-10000) processors has the potential to allow large-eddy simulations (LESs) of turbulent flows in more realistic outdoor environments, for example, flow over hills, atmosphere-land interactions, boundary layers with surface water wave effects, and weakly stable nocturnal flows. Here we describe a recently developed massively parallel algorithm for simulating incompressible Boussinesq atmospheric and oceanic boundary layers. The spatial discretization is conventional - second-order finite differences in the vertical direction and pseudospectral in horizontal planes. Typically with this flow model and numerical algorithm, the number of processors is limited to the maximum number of vertical levels. This is dictated by the elliptic pressure Poisson equation and the global character of spectral (Fast Fourier Transform) derivatives. Our new algorithm overcomes this limitation by decomposing the boundary-layer domain into rectangular prisms that span the x direction, but are split across y and z directions. The advantages of this 2-D decomposition are: (1) a large number of processors can be used; and (2) the solution of the pressure Poisson equation can be arranged so that no global all-to-all matrix transposes are required. Only local transposes f(x,ys:ye,zs:ze) <=> f(y,xs:xe,zs:ze) and f(y,xs:xe,zs:ze) <=> f(z,xs:xe,ys:ye) are needed. The parallelization is accomplished using the message passing interface (MPI) paradigm with MPI input/output employed in order to make the code robust across different machine architectures. We exercise this algorithm by first performing LES of the familiar convective atmospheric boundary layer. The finest resolution LES uses 8192 processors of a Cray XT4 for meshes with 2048³ gridpoints with spatial discretization dx = dy ~ 2.5m and dz ~ 1m. Flow visualization and statistics are compared for resolutions varying from 100³ to 2048³.