An existing code of a large-eddy simulation (LES) model for the study of turbulent processes in the atmospheric and oceanic boundary layer has been completely recoded for use on massively parallel systems. The new PArallelized LES Model (PALM) is designed to investigate open problems in boundary layer research where very high grid resolutions and/or large computational domains are necessary, for instance interactions between turbulent structures of different scales. Typical examples are cold-air outbreak flows, where large scale organized convection simultaneously occurs together with the usual small-scale convective plumes, or flows around obstacles, where the large eddies downstream of the obstacle have their origin in small-scale turbulence generated at the sharp edges of the obstacle.

Parallelization is achieved by two-dimensional domain decomposition in both horizontal directions, which has an advantage over the simpler one-dimensional decomposition since it ensures scalability even on machines with very large numbers of processors (>1000). PALM uses finite differencing since the numerous calculations of horizontal Fourier transforms necessary when using pseudospectral differencing would significantly decrease the model performance in case of a 2D-decomposition. Communication is realized by the message passing interface (MPI). With respect to local data dependencies caused e.g. by the central finite differences, additional layers of ghost points have been introduced at the side boundaries of the subdomains. Furthermore, tranpositions of 3D arrays are performed in order to treat the non-local data dependencies resulting from FFTs and tridiagonal sets of equations, which have to be solved as part of several numerical procedures (e.g. to solve the Poisson-equation for pressure). This transposition technique allows the usage of standard non-parallelized FFTs and tridiagonal solvers. The model also alternatively allows to solve the Poisson-equation with a parallelized multigrid method, which is faster than the transposition-FFT-method in case of subdomains with more than 64x64 gridpoints in the horizontal directions.

The model parallelizes very well and shows good performance on distributed memory systems (SGI/Cray-T3E, Compaq Alpha Server ES45) as well as on shared memory or clustered systems (SGI-ORIGIN 3800, IBM pSeries SP). An almost linear speedup is achieved up to the maximum available number of 512 PEs. The largest runs performed with the model so far used about 620x620x620 grid points, which seems to be the actual world record in resolution for LES models. Simulations with up to 1500x1500x1500 grid points will be possible on new hardware available in 6/2002.

The model compares well with results from other state-of-the-art LES models for the stratocumulus-capped convective boundary layer. It has recently been applied successfully to study boundary layer turbulence above inhomogeneous terrain, organized convection during cold air outbreaks and the turbulent flow in the vicinity of Arctic leads.