Analyzing Big Data Produced by Mesoscale-Domain Large-Eddy Simulation

Mesoscale-domain large eddy simulation (LES) has a numerical domain large enough to include mesoscale flows and resolution fine enough to explicitly resolve energy- and flux-containing turbulent eddies. In order to study the temporal evolution of the atmospheric boundary layer (ABL) in the afternoon and the early evening over heterogeneous surface conditions, we run mesoscale-domain LES, which has horizontal grid spacing of 50 m and mean vertical grid spacing of 25 m over a horizontal domain of 36 km by 36 km with a model height of 7 km. Thus, the grid number of this mesoscale-domain LES is 720 x 720 x 280 = 1.452 x 10⁸. In order to temporally resolve turbulence, ABL variables at these grids need to be printed out at least every minute, which produces an about 300-GB data file every 30 min in the Weather Forecasting and Research (WRF) model. For the 10-h integration (for example, from 11 a.m. to 21 p.m.), the WRF model produces a 6000 GB (or 6 TB) data file for a case. For 10 cases, it needs 600 TB. Exploring the big data face challenges. This presentation will talk about challenges and our solutions to those.