Extremely High-Resolution Weather Model Simulation, Data Processing, and Visualization
Advanced memory management techniques, deliberate I/O workflow, and sophisticated data processing and visualization techniques are well designed for this research based on existing infrastructures and available resources. During the simulation, we collected system-wide performance data, especially the memory usage information during our WRF runs to understand the memory usage pattern of the model. We then reconfigured the SLURM scheduler on the Stampede supercomputer and carefully managed the memory allocation to satisfy all memory requirements through the work. In addition, we took advantage of the local disk space and Stampede's Lustre parallel file system, then created an optimized WRF I/O workflow. Our optimized parallel I/O workflow dramatically raised the I/O performance and cut the load associated with metadata requests of the shared file system. Furthermore, we also scheduled thousands of sequential jobs to regroup and merge intermediate results and created target data files for visualization. The target data files were then converted to the VTK format and loaded into Paraview for visualization.
The resolution of our simulation in time and space is beyond almost all similar weather simulations as we are aware of. Benefiting from the crucial resolution, meteorologists are able to observe subtle weather changes at local areas. Furthermore, all these techniques are applicable to a great deal of memory-intensive and/or I/O-intensive applications, high-resolution simulations, and supercomputer platforms.
Our simulation results are being compared with other observational and computational results by our Raytheon and NCAR collaborators for further validation. Some follow-on studies will be performed over other domains of our interests. We will also investigate WRF's performance benefits from Intel Xeon Phi coprocessors as well as advanced parallel I/O applications in the near future.