A Scalable Implementation of Ensemble Data Assimilation for Very Large Models

The Data Assimilation Research Testbed (DART) is a freely available community facility for ensemble data assimilation. DART is designed to run efficiently on a wide range of computing platforms from laptops to the largest supercomputers. DART also supports a wide range of prediction models from low-order dynamical systems up to coupled climate system models at approximately 1/10 degree resolution. Previous releases of DART implemented a scalable ensemble Kalman filter that relied on the assumption that a whole model state vector could fit into the memory of a single process. This assumption is no longer valid because model resolution is continually increasing while per-process memory is, in general, decreasing. In the new DART implementation described here, local array access is replaced with an off-core memory retrieval using passive target one-sided MPI communication. This has removed the hard limit on state size while leaving the scalable assimilation algorithm untouched. Runtimes are comparable between the current and new versions of the code, however the new code can support much larger models. The DART IO has been modified to read/write and distribute appropriately-sized hyperslabs of data that fit into available per-process memory. However, IO at an acceptable level of performance for very large models is still an unsolved problem. A brief summary of the use of GPUs to accelerate DART performance is also included.