The Python-Based MPMC Test Suite for NOAA Operational Data Assimilation Systems (GSI/EnKF)

The DTC (Developmental Testbed Center) data assimilation group provides community support for NOAA operational data assimilation systems (GSI/EnKF). It is very important to keep a community software compatible, portable, robust to a wide range of computing platforms for different community users. This is not an easy task as GSI/EnKF has been adopted by lots of different applications, such as single observation assimilation experiments, 3DVAR, hybrid 3DEnVAR, hybrid 4DEnVAR, EnKF, chemical data assimilation as well as both global and regional applications. In the past, the DTC data assimilation group would spend several months to manually set up and compile GSI/EnKF on different computing platforms with different combination of Fortran compilers (such as Intel, PGI, GNU and different versions), and MPI libraries (Intel MPI, OpenMPI, mvapich2, etc), run selected test cases to make sure the latest GSI/EnKF codes from repository work correctly for community users before making a public release. The process involves tedious work and it is prone to make mistakes during the process.

To alleviate the workload and improve/enhance the test functions, the DTC data assimilation group developed an automatic test suite based on Python 3. This suite will test a given version of GSI/EnKF software automatically for all available combinations of compilers, MPI libraries, NetCDF libraries and Math Libraries. It automatically loads required modules on a given HPC or the GSI docker container, and tries to compile GSI/EnKF executables and related community tools. If compilation succeeds, for each of the above combination environments, the test suite will launch 15 community GSI/EnKF case tests to check whether the compiled executables have run-time issues for various applications. A utility is also provided to automatically check compiling and running results and report them in a user-friendly online table (https://dtcenter.org/com-GSI/MPMC/). If any problems are reported and require attention from developers, one can easily go to corresponding subdirectories to do further debugging work.

The test suite greatly accelerated the public release of GSIv3.7_EnKFv1.3 and helped DTC DA group to review NOAA Gerrit tickets for GSI/EnKF code commits. The test suite is now included in the master branch of the official NOAA GSI/EnKF repository and available for any developers to use. The usage is very simplified so that a user can easily run the test suite with minimum efforts. For advanced users, command line options are also available for some customized functionalities.