NetCDF-4 Performance Improvements Opening Complex Data Files

The netCDF-4/HDF5 format offers many advantages for science data users, including built-in compression, a rich data model, and support for parallel I/O. However, several performance bottlenecks have been identified which impact complex files with many variables and attributes defined. These files are especially prevalent in the domain of High Performance Computing (HPC). The performance issues appear when files include thousands or tens of thousands of variables, attributes, or dimensions, and include slow file opens. A focus on netCDF-4 performance issues has resulted in the elimination of several bottlenecks, including time to open a file. Significant improvement is demonstrated. Performance trials are conducted on NOAA’s Thea platform. Results are presented demonstrating the improved performance with real-world example files.