Creating a Community Dataset for High-Speed National Water Model Data Access

The National Water Model (NWM) input and output datasets, made freely available on cloud platforms through the NOAA Open Data Dissemination program, represent a treasure of hydrologic information to support research and accelerate improvements in continental-scale hydrologic prediction. However, early efforts by working groups within the Cooperative Institute for Research to Operations (CIROH) identified significant limitations in efficiently accessing and querying the NWM retrospective and operational datasets on these platforms. These limitations are primarily due to the format and size of the datasets. The existing format of the NWM files reflects a reasonably efficient configuration for operational simulations, but the cloud-stored files become cumbersome when used for case studies and other retrospective analyses common in research scenarios. In 2023, CIROH reported on a series of experiments exploring methods to overcome these critical obstacles and facilitate versatile access to the NWM data. Those experiments pointed to the potential for using the Kerchunk library to build a companion dataset of json-based reference files that provide pre-computed indexes to the variables and data layout in the NWM cloud file locations, significantly reducing the seek time for accessing subsets of data from the NWM files. Although a slightly more performant approach would be to create a complete, re-formatted version of the NWM datasets, the Kerchunk-produced reference files provide a significant benefit while using several orders of magnitude less space.

Building on those experiments, CIROH has developed a community-focused dataset to accompany the NWM files that, when combined with simple methods of parallel computing, allows flexible, high-speed access to the NWM operational and retrospective datasets. While this auxiliary reference dataset will be of great value to the hydrologic research community, the methods may also be a guide for provisioning output datasets in future operational hydrology modeling programs. We present here the details of the development of this new dataset along with examples of applications in model evaluation and input preparation for retrospective simulations. We invite the community to explore the dataset and example applications, and share feedback at dataaccess.ciroh.org.