Reprocessing the GOES-R ABI L1b Data Record

The GOES-R Series Advanced Baseline Imager (ABI) provides critical near-real-time imagery of Earth’s oceans, land and atmosphere to the National Weather Service and many other users. To maintain product quality, calibration and operations teams continually optimize the instrument calibration over the course of the mission through updates to algorithms, look-up tables (LUTs), and data formats. While these updates support the program’s mission, they come at a cost of image artifacts, calibration discontinuities, and other anomalies in the historical data record. In the interest of creating a cohesive data record, a NOAA/NASA Geostationary Imager Support Team (GIST) was formed and began work on developing and implementing a capability to reprocess ABI Level-1b (L1b) products from the raw Level-0 (L0) data.

The GIST team completed updates and enhancements to the GOES-R Program’s existing in-house L0-L1b software suite, the GOES-R ABI Trending and Data Analysis Toolkit (GRATDAT). Among these enhancements include preserving pixels at the edge of the earth, the addition of scan line start/end times, and significant code optimizations that reduce processing time. The team then deployed this Reprocessing variant of GRATDAT, rGRATDAT, to the NOAA NESDIS Cloud-sandbox Infrastructure Service (NCIS) as a way to distribute and parallelize processing. The NCIS cloud, hosted on Amazon Web Services (AWS), provides a scalable high-performance computing environment that provides 5000+ cores that can be optimized for memory, compute speed, and other parameters. Jobs were distributed across the cluster through in-house scripts that manage each step of the processing. The system was designed to optimize computing resources and time to reprocess the entire catalogue of GOES-16 and -17 ABI L1b products. Validation tools were developed to ensure the quality of both radiometric and geometric calibration, and quality-control scripts were written to look for any data gaps that might arise. Currently, reprocessing of the ~600 TB of L0 data is underway, with initial validation results indicating good data quality. As data is completed and ready for release it will be made available to the general public. This talk will provide an overview of the NCIS cloud architecture, including lessons learned that help continually optimize a system capable of large-scale reprocessing. It also will provide an update to the progress towards reprocessing all of GOES-16 and -17 ABI L1b, details on where users can access the data, a look at results comparing the reprocessed data with the operational products, and a sample of data validation results.