Surface data integration at NOAA's National Climatic Data Center: data format, processing, QC, and product generation

The National Oceanic and Atmospheric Administration's (NOAA) National Climatic Data Center (NCDC) acquires, quality controls, archives and provides dissemination services for global meteorological and climatological data. These data are processed with automated Quality Assurance/Control (QA/QC) software, with some of the software being network-specific. For example, the NOAA Automated Surface Observing System (ASOS) Network, National Cooperative Observers Network (COOP), and Climate Reference Network (CRN) are processed using systems unique to their network. Additionally, some of the data undergo additional interactive QC, which involves “visual/manual” inspection of the data.

While similar, QA/QC rules and algorithms for like parameters from different observing networks are sometimes not standardized. NCDC's goal is to integrate surface data into a standard format, and process the data through standardized QA/QC algorithms and procedures. To that end, NCDC has developed a new integrated surface database, called Integrated Surface Data (ISD). To date, numerous historical datasets have been integrated into ISD, with others to follow. Also, NCDC is designing a new QA/QC system - Integrated Surface Data Processing System (ISDPS), as an end-to-end system for processing in-situ data, where QA/QC is standardized, network independent and based on reporting frequency (hourly, daily, etc.). Together, ISDPS and ISD integrate the QA/QC algorithms into a unified system, use ISD as the input/output format, and provide integrated online products to customers.

This paper briefly describes the ISDPS/ISD methodologies, data format, QC/validation techniques, and the various products available to customers.