Surface data integration at NOAA's National Climatic Data Center: data format, processing, QC, and product generation
While similar, QA/QC rules and algorithms for like parameters from different observing networks are sometimes not standardized. NCDC's goal is to integrate surface data into a standard format, and process the data through standardized QA/QC algorithms and procedures. To that end, NCDC has developed a new integrated surface database, called Integrated Surface Data (ISD). To date, numerous historical datasets have been integrated into ISD, with others to follow. Also, NCDC is designing a new QA/QC system - Integrated Surface Data Processing System (ISDPS), as an end-to-end system for processing in-situ data, where QA/QC is standardized, network independent and based on reporting frequency (hourly, daily, etc.). Together, ISDPS and ISD integrate the QA/QC algorithms into a unified system, use ISD as the input/output format, and provide integrated online products to customers.
This paper briefly describes the ISDPS/ISD methodologies, data format, QC/validation techniques, and the various products available to customers.