Implementing new quality control and processing systems for hourly precipitation data

For decades NOAA's National Climatic Data Center (NCDC) has collected, quality controlled, and archived data from the COOP Fischer & Porter (F&P) network of more than 2000 stations. This dataset, commonly referred to as DSI-3240, is produced through ongoing quality control processing that includes extensive manual review and intervention by a trained meteorological technician. Not only does this require extensive resources in time and personnel, it results in delays of up to six months before a month of observations are fully quality controlled and available for public use. It also relies on subjective assessments, the unique skills of meteorological technicians, and can result in inconsistent evaluations and determinations of data quality through time. Now using new methods of automated quality control and by combining the F&P data with hourly observations from other networks (e.g. ASOS, USCRN, NWS Hydrometeorological Automated Data System), a new system of automated quality control processes are being developed.

The new system is based upon data collection and quality control principles developed and applied as part of NCDC's Global Historical Climatology Network-Daily (GHCN-D) dataset. Advantages to this method include the removal of the subjective component intrinsic to any process with a human interface. This method also provides a consistent set of quality control checks throughout the period of record, instead of antecedent practices that introduce new quality measures at various times throughout the period of record. Most importantly, the ability to process the entire period of record makes it possible to apply quality control retrospectively as new methods are developed and to do so in a consistent manner throughout the life of the data.

Although this system is not fully developed, an experimental dataset containing the more than 2000 F&P stations along with thousands of additional stations with hourly observations is available online at http://www.ncdc.noaa.gov/hrlyhpd/ These data have been quality controlled using a set of checks including checks for spikes, global extremes, gaps, and climatological outliers. Efforts are ongoing to implement a complete suite of quality control procedures developed through empirical assessments of false positive and flag rates. Once completed these new quality control procedures will replace the current process of manual review and editing which is part of the DSI-3240 Hourly Precipitation Dataset. This paper will provide a synopsis of the new processing system, including data ingest, as well as an extensive overview of the current and proposed quality control algorithms.