Extended Abstract (100K)7.2
New techniques in quality assurance of hourly meteorological data: Resolving multiple flags through a decision tree
Daniel Y. Graybeal, Cornell University, Ithaca, NY; and A. T. DeGaetano and K. L. Eggleston
As part of the NOAA Climate Database Modernization Program, historical (pre-1949) hourly surface airways observations have been digitized and will be released in early 2004. An important component of data set production has been improving and implementing quality assurance (QA) procedures. Both the framework, known as Complex QA, and its individual component checks, such as for dew point depression being inconsistent with diurnal temperature range, have been improved for historical hourly surface meteorological data. In comparison with individual component checks recently developed for mesoscale observing networks, range, internal, and temporal consistency checks previously employed for QA of these data were rudimentary and often subjective. Therefore, one main drive in improving QA has been in scientific development of individual component checks and thresholds they use for flagging. Development of these thresholds in light of probabilistic models and target flag rates based in experience is detailed.
To judge from among the various component flags thrown which element(s) to flag in the target database and what code level to use (e.g., "Good", "Suspect", or "Erroneous"), a decision tree was developed. All component flags thrown are considered and weighed before it makes a decision. For example, if flags from more than one component type are thrown on an element, such as violations against both limits and temporal consistency, that element becomes a candidate for an "Erroneous" assignment. Application of this implementation of Complex QA will be discussed, along with lessons learned, particularly about the continuing need for manual intervention.
Session 7, Data Reliability, Quality Assessment and Usability (Room 619/620)
Thursday, 15 January 2004, 1:30 PM-5:30 PM, Room 619/620
Previous paper Next paper