Incorporating data-denial statistics into real-time quality control filtering of surface observations

The Weather Channel in Atlanta, Georgia (TWC) and Weather Services International Corporation in Andover, Massachusetts (WSI) jointly developed a system called HiRAD (High Resolution Aggregate Data) that produces synthetic current conditions or surface weather conditions for any arbitrary point within the conterminous United States (CONUS). Concurrently, a shadow HiRAD system produces synthetic surface weather conditions that differ from the primary system only as a result of the systematic withholding of input surface observations or METARs from the core system. The output of this alternative process is called the data-denial result, which is used directly or comparatively as validation information to infer skill, variability and bias of the HiRAD estimation techniques. The data-denial version of HiRAD runs each hour and contributes to a continuous and growing statistical database containing valuable information about the spatial and temporal behavior of the HiRAD surface analysis error field.

A crucial need for HiRAD or any assimilation system is robust quality control and filtering of incoming observation reports. While the incidence of bad observations is low with such data sources as ASOS and AWOS METAR reports, appreciable and egregious observation errors do occur across every important variable. These errors can be very large or quite persistent (i.e. the bad reports continue over many hours or even days); conversely, they can be subtle and very difficult to discriminate from so-called good observations. Fixed interval checks, spatial buddy checks, and temporal consistency checks are all well documented techniques for filtering bad observations. However, each approach has inherent weaknesses or involves costly calculation overheads – especially when measured against the desire of rapid processing and near real-time publication.

TWC recently completed a functional enhancement to HiRAD that includes a novel observation filtering technique called data-denial enhanced quality control or DDEQC. In this approach, interval checks for bad observations are distinct and varying for each observation point and each observation time. The variable intervals are based in part on the evolving database of data-denial results accumulating in the HiRAD system. These data-denial results serve as standard mean errors for a given point and time based on the accumulated knowledge of the nominal separation between the data-denial and primary versions of the observation. Thus, interval-based filtering for a point with large data-denial standard errors will result in conservative filtering or flagging of a bad observation. Conversely, a point with low data-denial standard error signals the system to be more aggressive in withholding a suspicious observation value.

Currently, this DDEQC approach is used for temperature, dew point, wind speed, and wind gusts and acts on all surface observations flowing into the HiRAD system. Results to date are very encouraging. The DDEQC approach does not replace all other observation quality monitoring techniques, but can be a useful additional test when judging observation quality and confirming suspicious observations.

This paper and oral presentation will split time between describing the DDEQC technique, presenting results from six months of operational use, and discussing the future directions and areas of continued work on quality control and observation filtering within the HiRAD environment.