Incorporating data-denial statistics into real-time quality control filtering of surface observations
A crucial need for HiRAD or any assimilation system is robust quality control and filtering of incoming observation reports. While the incidence of bad observations is low with such data sources as ASOS and AWOS METAR reports, appreciable and egregious observation errors do occur across every important variable. These errors can be very large or quite persistent (i.e. the bad reports continue over many hours or even days); conversely, they can be subtle and very difficult to discriminate from so-called good observations. Fixed interval checks, spatial buddy checks, and temporal consistency checks are all well documented techniques for filtering bad observations. However, each approach has inherent weaknesses or involves costly calculation overheads – especially when measured against the desire of rapid processing and near real-time publication.
TWC recently completed a functional enhancement to HiRAD that includes a novel observation filtering technique called data-denial enhanced quality control or DDEQC. In this approach, interval checks for bad observations are distinct and varying for each observation point and each observation time. The variable intervals are based in part on the evolving database of data-denial results accumulating in the HiRAD system. These data-denial results serve as standard mean errors for a given point and time based on the accumulated knowledge of the nominal separation between the data-denial and primary versions of the observation. Thus, interval-based filtering for a point with large data-denial standard errors will result in conservative filtering or flagging of a bad observation. Conversely, a point with low data-denial standard error signals the system to be more aggressive in withholding a suspicious observation value.
Currently, this DDEQC approach is used for temperature, dew point, wind speed, and wind gusts and acts on all surface observations flowing into the HiRAD system. Results to date are very encouraging. The DDEQC approach does not replace all other observation quality monitoring techniques, but can be a useful additional test when judging observation quality and confirming suspicious observations.
This paper and oral presentation will split time between describing the DDEQC technique, presenting results from six months of operational use, and discussing the future directions and areas of continued work on quality control and observation filtering within the HiRAD environment.