ARM Data Quality Office: Fusion of Automated and Manual QA/QC Techniques

The Atmospheric Radiation Measurement Program (ARM) collects data from over 350 instruments on a daily basis. These instruments span from simple surface meteorological observations to scanning polarimetric radar data. The ARM Data Quality (DQ) Office is tasked with ensuring that data downloaded by the end-users are of the highest quality. The DQ Office employs a mixture of automated processing and problem identification techniques along with manual inspection of the data by undergraduate students to determine if there are problems with the instruments. If problems in the data are found, the appropriate personnel are informed, the instrument is repaired, and the data are flagged with a data quality report. As ARM has grown to collect more and more data, so to have the DQ Office processes and tools. Currently, the DQ Office runs approximately 10,000 processes a day to run tests and visualize the latest data coming in. This results in the daily creation or updating of over 5,000 images and over 400 daily summaries of QC tests that are used in the analysis of the data. New methods are always being tested in order to increase the efficiency of the process. A recent upgrade to utilize multiprocessing techniques has greatly enhance the DQ Office capabilities and will pave the way for new processes to review the ~25 years worth of climate data that ARM has collected since its inception in the hopes of further improving the dataset.