Assessing the quality assurance system for the Oklahoma Mesonet with accuracy measures

There are many components to the Quality Assurance (QA) system of the Oklahoma Mesonet. QA meteorologists, laboratory technicians, field technicians, and automated algorithms all interact to detect sensor problems. QA meteorologists issue trouble tickets to have technicians fix or replace problematic sensors. Sensors return to the Oklahoma Mesonet laboratory if there was a sensor problem that could not be fixed in the field or as part of preventative maintenance. Once the sensor is in the lab, a lab technician verifies the sensor problem or determines if there was a subtle issue with the sensor (e.g., sensor drift). If there was a trouble ticket issued for the sensor, and the lab verified the problem, data flags (that result from the sensor malfunction) are not changed. However, if there was not a problem with the sensor, or a rotated sensor did indeed have a problem, data flags are modified.

In 2007, QA meteorologist performance was assessed based on calibration lab results. One way to quantify how well the QA meteorologists identified sensor problems is to calculate accuracy measures (analogous to the way the National Weather Service determines success when issuing warning). It was found that Proportion of Correct values were greater than 80% for relative humidity, pressure, and wind speed sensors; False Alarm Rates were less than 20% for soil temperature and relative humidity sensors. Based on Bias, Mesonet QA meteorologists greatly under-ticket the wind speed at 2-meters (cup anemometer), while they over-ticket wind speed at 10 meters (prop anemometer). The Probability of Detection was greater than 75% for air temperature, relative humidity, and soil temperature sensors, and the Threat Score was greater than 75% for relative humidity and soil temperature sensors. Results from the accuracy measures allow the QA meteorologists to know what sensors are being ticketed correctly and where focus needs to be directed.