The original system was a multiple step process. The dataloggers would perform internal cursory checks of data validity and range, applying a datalogger flag if data was collected with obvious issues. ATG staff members would perform a quality assurance review of the data. This would consist of a somewhat disconnected process with several steps. First, a custom SAS front end providing direct access to an Oracle relational database was opened. Weekly hard copy plots of the time history of the similar instruments were displayed side by side for the towers across SRS. Each plot was visually inspected and suspect periods were manually marked. The staff member would validate the visual inspection from the graphical plots using the numerical values in tabular displays in the original SAS interface. Time periods of suspect data were labeled as anomalies. The person performing the QC would enter another software program to obtain access to the Oracle database. These periods of bad, unrepresentative or biased data where then given a descriptive letter designation and often a text description describing the cause, if known, of the instrument malfunction. While straightforward, the process was laborious since the person performing the QC had to sift through dis-similar data types and manually reconcile differences.
The High-Performance Computer (HPC) group at SRNL assisted ATG with the replacement of the tower instruments, programming of the new dataloggers along with the transition of the database from Oracle to PostgreSQL relational database. It was desired to create a develop a single software package to allow all the described from the original QC process into a single integrated software package. The multiple interfaces retained the original functionality (to assign the anomaly descriptions to time periods) but using computer graphics and shading techniques would be more easily be able identify and reconcile the different views of the data.
The software is known as “Fishnet” since it combs through the data, pulling out the unrepresentative data and allowing the good data to pass through. The HPC group developed “Fishnet” as a Python application that queried the new PostgreSQL database, plot the data into screens containing similar on-screen plots, with color coding and flagging. Datalogger flags were automatically displayed on the graphics to allow the QC person to quickly view known issues identified by the logger. Potential outliers that may have outside expected range from the site mean value, or from manual intervention have opaque colors overlaid on top of the time series plot. A second sheet of numerical cells, with allow the user to closely inspect the values from all the plots contained on the screen at the time, with the same color coding that is used on the graphical time history display. The user can toggle back and forth between the two different displays (graphical and numerical) and determine if the data is representative of the conditions occurring compared to the other towers in the SRS domain. The QC technician then uses the final screen, where one or more time periods are displayed as anomalies, which then get the data QC flag added, if necessary to describe instrument issues, fixes and biases that may exist. The text description of the issues identified are added, and then automatically applied to the selected period.
- Indicates paper has been withdrawn from meeting
- Indicates an Award Winner