Real time data monitoring at NCEP

Phase I of NCEP's Real Time Data Monitoring System (RTDMS), which focuses on monitoring the quantity and timeliness of operational data, became operational in July 2005. The primary purpose of this data monitoring system is to compare current and monthly average counts of many different observational data types for different model runs as well as for hourly time periods both for later display on the web and for possible problem alerting. The web site http://www.nco.ncep.noaa.gov/pmb/nwprod/realtime displays various color-coded summary pages to quickly indicate what data counts are either normal or abnormally low or high. With this tool, NCEP can quickly detect data receipt problems and either take corrective action or notify data suppliers of a problem. By selecting the right options, a user can view time series plots of the current data counts compared to their monthly average. Besides displaying data counts for different model runs, it also has data counts for dumps of data for one hour time periods with different time delays. The checks on hourly data counts allow an early warning, which can provide time to correct the problem before major model runs. To assist in resolving data flow problems, the web site has a “trouble shooting guide” where one can check on the source of the different data types and their path to NCEP. Additionally, this web site allows other NWP centers to compare data counts with NCEP, and for data suppliers to check on their data counts at NCEP. This system will be described and demonstrated and its utility reviewed. Phase II of this system is designed to monitor both the counts of model data in different pressure categories as well as the quality of the data compared to the NCEP guess. Progress on Phase II will be described, which includes the challenge of making automated alerts on the quality of many types of data based on recent differences to the guess compared to a running set of statistics. Work so far has shown that it is difficult to reliably produce alerts for real problems with out making excessive false alarms as changes in weather regimes can result in statistics versus the guess appearing abnormal.