In order to ascertain the quality and timeliness of the Unidata IDD, participants are asked to provide automated statistical feedback on the performance of their data reception. Currently, statistics are collected by the Local Data Manager (LDM) software on each machine participating in the IDD, with an hourly summary delivered to a central location via standard internet mail protocols. The growing volume of data has lead to the continued advancement of the LDM to become increasingly efficient. However, the collection and reporting of statistics has remained relatively unchanged. In order to move toward automated topology and failover, a more timely and reliable method for reporting statistics must be created. Moreover, as the topology becomes increasingly dynamic, the routing history of each product must be preserved in order to monitor the performance of the distribution network.
Many universities employ firewalls and other security methods for protecting their networked hosts. The current use of sendmail for delivery of statistics is well suited to this end since universities typically provide campus mail servers. However, due to the handing off of mail messages to other servers, the timeliness of this method can be a major factor when considering delivering statistics of greater temporal resolution. Other highly reliable web based services can fulfill the need for proxy or gateway delivery and will be discussed. A real-time statistical catalog for the IDD network will be explored in order to make automated topology decisions.
Supplementary URL: http://www.unidata.ucar.edu/staff/chiz/latency/stats/