Extended Abstract (16K)9.6
Toward Real-Time Collection of Internet Data Distribution Statistics for Automated Topology and Performance Monitoring
Steven R. Chiswell, UCAR, Boulder, CO
The Unidata Internet Data Distribution (IDD) has been a successful innovation in the delivery of meteorological and related data to universities through scaleable data routing topoligies. The routing topologies are currently manually maintained based on site data requirements and the ability for upstream hosts to deliver the requested data to the recipient. The increasing volume of data which can be provided to universities in the realm of competitive bandwidth considerations provides interesting challenges for the distribution of data in real-time. The ability to reliably deliver the soaring volume of data is paramount to the successful integration of these products into teaching and research activities.
In order to ascertain the quality and timeliness of the Unidata IDD, participants are asked to provide automated statistical feedback on the performance of their data reception. Currently, statistics are collected by the Local Data Manager (LDM) software on each machine participating in the IDD, with an hourly summary delivered to a central location via standard internet mail protocols. The growing volume of data has lead to the continued advancement of the LDM to become increasingly efficient. However, the collection and reporting of statistics has remained relatively unchanged. In order to move toward automated topology and failover, a more timely and reliable method for reporting statistics must be created. Moreover, as the topology becomes increasingly dynamic, the routing history of each product must be preserved in order to monitor the performance of the distribution network.
Many universities employ firewalls and other security methods for protecting their networked hosts. The current use of sendmail for delivery of statistics is well suited to this end since universities typically provide campus mail servers. However, due to the handing off of mail messages to other servers, the timeliness of this method can be a major factor when considering delivering statistics of greater temporal resolution. Other highly reliable web based services can fulfill the need for proxy or gateway delivery and will be discussed. A real-time statistical catalog for the IDD network will be explored in order to make automated topology decisions.
Supplementary URL: http://www.unidata.ucar.edu/staff/chiz/latency/stats/
Session 9, Internet APPLICATIONS FOR EDUCATION AND RESEARCH
Tuesday, 15 January 2002, 1:55 PM-5:30 PM
Previous paper Next paper