The Unidata Internet Data Distribution (IDD) System: A Decade of Development

The mission of the Unidata program of the University Corporation for Atmospheric Research (UCAR) is to provide universities with innovative applications of current computing and networking technologies to access and use atmospheric and related data for education and research. One application, the Internet Data Distribution (IDD) system, is an event-driven network of cooperating Unidata Local Data Manager (LDM) servers that distributes discipline-neutral data products in near real-time over wide-area networks.

The IDD was developed in the early 1990s in response to challenges related to weather-data ingest via satellite broadcast (e.g., local sources of terrestrial interference, data outages caused by solar occultations, weather-related outages due to signal degradation, and the difficulty in locating satellite reception systems near departmental resources) and to provide access to datasets that were not commonly available. The IDD has been the primary meteorological data delivery vehicle used by US universities with atmospheric science curricula for over a decade. Starting with a modest goal of internet delivery of data available in the NWS Family of Services satellite broadcast, the IDD has grown to become the leading Internet2 advanced-application, currently delivering over 20 terabytes (TB) of data per week to participating institutions. Stress testing conducted at the Unidata Program Center offices in the summer of 2005 demonstrated that a cluster approach to data relay was limited only by the bandwidth available in the underlying (gigabit) network thus ensuring future IDD extensibility.

The Unidata IDD has recently expanded from a US-centric delivery system to one that includes 13 countries on 5 continents. Additionally, the LDM is being used as the data distribution engine in systems akin to the Unidata IDD: by private industry; by several US government agencies including the National Weather Service and NASA; and by the national weather services of South Korea and Spain.

This paper reviews the evolution of the LDM/IDD over the past decade, and provides a vision for future development.