Unidata's path for NOAAport data distribution

Unidata initiated widespread reception of weather data at universities by developing the Local Data Manager (LDM) software for ingesting and processing the Family of Services (FOS) data which were received from a commercial satellite feed. As participation in Unidata increased and greater numbers of universities became connected to the Internet, the feasibility of distributing FOS data in a cooperative network of universities emerged into the Internet Data Distribution project. The LDM software was expanded to serve as the backbone of the IDD, which allows sites to inject and relay data to other universities, as well as to receive and process data products locally. The utility of the LDM data queue mechanism allows for near-term recovery of missed products from an upstream site. Data reception at a site is transparent to the LDM as products can be received locally or from an upstream relay. The LDM software is capable of handling arbitrary data types, and thus has been employed in a wide range of real-time applications.

Unidata transitioned the IDD broadcast source from FOS to NOAAport's NWSTG channel in December, 1998 as the AWIPS data system matured. The deployment of NOAAport by the National Weather Service (NWS) promised a reliable and timely distribution of data. By deploying several strategic NOAAport downlinks at cooperating Unidata sites, the IDD has developed a redundant system for data injection into the IDD whereby interconnected relay sites can maintain data reception through solar outages, network interruptions, and hardware down time.

To enhance data availability to Universities, we have now coupled the LDM software with off the shelf hardware to allow simultaneous reception of all four NOAAport channels using a single modestly configured computer. Our initial tests show that this system provides a very stable and reliable ingestion platform which can serve as an IDD ingestion node without additional overhead. Moreover, this configuration provides a seamless method for expanding the data being distributed via the IDD as additional products are added to the NOAAport broadcast. Mechanisms for providing this data to the Unidata community will be presented.

The LDM remains the cornerstone of the IDD, which has evolved into a fully hybrid system, utilizing the best features of both a satellite reception system and Internet distribution system. The increasing volume of data which can be provided to universities in the realm of competitive bandwidth considerations provides interesting challenges for the distribution of data as Unidata approaches the the new century.