Antarctic Internet data distribution (Antarctic-IDD) system

The Antarctic meteorological community is constantly faced with the challenges of acquiring and distributing meteorological data. Internet connectivity to and from the Antarctic continent is costly and generally of low bandwidth. The distributed nature of meteorological research and observational efforts in the Antarctic (i.e., many small projects and local programs) results in a variety of small but valuable datasets, yet distributing these data to other researchers, forecasters and logistic decision makers has been a continuous challenge. Solutions to some of these difficulties, such as better Internet communications, are some years away. In the meantime, sharing and distributing Antarctic data has often been done in an informal and ad hoc manner, often based on personal contacts. Operational forecasting for logistical activities in the Antarctic, and active Antarctic meteorological research programs are in need of a reliable, steady flow of meteorological observations, model output, and other related data in what is a highly collaborative environment.

Over the last few years, discussions spear-headed by the National Science Foundation (NSF), the agency that oversees the United States Antarctic Program (USAP), have led to a community discussion on collaborative efforts. The June 2004 Antarctic Automatic Weather Station – Antarctic Meteorological Research Center – Antarctic Mesoscale Prediction System (AAWS-AMRC-AMPS) joint annual meetings included a discussion on the synergy of the Antarctic meteorological community. The initiative to come from this discussion brings to life a community effort, with the leveraging help of another NSF funded project, the Unidata program. The result is the establishment of the Antarctic Internet Data Distribution (Antarctic-IDD) System, the objective of which is to be a system to reliably share and distribute Antarctic meteorological data.

The Antarctic-IDD is based on the Local Data Manager (LDM) software developed by Unidata and compatible with the overall Unidata IDD/LDM architecture. Each participating site runs LDM software, which is in almost continual communication with one or (preferably) more other sites. Data files properly identified and inserted at one site into a local database file (called a “product queue”) are then available for almost immediate transfer to the product queues at other sites. The result is a collaborative network of sites, each sharing the datasets available to them. This system was setup in a test mode and demonstrated in the spring of 2005. Currently, the Antarctic-IDD is growing to include a variety of data sets from a variety of data providers for a variety of users. At this time, the Antarctic-IDD carries surface and upper air observations, satellite observations and products, as well as numerical model output.

This presentation will review the initial setup of this system, and the current status as well as outline future plans. The presentation will also touch on some issues facing the Antarctic-IDD, including data formats, data compression, firewall limitations, transmission file naming protocol, etc.