Evolving data requirements for research and education

The volume of data available for research and education continues to grow. By expanding the timely access of these data sets to the atmospheric science community, novel and innovative developments toward research and education initiatives can be fostered. While forecasts have extended increasingly into the future, the analysis of real-time products has pushed the focus of data into the present. Many data products such as radar, lightning, profiler, aircraft, GPS and rapid scan satellite imagery are currently provided on time scales of five minutes or less. Many high resolution products are used for model data assimilation and rapid update cycles as well. The ability to deliver the increasing volume of data is paramount to the successful integration of these products into teaching and research activities.

Unidata's Local Data Manager (LDM) software has been a cornerstone for data delivery to the university community for over a decade. At present, numerical model data comprises the greatest volume of data being distributed routinely to universities through the Internet Data Delivery (IDD) system. Model data continues to be provided at higher temporal and spatial resolutions as well as for greater forecast durations and additional initialization times. This growing volume of data has lead to the continued advancement of the LDM to become increasingly efficient. However, with the addition of real-time NIDS products to the freely available data streams in Fall 2000, the timely distribution of these products provides interesting new challenges to data delivery. In the past, the products comprising the largest data volumes in the IDD data streams (model & satellite imagery) were broadly applicable to most recipients throughout the data delivery system. However, with the focus of NIDS products toward regional interest, the emphasis on timely arrival of localized products becomes more complex. The ability to prioritize the data need as rapidly as possible, while maintaining the ability to access the broad selection of data available has demonstrated the evolution of data requirements for universities. The increasing volume of data which can be provided to universities in the realm of competitive bandwidth considerations provides interesting challenges for the distribution of data in real-time.