User Driven Automatic Data Request Service - Providing User Access to Terabyte-sized Datasets
In the Research Data Archive (RDA, rda.ucar.edu) at NCAR we maintain many historical and ongoing, observational and model-produced, atmospheric and oceanographic data products. The data product files are archived in a tape-based High Performance Storage System (HPSS), and a copy of the most active data files are also stored on a central disk-based file system. A stable, scalable and distributed controller, DataSet ReQueST (DSRQST), has been designed and implemented for auto-processing user requests, including data subsetting, format converting, and data staging for individual users. The system runs unattended 24x7, has fault resistant recovery procedures, uses the HPSS and central file systems for data access, a MySQL database for record keeping and job control, and a large capacity multi-node large memory cluster for fast and efficient computations. The DSRQST workflow has been designed to easily implement new data services as user needs and resource availability dictate. For example DSRQST could drive user specified re-gridding of model data and algorithmic application to native parameter fields to create additional products.
More than 300 RDA data products are served by DSRQST, in addition to the traditional methods. In this presentation we will discuss the highlights of the DSRQST work flow, and illustrate the positive impact it is having on the users with more than 2000 individual requests per month.