SCOOP was first funded in 2003 as a prototype to examine distributed numerical modeling processes, both in closely coupled computational clusters and geographically diverse nodes using virtualization services. In 2004, additional information technology was added to bolster the more conventional Grid computing concepts as well as enhancing the standards work in both data and metadata. At the same time, a regional archive system was proposed to allow data sharing among the partners, and a data transport plan was suggested to reduce workload and duplication of effort by modelers who were seeking common data to initialize numerical ocean models.
Archive systems work was initiated for the 2004 Atlantic Hurricane Season. Estimates were solicited from the SCOOP partners, and two sites, Louisiana State University (LSU) and Texas A&M University (TAMU) agreed to provide archive services. Initial estimates provided by the modeling contingent suggested 1-2 terabytes (TB) of data would be generated over the course of the hurricane season. Both sites entered the season with sufficient media capacity to fulfill SCOOP archiving requirements based on the early estimates. By the end of an unusually busy hurricane season (which extended into January 2006), Texas A&M was providing a total of 7 TB of disk storage for SCOOP activities with expansion plans; LSU had undertaken agreements with the San Diego Supercomputing Center (SDSC) to utilize up to 7 TB of SDSC Storage Resource Broker resources to facilitate SCOOP requirements. LSU also began investigations using Grid-toolkit methods for archiving and distributed storage.
SCOOP partners had gained significant experience in the use of various catalog services including THReDDS and OpENDAP, and Live Access Server to facilitate data discovery, subsetting, and access/transport. They also employed web-scraping and file transport protocol methods to acquire data for model initialization, data assimilation and model verification. Beginning in 2004, partners with experience in the Unidata Internet Data Distribution (IDD) system and the underlying Local Data Manager (LDM) software promoted the use of LDM as a transport for datasets between partners. An ad hoc point-to-point IDD was established and the results of model outputs as well as initialization data were transported to the Partners using this system. A dedicated SCOOP Catalog service was also designed and implemented. This presentation will identify the successes and failures encountered and the process used to improve performance in data management and transport.
The NOAA Coastal Services Center and the Office of Naval Research provide funding for the SURA/SCOOP Program. The following institutions are collaborators within the SCOOP Program: Bedford Institute of Marine Science, the Gulf of Maine Ocean Observing System, Louisiana State University Center for Computation and Technology, MCNC, Renaissance Computing Institute, Texas A&M, the University of Alabama in Huntsville, the University of Florida, the University of Miami Center for Southeastern Tropical Advanced Remote Sensing, the University of North Carolina, and the Virginia Institute of Marine Science.