Extended Abstract (592K)P2.39
Architecture of MADIS data processing and distribution at FSL
Christopher H. MacDermaid, NOAA/ERL/FSL, Boulder, CO; and R. C. Lipschutz, P. Hildreth, R. A. Ryan, A. B. Stanley, M. F. Barth, and P. A. Miller
Architecture of MADIS Data Processing and Distribution at FSL
Chris MacDermaid*, Robert C. Lipschutz*, Patrick Hildreth*, Richard A. Ryan*, Amenda B. Stanley*, Michael F. Barth, and Patricia A. Miller
NOAA Research-Forecast Systems Laboratory Boulder, Colorado
*In collaboration with the Cooperative Institute for Research in the Atmosphere (CIRA), Colorado State University, Fort Collins, Colorado
The NOAA Forecast Systems Laboratory's (FSL) Meteorological Assimilation Data Ingest System (MADIS) (Miller, et al. 2005) supplies quality controlled (QC) observational data to a large and growing segment of the meteorological community, including operational National Weather Service modeling and forecasting users. The requirements for this system have dramatically expanded to accommodate the increasing volume and types of data handled, as well as the growing user base. Today, the system ingests and merges data from dozens of sources, and supplies the resultant quality controlled products to hundreds of users via several standard protocols. Both real-time and saved MADIS datasets are made available. While it needs to be highly reliable, the system must be sized for the CPU-intensive QC analyses, and must provide controlled access to the data so that proprietary datasets are only available to appropriate users. Methods have also been developed for keeping track of the large, diverse user community and consistently responding to specific user needs, for example, notification of significant events.
To ensure reliability of MADIS, the system runs within the FSL Central Facility, a controlled environment maintained by FSL's Information and Technology Services, which provides system administration, networking, configuration management, data acquisition/distribution, and monitoring support (Lipschutz and MacDermaid, 2005). MADIS comprises a distributed architecture for ingest, processing, and data distribution functions. In addition, in a recent architectural advance, the various functional hosts are arranged in pairs using High-Availability (HA) Linux to provide automated failover in case of system failure. Real-time data distribution methods include FTP, Local Data Manager (LDM), and the Web-based OPeNDAP (OPen source project for Network Data Access Protocol). Special consideration for protecting proprietary datasets has added an extra layer of complexity, but is well handled by the configuration methods implemented on the LDM and FTP servers.
In this paper, we review the requirements and architecture of the operational implementation of MADIS at FSL. We describe our use of High-Availability Linux toward achieving high system reliability, and consider our implementation of the data distribution methods. Finally, we also discuss our user support infrastructure.
Corresponding author: Chris Macdermaid NOAA Forecast Systems Laboratory R/FST 325 Broadway Boulder, CO 80303
Phone: 303-497-6987 FAX: 303-497-7259
e-mail: Chris.MacDermaid@noaa.gov
Supplementary URL: http://www-its.fsl.noaa.gov/dsg
Poster Session 2, IIPS Poster Session II
Wednesday, 12 January 2005, 2:30 PM-4:00 PM
Previous paper Next paper