5A.4
A Robust Framework for Distributed Processing of GIFTS Hyperspectral Data
Abhishek Agarwal, Nortel Government Solutions, Lanham, MD; and S. Tehranian, A. Swaroop, and K. Mckenzie
It is estimated that future satellite instruments such as the Advanced Baseline Imager (ABI) and the Hyperspectral Environmental Suite (HES) can provide raw data of the order of multiple Terabytes per day. Due to the high data rate, satellite ground data processing will require considerable computing power to process and archive this data in near real-time. Cluster technologies employing a multi-processor system combined with a parallel file system is the only cost effective solution for such processing and storage. However distributed systems are inherently unstable; failure of one component may result in catastrophic failure of the system if necessary measures are not taken. Operational real-time systems need to be reliable and fault-tolerant, operate on continuous data streams and be operator friendly. To sustain high levels of system reliability and operability in a cluster-oriented operational environment, a data processing framework is proposed to provide a platform for encapsulating science algorithms for satellite data processing. The science algorithms together with the framework are hosted on a Linux cluster. The primary mission of the National Oceanic and Atmospheric Administration (NOAA) is to understand and predict changes in the Earth's environment which requires a continuous capability to acquire, process and archive data in real-time. NOAA is using Geosynchronous Imaging Fourier Transform Spectrometer (GIFTS) technology to measure these capabilities and use them in the design of the NOAA GOES-R series of imager and sounder instruments. The GIFTS instrument uses a combination of Large area Focal Plane Arrays (LFPA's), and a Fourier Transform Spectrometer (FTS) providing a spectral resolution of 0.6 cm-1 for a 128 x 128 set of 4 km foot-prints every 11 seconds. It is anticipated that the GIFTS Level-0 data rate is about 55 Mbps or about 1.5 Terabyte per day. The volume of Level-1 data is approximately the same as Level-0. Since there is little reduction in data volume from Level-0 to Level-1, producing level-2 data also requires significant computing power. GIFTS data processing system is required to generate critical products within 5 minutes of gathering observation. In this paper we present an architectural model based on the master-worker paradigm and a system prototype for providing performance, reliability, and scalability of candidate hardware and software for a satellite data processing system using the Message Passing Interface (MPI) standard. We propose a robust framework for real-time data processing of satellite data which separates science algorithms and their implementation from the complexities involved with dataflow, parallel processing, and operation of the ground station, respectively. This alleviates the need for earth scientists to understand parallel computing and fault-tolerant operations. Furthermore we investigate various task scheduling and migration strategies, worker as well as master fail-over scenarios. Benchmarking results are presented for a selected number of science algorithms for the GIFTS instrument showing that considerable performance can be gained without sacrificing the reliability and high availability constraints imposed on the operational cluster system. We have used Lustre as a cluster file system to increase the I/O scalability. The Lustre file system is based on a highly scalable object-based storage architecture that scales to tens of thousands of clients and Petabytes of data by separating access to file data from file metadata. Figure 1(a) below shows the total execution time versus number of processors for a datacube of 128x128 pixels in long wave and short-medium wave bands. Figure 1(b) below shows the speedup versus number of processors for a datacube of 128x128 pixels in long wave and short-medium wave spectral range. A maximum speedup of 58.81 (91.9% efficiency) is obtained for a total number of 64 processors over a data cube of 128 x 128 pixels in the long wave and short-medium wave spectral range. This prototype system shows that considerable performance can be gained for candidate science algorithms without sacrificing reliability and high availability needed for a real-time system.
Session 5A, Satellite IIPS and Applications
Wednesday, 17 January 2007, 8:30 AM-10:00 AM, 216AB
Previous paper Next paper
Extended Abstract (256K)
Recorded presentation