The Data Processing Error Analysis System (DPEAS) is a parallel data processing system for cross-sensor satellite data merger and analysis. This paper discusses the various features of the system and how they are used in the NOAA/NESDIS Advanced Microwave Sounding Unit (AMSU) data processing system at CIRA. The system simplifies the near real-time merger of multiple cross-sensor data sets from several sensors and satellite platforms. This processing is facilitated by the use of the NASA HDF-EOS data format as a common data structure within the system. The practical impact to similar scientific technology transition efforts is also discussed.

The system is a peer-to-peer (P2P) distributed application with centralized management and fault-tolerant system behaviors. This allows the parallelism to be temporally dynamic with worker nodes entering and exiting the processing cluster at will. The central administration (or “hive”) of the entire cluster is able to migrate across multiple nodes automatically. The system maintains a replication file system topology in near real-time that allows for fault-tolerance of the entire system state. The granularity of the parallelism is large, thus the system is adaptable to new Internet standards (such as XML and SOAP) as they mature.