ESPDS: Over 1 Billion Served—Three Years of Operations for the Environmental Satellite Processing and Distribution System

NOAA’s National Environmental Satellite, Data, and Information Service (NESDIS) Environmental Satellite Processing and Distribution System (ESPDS) recently hit an important milestone, having received and processed its one-billionth satellite meteorological data product after three years of being in operational status. ESPDS has been in development by Solers, a Peraton Company, since 2010, and provides Product Generation and Product Distribution services for more than 84 distinct satellites, including the GOES-R and JPSS series spacecraft as well as legacy NOAA satellites (e.g., POES, GOES), NASA satellites (e.g., AQUA, TERRA), and satellites operated by our foreign partners (e.g., EUMETSAT, METOP, Himawari 8). ESPDS ingests and processes over 1.15 million files per day (13 files per second on average, with fluctuations that reach up to 80 files per second), performs Product Generation for over 320,000 files daily, and distributes over 2.3 million files per day to over 2,000 unique destinations, with system overall availability that greatly exceeds our 99.44% uptime requirement.

ESPDS provides a flexible, enterprise solution that has allowed us to continually integrate new data providers and data consumers without the need to change software or hardware architecture. ESPDS has been demonstrated to be capable of managing a continuous load of 15TB a day of data ingress and 42TB per day of data egress while maintaining the stringent security, resource availability and processing latency constraints spelled out in our requirements. ESPDS’ robust, yet flexible algorithm-processing engine allows for the introduction of new and improved Product Generation algorithms without the need for code modifications. ESPDS’s self-service User Portal allows data consumers to search for and subscribe to data products arriving at ESPDS using a broad range of metadata characteristics as discriminators. Users are also able to specify a robust set of Tailoring operations to modify selected products that include: geographic subsetting (sectorization), data layer subsetting, remapping (re-projection), compression, image type conversion, image scaling, and bit depth conversion.

ESPDS continues to grow and improve as we work toward our System Acceptance Review, and is currently undergoing a tech refresh cycle that replaces end-of-life third party software components, updates the underlying data storage mechanism (Isilon NAS), and provides an improved development environment that is being transitioned for shared use as the system advances toward sustainment. Recently, Solers has worked with the NOAA A-CIO’s office to provide a proof-of-concept demonstration project that re-hosted the ESPDS Product Generation engine within the AWS Cloud. Additionally, Solers has recently introduced the Elastic Stack schema-less database search engine for providing an enhanced reporting mechanism that greatly improves system situational awareness for this large, distributed system. This addition to ESPDS has demonstrated the system’s flexibility and scalability, providing a plug-and-play add-on to our architecture in response to evolving operational monitoring and reporting needs.