A Scalable, Cloud-Based Implementation of a Complete Ground Processing System

The latest generations of meteorological satellites represent a massive increase in the amount of available remote sensing data. Processing these large data streams through complex, modern algorithms requires significant computational resources, and poses unique engineering challenges. Ensuring the timely and complete flow of data through complex networks of processing components requires careful engineering. Providing the computational resources necessary to produce desired products from these data streams can pose a challenge, even for organizations with limited areas of geographic interest. In this work, we present a software system that solves both of these problems by employing a modern component-based enterprise ground framework, and deploying its processing elements on the cloud. The result is a complete multi-mission ground system operating within a scalable cloud environment, containing the full GOES-R algorithm processing chain and operating on data from the Himawari-8 AHI as well as simulated MeteoSat Third Generation FCI imagery.

Using the AER Algorithm Workbench technology, we are able to design the complete processing flow of a ground processing system, including area of interest, desired end products, and final output formats. This information is encoded in machine-readable system configuration files, which can be analyzed for correctness, visualized, or sent to a processing system to execute. These configurations can be run on small workstations for test or demonstration, sent to local processing servers, or run using the associated cloud-enabled processing system. This system provides web-based visualization tools that allow users to remotely monitor the health and status of the system, as well as view a live image of the results. The processing elements can scale to multiple nodes, providing users with the throughput capabilities necessary to generate live data products based on satellite downlinks. By providing the capability of local development and test, coupled with the scalable cloud infrastructure and complete chain of GOES-R algorithms, the Algorithm Workbench allows users to develop their systems on modest computational platforms, automatically scaling them up to real-time processing needs when necessary.