Cloud-Based Ingest & Processing Framework (I&PF) – A Cloud-Hosted Framework for Science Algorithm Development and Integration

As part of an Internal Research & Development (IR&D) project, Solers created and demonstrated a Cloud-Based Ingest & Processing Framework (I&PF), as a mechanism to enable fast/easy integration of data sources, product algorithms, and data consumers within a cloud-hosted workflow (or “data pipeline”) framework.  The Cloud-Based I&PF is hosted within the Amazon Web Services (AWS) public cloud infrastructure, and leverages multiple AWS cloud services along with open source software, including:

• AWS EC2
• AWS Elasticsearch
• AWS ElastiCache
• AWS S3
• Apache NiFi Workflow Engine

The Cloud-Based I&PF provides easy to use web-based user interfaces for search and access (for end users), as well as workflow monitoring and management (for system operators/admins).  It also provides RESTful web services for other developers, scientists, etc. to discover and access the ingested/processed data and metadata, for use in other research / engineering initiatives (e.g., developing a new product algorithm).

As part of the IR&D project, Solers implement 3 different NOAA-specific use cases to demonstrate the flexibility of the Cloud-Based I&PF:

1. NOAA S-NPP ATMS and MiRS: Ingests and inventories S-NPP ATMS data records.  Generates MiRS products from the ATMS data records.  Makes ATMS data records and MiRS products searchable and accessible.
2. NOAA Nexrad II Weather Radar: Ingests and inventories NOAA Nexrad II Weather Radar data sets that were published on AWS S3 as part of the NOAA Big Data Initiative.  Makes NOAA Nexrad II Weather Radar data sets searchable and accessible.
3. MiRS / Nexrad II Blended Product: Leverages the available MiRS products and NOAA Nexrad II Weather Radar data sets to produce a new blended product that combines the MiRS snow/water data with the Nexrad II radar data over the mountainous regions near Oregon.

In addition to commercial utility, the Cloud-Based I&PF could be an ideal capability to use as a development, integration, and test environment for Government satellite ground systems, such as those from NOAA/NESDIS.  It could be leveraged to perform development and Cal/Val of new product algorithms that leverage multiple satellite (and other) data sets within a scalable cloud-hosted framework, prior to integrating them into operations, without requiring on-premise infrastructure.  Such a cloud-hosted framework could also enable interaction and participation with scientists and engineers outside of NOAA/NESDIS (e.g., non-Government and International partners), allowing algorithms to be refined and tested thoroughly with other community partners in a cloud-hosted environment, prior to integrating them into on-premise production satellite ground systems.