Enterprise Data Management (EDM) and Enterprise Product Generation (EPG) Proving Ground in the Amazon Web Services (AWS) Cloud

As part of the NOAA/NESDIS/OSGS Environmental Satellite Processing and Distribution System (ESPDS) program, Solers has been funded to create a proving ground for Enterprise Data Management (EDM) and Enterprise Product Generation (EPG) services in a FedRAMP-approved Amazon Web Services (AWS) cloud environment, leveraging native AWS cloud services and NESDIS product generation algorithms. The EDM and EPG services are derived from capabilities that exist within the existing on-premise ESPDS operational product generation subsystem known as NOAA Data Exploitation (NDE), with refactoring and enhancements to take advantage of native AWS cloud services and scalability features.

The EDM service provides data storage, a flexible and searchable inventory/catalog of product metadata, and science data manipulation through RESTful interafces capable of supporting all NESDIS products. It leverages a variety of native AWS cloud services including Simple Storage Service (S3), Elasticsearch Service, Relational Database Service (RDS), Lambda, and API Gateway. The EPG service is capable of the generation of all NESDIS level 1+ sensor, science, and tailored products. It also leverages a variety of native AWS cloud services including Elastic Compute Cloud (EC2) with Auto-Scaling, Simple Notification Server (SNS) and Simple Queue Service (SQS). The EDM and EPG services will provide NESDIS with added flexibility in provisioning computational processing, metadata management, and data storage on resources that best meet product, mission, and cost requirements.

The primary objectives of this EDM and EPG Proving Ground initiative are:

1. To leverage the flexibility and agility provided by a cloud environment to prototype candidate architectures and implementations for EDM and EPG services, and evaluate them for efficacy, performance, scalability, and maintainability.
2. To demonstrate the flexibility of the proposed EPG service to execute multiple types of algorithms, such as existing NDE 2.0 product algorithms, JPSS Risk Reduction algorithms, Enterprise Algorithm implementations of legacy products, and/or GOES-R L2+ Product Algorithms; and to assess the cost of running these algorithms in a cloud environment.

The secondary objectives of this EDM and EPG Proving Ground initiative are:

1. To consider how cloud-hosted EDM and EPG services could be used for collaboration and integration of future product generation algorithms, both within NESDIS and with collaborative research organizations.
2. To identify cost breakpoints for technology, ingress & egress, performance, etc.

This EDM and EPG Proving Ground initiative demonstrates the ability to run EDM and EPG services for the generation and management of NESDIS products in a FedRAMP-authorized cloud environment, which has the potential of reducing facility, hardware & software, license & maintenance, administrative, and operations lifecycle costs. Efficiencies in these areas could be achieved from migrating operational NESDIS product generation into a cloud environment, as well as from hosting development/test and collaborative research capabilities in a cloud environment.

S-NPP, JPSS-1, and GCOM-W data is currently being ingested near-real-time into the AWS cloud-hosted EDM and EPG Proving Ground environment from the on-premise ESPDS Integration & Test (I&T) Environment at the NOAA Satellite Operations Facility (NSOF) in Suitland, MD. GOES-16 data is also currently being ingested near-real-time into the AWS cloud-hosted EDM and EPG Proving Ground environment from the NOAA Big Data Project’s GOES-16 AWS S3 bucket.

The following products are currently being generated and inventoried/cataloged near-real-time within the AWS cloud-hosted EDM and EPG Proving Ground environment:

• GCOM-W (AMSR2-L1, GAASP)
• JPSS-1 (Active Fire, JRR, NUCAPs, OMPS, Tailoring)
• S-NPP (Active Fire, GVF, JRR, MiRS, NUCAPS, OMPS, Tropical Cyclone, SR, VH, VI, Polar Winds, Tailoring)
• GOES-16 (Level 2 products from the STAR-developed CSPP AIT Framework)
• Other (VIIRS Imagery, Polar2Grid from the STAR-developed CSPP AIT Framework)

The EDM and EPG Proving Ground initiative is expected to continue into March 2019, concluding with a final report that documents the results and recommendations from the prototyping efforts.