A Serverless Computing Implementation of Flood Inundation Mapping

NOAA’s Office of Water Prediction (OWP) develops and maintains flood inundation mapping software (https://github.com/NOAA-OWP/inundation-mapping) designed to produce Height Above Nearest Drainage (HAND) grids and synthetic rating curves for use in the production of flood inundation mapping (FIM) forecasts. The goal of OWP’s flood inundation mapping project is to forecast flood inundation in near real time across the United States, Puerto Rico, and the U.S. Virgin Islands, and disseminate these forecasts to National Weather Service Weather Forecast Offices, River Forecasting Centers, core stakeholders, and the public.

The Python scripts predicting water levels nationwide, at 10-meter spatial resolution, require significant computational resources. The spatial processing unit is the USGS 8-digit Hydrologic Unit Codes (HUC-8), of which there are 2,135 within inundation mapping’s processing domain. Currently, running inundation-mapping at the continental scale takes around 56 hours to complete using 80 CPU cores on NWC servers, which consist of 96 cores and 500 GiB of RAM. Network maintenance/reboots, required post-processing, quality assurance and validation steps all hinder the pace of development.

Recent advances have resulted in a platform agnostic solution by taking advantage of Docker containers to fulfill external library and dependency requirements. While the inundation-mapping codebase has yet to reach full maturity, with ample room for optimization, the time spent refactoring to run in a serverless environment was the quickest way to gain a significant performance benefit. A primary advantage of using Amazon World Services’ (AWS) serverless resources (Lambda, Step Functions, ECS, Fargate) is the ability to horizontally scale and parallelize the processing of the 2,135 HUC-8 regions. As opposed to waiting for cores to become available for each HUC8 to process, using the serverless approach, each HUC8 level processing unit is assigned to an individual Fargate container. The serverless workflow takes roughly 8.5 hours to complete, translating to a decrease in runtime by a factor of six from on-prem systems.

This presentation will include a description of the serverless infrastructure and workflow utilized in porting inundation-mapping to AWS, highlighting the advantages and challenges faced during the transition to, as well as employing serverless cloud based resources.