Implementation of a Scalable, Economical Cloud-Based Real-Time Radar Mosaicking System

In December 2017, Weather Decision Technologies (WDT) moved the bulk of its real-time radar data processing system from a data center to the cloud using Amazon Web Services (AWS). WDT provides real-time radar mosaics and products derived from those mosaics to its clients. Radar mosaics are made possible by ingesting data from over 250 individual radars from around the world. The backbone of the radar mosaic system is the Warning Decision Support System – Integrated Information (WDSS-II), which is also used by the National Severe Storms Laboratory for the Multi-Radar/Multi-Sensor System (MRMS).

Prior to the move to AWS, two years were spent in feasibility and efficiency studies to determine requirements for making a successful move to the cloud. To make the move economical, the system had to scale itself automatically in response to changing precipitation patterns. To do this, a new head-node concept was developed for WDSS-II. The head-node manages the assignment of small chunks of processing to compute instances, shifting these chunks as some servers become overloaded and others become under-utilized, thereby adding or removing compute capacity automatically as conditions warrant. This results in a self-monitoring, self-regulating system that makes efficient use of AWS resources producing output with very high spatial and temporal resolution; mosaics for the conterminous United States with grid spacing as low as 250 meters and update frequency as low as one minute can be produced if desired.

Since the move to AWS, previously unknown issues were discovered, and new capabilities offered by AWS have necessitated some enhancements in the system. For example, large static, configuration files are used by WDSS-II in production of 3D mosaics. AWS’s Elastic File System (EFS) provides for easy storage and practical use of such files by many compute instances simultaneously. However, AWS regulates the amount of I/O in accessing EFS files, requiring additional tweaks to the head-node and processing instance configurations to optimally balance I/O on the EFS and memory consumption on the processing instances.

Also, AWS’s DynamoDB database service implemented a new method to allow for dynamically adjusting read/write capacities on existing tables that provided another opportunity to take advantage of the scalability of the cloud and cut costs. AWS recently instituted a policy change to charge for compute instances by the second rather than by the hour while new EC2 instance types are continually being made available while older ones are deprecated.

Addressing these issues requires the ability to test various configurations without interfering with the production system, which is another great advantage of the cloud. While WDT’s radar processing system is running within a production environment, a new configuration of that system can easily be tested within a separate development environment. This was not possible within our previous data center.

Overall, the move from a data center to AWS has been very positive. An extended period of time was required to determine optimal setup for our processing infrastructure within the cloud. New capabilities such as adding a head-node to automate and regulate use of AWS resources have proven vital. And flexibility has been necessary to take advantage of new cost savings AWS continually makes available. Ability to develop and test new configurations for WDT’s system within separate environments allowed us to continue optimizing mosaic production and make allowances for new capabilities AWS provides.