Utilizing Cloud Computer Services to Accomplish Low Cost High Availability Data Stream Processing (Invited Presentation)

In weather data processing, cloud services have proven to be incredibly useful, especially in the realm of data stream processing. Cloud computing’s extreme flexibility allows a user to provision a service using a baseline level of service and easily allow for variances in the incoming data rates. When it comes time to process the data a system can be set up to automatically start any number of processing systems and then shut them off once the calculations are complete. This means that organizations no longer need to purchase large amounts of computing power that will merely sit idle when there are no models or calculations to be processed. Many cloud providers have developed systems that can be set up to perform calculations based on a certain event where a developer does not need to provision any actual hardware. These “server-less” processing systems have the advantage that they can be scaled automatically by the provider based on the level of work the process is being asked to do.

In our Mobile Platform Environmental Data system (MoPED) we decided to leverage the cloud computing resources of Amazon Web Services. We mainly rely on Amazon Kinesis, a service that accepts data and keeps it in a pre-defined number of queues to be consumed by another service, Amazon Lambda, a server-less processing system that can consume data from a Kinesis stream queue, run the data through a processing script, and then output to another AWS service, Amazon S3, an abstract storage service that has a nine ‘9’s durability and two ‘9’s availability design. Utilizing these services, we have been able to maintain 99.9% up-time while processing approximately 1 million observations per day and providing 2.5 Gb of textual weather data to the weather community every day while only having a single system administrator.