Operational Global Weather Radar Products Generation Using Python and Docker Container

In The Weather Company, an IBM business, a large variety of weather radar are acquired from different countries. Those data are cleaned, digest and processed in real-time to generate hundreds of different radar products to serve users within the company (e.g. radar data assimilation to global scale operational run of The Model for Prediction Across Scales, 24/7 customer response team) and customers outside company (e.g. weather.com, StormRadar mobile apps, aviation and media clients). In order to meet the business requirements of delivering real-time products with resilience to hardware faulty, availability under accidental networking issues, scalability under heavy requests, cost efficiency under different scenarios, a new architecture of radar processing system is being designed and developed to support operations. In this new radar processing architecture, Python programming language is extensively applied at every stage of the workflow of radar product generation. Legacy components including Open Radar Product Generator (ORPG) and Warning Decision Support System -- Integrated Information (WDSS-II) are completely rewritten using Python, with Cython and Number applied to improve performance at several critical locations. Each kind of radar data sources from different countries (volumetric data based on single stations, pre-mosaicked products, even web- scraped images) are ingested and converted into CF-Radial or plain raster formats, based on nature of data. Each component is packed into a Docker container image and data are piped between using LDM server. Each docker image is self-monitored by using "supervisord" application, while all containers are organized as a cluster managed by Kubernetes, running on Amazon Web Services. With Docker container techniques, automatic scaling based on load is possible, and it will help to achieve cost efficiency in normal weather day and ensure low-latency in heavy weather day. Additionally, deep learning algorithms (implemented based on Keras and Theano) are being adopted to merge feedback from final quality assurance team and external customers to increase the capability of quality control, identifying extreme weather events (e.g. tornadoes, flashing floods) from radar images. In summary, Python is proven robust and efficient in supporting operational radar data processing at global scale.