Precipitation Nowcasting Leveraging Deep Learning and HPC Systems to Optimize the Data Pipeline

Automating very short-term precipitation forecasts can prove a significant challenge in that traditional physics-based weather models are computationally expensive; by the time the forecast is made, it may already be irrelevant. Deep Learning offers a solution to this problem, as that a computationally dense machine can train a neural network ahead of time using historical data and deploy that trained network in real-time to produce a new prediction within seconds or minutes. By pairing large-scale distributed systems and deep learning technology, our team intends to demonstrate the full capability of machine learning in short term weather prediction. An end-to-end data pipeline is constructed in such a way as allow rapid dataset iterations on a Spark-based analytics platform and accelerated data-parallel neural network training on a high density GPU system. The short-term precipitation forecasting system is based on a pre-trained neural network model consisting of convolutional recurrent neural network cells. A Convolutional Long Short-Term Memory (ConvLSTM) network was chosen for its unique ability to extract spatial as well as temporal relationships within a dataset. This neural network cell was implemented in an encoder-decoder structure that can both process and predict sequences of multi-dimensional tensors. Our approach was trained and validated against a dataset extracted from NOAA NEXRAD Level II data from various sites across the continental United States. The neural network model ingests a cartesian representation of multiple products derived from the radar scans including reflectivity, radial velocity and others. The length of the training history is allowed to vary in terms of the number of years of data the network will train on and the effects of a more thorough history are analyzed to determine an ideal dataset size. Multiple radar sites are included in training, both in terms of a general radar model and site-specific models that will learn local effects including orographic and climatic actors. Raw predictions are in the form of scaled reflectivity that can be directly compared to actual data derived from the radar sites. These values are converted into rainfall rates for analysis. In doing so, we are able to provide direct comparisons to both persistence as a baseline and more advanced precipitation nowcasting systems.