Nowcasting Lightning Events Using a Cloud-Based Deep Learning Approach

In this talk, we develop a nowcasting algorithm to predict lightning events 30 minutes in the future using a modern computing and machine learning approach. The data comes from the GOES-16 satellite infrared channels and GLM channels. The prediction algorithm is formulated both using a traditional machine learning approach and using a modern approach that takes advantage of cloud computing and deep learning. In the traditional machine learning approach, feature engineering is performed manually and the datasizes are relatively small. Features are extracted from a sequence of infrared and lightning event grids and the label used is the presence or absence of a lightning event. A novel data selection approach is utilized to keep the training dataset relatively balanced. As a first step, manual feature engineering is dropped in favor of deep learning using convolutional networks employing many of the standard deep learning tricks. It is shown that the data size is insufficient is take advantage of such an approach, and a distributed preprocessing job is utilized to create an augmented dataset of sufficient quantity and complexity. On this dataset, the traditional GPU infrastructure fails to scale, and is replaced by custom ASIC chips for machine learning. All the work is carried out on a Jupyter notebook on a low-cost compute environment, with scaling delegated to powerful cloud-based components. It is hoped that, beyond this single example, the general approach will provide a template for a plethora of other applications involving remotely sensed imagery and numerical weather model outputs.