Combining nonlinear DA and generative ML methods for fast high-dimensional Bayesian Inference

Extreme events are often characterized by highly nonlinear interplay of several processes. To predict these events well, one has to ensure that all information in earlier observations is incorporated in the forecast, asking for fully nonlinear data assimilation and/or machine learning techniques. While it is possible to learn machines to focus only on extreme events, it is much more practical if the prediction system can handle the whole range of atmospheric phenomena. This is still an area where machines tend to perform less satisfactorily, and data assimilation is superior. Both field have made huge strides recently into the nonlinear regime, and combining their ideas will be most beneficial.

Here we will focus on combining recently developed fully nonlinear data assimilation methodology for high-dimensional systems, so-called Particle Flow Filters, with generative machine learning methods, specifically so-called diffusion methods. Particle Flow filters differ from Particle Filters in that they are not degenerate by construction, and they have been shown to work extremely well in high-dimensional highly nonlinear atmosphere and ocean settings. But they are expensive. Diffusion methods, such as GenCAST, have been shown to be highly successful for weather forecasting, based on ERA5 data, and, after complicated and expensive training, are computationally cheap.

It turns out that diffusion methods such as GenCAST can be extended such that they can incorporate observations using forecast ensembles. Present-day extensions are ad-hoc, but when combined with Particle Flow Filter ideas highly efficient and accurate methodologies can be derived. I will present the basic ideas with emphasis on applications, and demonstrate performance on a highly nonlinear squall-line case.