Unbiased Fully Nonlinear Data Assimilation: the Stochastic Particle Flow Smoother

The significance of nonlinearities in geosciences numerical models and in observation operators demands more advanced data assimilation techniques. The Particle Flow Filter (PFF) is a fully nonlinear and efficient sequential Monte Carlo filter that removes the weight degeneracy in particle filters. The deterministic version of the PPF has been successfully applied to high-dimensional systems. With a small number of particles, however, it can exhibit low-bias ensemble spread, particularly in observed parts of the state space. To address this challenge, we extend the PFF in two ways. First, we developed a stochastic version incorporating Gaussian noise into the Stein Variational Gradient Descent dynamics, ensuring unbiased ensemble spread for any ensemble size. Second, we extended the filter to a smoother, resulting in the Stochastic Particle Flow Smoother (SPFS). The SPFS has computational similarities with the Ensemble of 4DVars during minimization but utilizes interacting particle systems, allowing it to solve the fully nonlinear smoother problem.

We evaluate the performance of the SPFS in high-dimensional systems such as the 1000-dimensional Lorenz-96 model. Our results demonstrate that the SPFS successfully avoids particle collapse and reduces bias in ensemble spread compared to the PFF. Additionally, the smoother solutions better capture particle evolutions within assimilation windows, further reducing analysis errors. By using an adjoint model for propagating the observation information, the SPFS effectively handles model nonlinearity. We also present results from a high-dimensional MOM6 ocean model, demonstrating that we are very close to solving the nonlinear data assimilation problem.