Toward Fully Non-Parametric Data Assimilation for Numerical Weather Prediction

Recent data assimilation developments for weather models have permitted the use of particle filters with ensemble sizes that are affordable by operational prediction centers. These methods operate by extending importance weights into vectors, which can be localized in the same manner as ensemble Kalman filters. The calculation of weights only requires that a function be specified for the likelihood of ensemble members (also called “particles”) based on environmental measurements. These likelihoods depend mostly on choices for error distributions for measurements and are typically assumed to be fixed and follow a Gaussian shape. For particle filters, there are few restrictions on the choice of error distribution for observations, thus providing an alternative pathway for assimilating observations exhibit regime-dependent or non-Gaussian errors.

The current study exploits the flexibility of particle filters to explore new methodology for using innovations accumulated during data assimilation to estimate likelihood functions in the presence of unknown distributions for errors in measurements or measurement operators. The resulting likelihoods can then be used for state or parameter estimation. We explore the use of kernel-estimated likelihood functions trained from prior innovations accumulated over time windows and regime-dependent likelihoods represented using kernel embeddings of conditional distributions. This approach allows for non-Gaussian estimates of likelihood functions that can be used directly by particle filters—or used to compute expectations of bias and error covariance for Gaussian-based data assimilation. Of equal importance, this methodology scales well for high dimensional applications, which are needed for capturing error dependence across observations. This research is motivated by challenges associated with sensors that provide “novel” environmental information, such as field measurements, satellite radiometers, and radar reflectivity, which measure quantities that are not easily validated by independent observing systems. The end result is a data assimilation methodology that is entirely “non-parametric” in that none of the required error distributions follow specific “shapes” determined by parameters (e.g., mean and covariance for a Gaussian).