Data assimilation through machine learning methods

Recent research has shown that machine learning techniques have a wide range of application to meteorological problems. In an era when observations from land, atmosphere, ocean and space-based platform lead to massive amounts of data available to incorporate into models, techniques that efficiently process such observations are at a premium. Kalman filtering has been a stalwart method for the past few decades. Recent advances in such filters include the use of ensembles (now used operationally) and the extended Kalman filter (not used operationally for large-scale problems). Despite their popularity, Kalman filters and ensemble Kalman filters are suboptimal in the sense that they make unrealistic assumptions (e.g., linearity, prior knowledge of the “correct” atmospheric model, properties about the error covariances and questions about the “correct” number of ensemble members) and are not particularly efficient when the datasets become large. Machine learning may provide an alternative to Kalman filtering to predict the future of a dynamic system without any knowledge of the underlying physical model and make minimal assumptions about the data and error properties.

The aim of the present research is to assess the practicality for a computationally efficient alternative to the EnKF with machine learning and kernel methods. Three types of Kalman filters (Ensemble, Ensemble Square Root and Extended), considered state of the science in meteorology, are applied and compared the machine learning approaches for error as a function of the time step, level of chaos and problem complexity. Preliminary results on a free fall model, as well as on the 40 variable dynamical system of the Lorenz and Emanuel model, suggest that the kernel methods have similar RMSE to the Kalman filter approach. Advantages of the kernel methods include the ability for massive parallelization, where the algorithm can be divided into small spatially discrete independent subproblems. With EnKF, the observations over all grid points are considered simultaneously leading to massive covariance matrices and computational inefficiency, unless patches are selected or unrealistic assumptions about the statistical properties of the atmosphere and errors are made. Moreover, the computational complexity and numerical stability of the problem is enhanced using kernel methods.