A Monte Carlo-Based Adaptive Kalman Filtering Framework for Soil Moisture Data Assimilation

Soil moisture data assimilation most commonly pertains to state updating using the well-known Kalman filter (KF) or its variants (e.g., the Ensemble KF; EnKF), while aiming to ingest in situ and/or satellite measurements into modern land surface models.The success of any data assimilation (DA) system hinges on reliable estimates of the uncertainties of the observations and of the model into which these observations shall be assimilated. Often, these uncertainty estimates are tuned manually to optimize internal data assimilation diagnostics and/or the skill of DA analyses with respect to independent reference data. Alternatively, so-called adaptive DA techniques estimate model and observation uncertainties as integral part of the DA system. However, the existence, or more precisely, a lacking knowledge of error auto- and/or cross-correlation often limits the performance of such adaptive methods. We propose a novel Monte Carlo based adaptive Kalman filtering framework (MadKF) that harnesses the well-known triple collocation analysis (TCA; applied to the observation, open-loop model forecast, and DA analysis time series) to obtain the error variance information necessary to parameterize an EnKF-based state updating scheme. Error cross-correlations, which occur due to the reliance on non-independent data triplets and bias TCA estimates, are diagnosed and corrected for using Monte Carlo simulations, i.e. model and observation ensembles. The proposed MadKF is tested in a synthetic environment and by assimilating real satellite soil moisture retrievals from the Advanced SCATterometer (ASCAT) into the Antecedent Precipitation Index (API) model forced with daily aggregated Multi-Source Weighted-Ensemble Precipitation (MSWEP) data. Synthetic experiments indicate a good convergence of model and observation uncertainty estimates. Internal DA diagnostics, i.e. the innovation auto-correlation (IAC) and the variance of the normalized innovations, asymptotically converge to their desired values, which indicates that the filter is operating near its optimum and reliably estimates analysis uncertainty. Real-data experiments assimilating ASCAT observations into the API model further indicate that the MadKF is robust against observation error auto-correlations, which typically cause problems in conventional IAC-tuning based adaptive filtering approaches. A performance evaluation over 264 in situ sites within the contiguous United States shows that the MadKF leads to a significant skill gain in surface soil moisture estimation.