Hidden Error Variance Theory: Deriving Optimal Combinations of Static and Flow Dependent Variances

Hybrid 4D-VAR schemes employ forecast error covariance models that are linear combinations of static covariance models and flow-dependent ensemble based covariance models. Currently, to determine the optimal weights for the flow-dependent and static parts one must perform computationally expensive trials to compare each set of plausible weights. Since it is likely that the accuracy of ensemble based error covariances varies with vertical level and region, it also seems reasonable to assume that the optimal weights for a Hybrid covariance model would be a slowly varying function of latitude and height. However, the amount of tuning required to optimize such spatially varying weights would be prohibitively expensive. Here, we introduce and demonstrate a new method for deriving a spatially varying hybrid error covariance matrix, which optimally combines static and flow dependent error covariances. This method is based on a univariate model of the stochastic relationship between ensemble variance and error variance. If the assumptions of the univariate model were true, the approach would recover optimal weights for linear combinations of ensemble variances and climatological variances from archives of innovation and ensemble variance pairs. Our first experiment evaluates the assumptions of the univariate model applied to the Lorenz ‘96 model. In the second experiment, we test our derivation of the optimal hybrid error covariance matrix in a toy model setting, using an Ensemble Transform Kalman Filter (ETKF) with a perturbed observation ensemble update. These results are then compared to those found by tuning the weights of ensemble based versus static error covariances. Our last set of experiments creates spatially varying weights for our newly developed NAVDAS-AR Hybrid by dividing the atmosphere into six regions: low-level N.Hemisphere, Tropics and S. Hemisphere and upper-level N. Hemisphere, Tropics and S. Hemisphere. The performance of the NAVDAS-AR Hybrid using these spatially varying weights is then compared against its performance using non-spatially varying weights derived by brute force tuning. At low resolution, we found that the spatially varying weights derived using the univariate theory match the performance the invariant weights obtained from tuning. If available, higher resolution results will also be presented.