This study examines the effectiveness of ensemble Kalman filters in data assimilation with the strongly nonlinear dynamics of the Lorenz-63 model, and in particular their use in predicting the regime transition that occurs when the model jumps from one basin of attraction to the other. Four configurations of the ensemble-based Kalman filtering data assimilation techniques, including the ensemble Kalman filter, ensemble adjustment Kalman filter, ensemble square root filter and ensemble transform Kalman filter, are evaluated with their ability in predicting the regime transition (also called phase transition) and also are compared in terms of their sensitivity to both observational and sampling errors. The sensitivity of each ensemble-based filters to the size of the ensemble is also examined.

It is found that: 1) ensemble-based Kalman filters can help improve the prediction of the nonlinear dynamics during the phase transition; 2) all methods are sensitive to observational errors; when observational errors become larger, the forecast of nonlinear system tends to be more uncertain. However, different filters may have different abilities to tolerate the observational errors; 3) accuracy of the analysis depends on the size of ensemble in the analysis cycle, and in order to achieve the same accuracy one method may require a smaller ensemble size than another; 4) phase transition prediction skill is sensitive to the magnitude of the inflation factor, implying that sampling error may be present in these experiments.