CMIP5 Model-Analog Seasonal Forecast Skill: A Metric for Model Evaluation of ENSO

A recent study by Ding et al showed that tropical Indo-Pacific SST forecast skill from the North America Multi-Model Ensemble (NMME) can be matched or even exceeded with a simple "model-analog” method, applied to the corresponding long control runs from each NMME model, raising the possibility that any CGCM with a sufficiently long control run could similarly be used to produce skillful seasonal forecasts. In this talk, we test this idea by applying the model-analog method to preindustrial simulations made by 28 different CGCMs that are available as part of the CMIP5 database. Furthermore, we examine the effect of including historical radiative forcing on the model-analog forecasts.

Model-analogs are determined only within the tropical Indo-Pacific domain, using observed monthly SST and SSH anomalies. These initial observed anomalies were then “detrended” by removing the projected effects of historical radiative forcing which were determined from the time evolving global and ensemble mean from 45 CMIP5 radiatively forced (historical and RCP4.5) runs. Analog ensembles corresponding to the resulting “detrended” anomalies were then found in each of the 28 CMIP5 preindustrial control simulations. Their subsequent evolution serves as the model forecast. Retrospective forecasts of SST and precipitation during 1982-2010 were made for leads of 1-12 months in the tropical Indo-Pacific.

While most of the models generate skillful forecasts, the skill varies from model to model. For instance, the anomaly correlation between analog 6-month lead forecasts from the individual CMIP5 model and observations ranges from 0.46 to 0.75 for SSTs in the Nino3.4 region. The 28 models and a subset of 8 models, denoted as “best-8” because they had the highest correlations at a 6-month lead for Nino3.4 SSTA, were employed to construct grand model-analog forecast ensembles. In general, both the grand ensemble means yield better forecast skill than most of the single model ensemble mean. Furthermore, the “best-8” grand ensemble displays slightly better ENSO forecast skill than that of the 28-model grand ensemble mean. The “best-8” model-analog grand ensemble has deterministic and probabilistic forecast skill comparable to that of the NMME grand ensemble for both SST and precipitation in the ENSO region for lead times of 1-9 months. For longer lead times, the “best-8” model-analog grand ensemble mean also displays skillful forecasts of SST with no additional computational costs, an advantage of the model-analog technique.

The inclusion of the projected effects of historical external radiative forcing significantly improves the model-analog SST forecasts in the tropical Indian Ocean, the northwestern tropical Pacific and South Pacific, although not within the tropical central and eastern Pacific. The resulting SST forecast skill is comparable to the NMME hindcast skill in most parts of the tropical Indo-Pacific Ocean. Nevertheless, external forcing has a trivial effect on model-analog precipitation forecasts. Our results suggest that model-analog technique may allow an evaluation of the realism of each of the CMIP5 models, since we can compare how each model captures the observed evolution of ENSO.