Wednesday, 25 January 2012: 11:30 AM
Misrepresentation of Model Performance by Root-Mean-Squared Errors: From Mathematical Proof to Case Demonstration
Room 238 (New Orleans Convention Center )
Root-Mean Squared Error (RMSE) has long been used as a performance metric for model evaluation. However, RMSE can at times misrepresent model performance. In this presentation total mean squared error (MSE) is decomposed into two parts: the error due to difference in the mean and the error due to difference in pattern variation. The later depends on standard deviation over the verification domain and anomalous pattern correlation to observation/analysis. If a forecast has a larger mean bias than the other, its MSE can still be smaller if it has much smaller error in pattern variation, and vice versa. It is further demonstrated that the error of pattern variation (Ep) is unbiased and can be used as an objective measure of model performance only if the anomalous pattern correlation R between forecasts and analysis is one (or very close to one). If R is smaller than one, Ep is biased and favors smoother forecasts that have smaller variances. A normalized Ep is proposed; however, it is still biased and favors forecasts with smaller variance if R is not perfect. The NCEP GFS forecasts with different model configurations are used for demonstration. It is recommended that an unbiased model verification should include anomalous pattern correlation, ratio of forecast variance to analysis variance, error of mean difference, and error of pattern variation.
Supplementary URL: