Quantifying How Positional and Timing Errors in the Prediction of Synoptic Scale Systems Extract a Penalty When Verifying Weather Forecasts

Recently completed is a piece of work exploring trends in the skill of weather prediction at lead times of 1 to 14 days for Melbourne, Australia (Stern and Davidson, 2015). Grams et al. note that “… traditional verification statistics severely penalize a precipitation system that may have been forecast with a small positional error or incorrect shape, with resultant low correlation coefficients, high root-mean-square errors (rmse), and poor values of categorical statistics (Ebert and McBride 2000; Baldwin and Wandishin 2002) …”. The same comment may be applied to the verification of predictions of other weather elements.

     The purpose of this presentation is two-fold. Firstly, the verification statistics documented by Stern and Davidson (2015) are updated. Secondly, the proposition discussed by Ebert and McBride (2000), Baldwin and Wandishin (2002) and Grams et al. (2006), in regard to how positional and timing errors in the prediction of synoptic scale systems extract a penalty, is explored utilising forecast verification data sets for minimum temperature, maximum temperature, amount of precipitation and probability of precipitation. By this means, the proposition is demonstrated to have validity in the context of predictions for a range of weather elements. The methodology employed to achieve this demonstration is to separate the inter-diurnal component of the percent variance of the observations explained by forecasts, from the total percent variance explained.

     Regarding minimum temperature, an overall increase in accuracy is evident, for Day-1 predictions from 50% explained fifty years ago to 85%, now, for Day 2-4 predictions from 40% twenty years ago, to 75%, now, and for Day 5-7 predictions from 20% fifteen years ago, to 50%, the level displayed by the Day-1 predictions fifty years ago, now. It may be shown that in regard to the inter-diurnal (that is, day-to-day) minimum temperature fluctuations, small positional and timing errors in the forecasting of major synoptic systems do, indeed, extract a penalty on account of the resultant errors in the prediction of the day-to-day fluctuations. To illustrate, for Day-1 predictions, the inter-diurnal component of the variance explained is less than 75%, whilst the total variance explained is greater than 80% - not a great difference, but a difference, nevertheless.  For longer lead times, the proportional difference grows, for Day-5 predictions, the respective components explained being 45% and 60%, whilst by Day-10, almost none of the inter-diurnal component of the variance is explained.

     Regarding maximum temperature, an overall increase in accuracy is evident, for Day-1 predictions from 50% explained fifty years ago to over 85%, now (with major errors occurring only rarely), for Day 2-4 predictions from 30% thirty years ago, to nearly 80%, now, and for Day 5-7 predictions from 20% fifteen years ago, to 50% (which is the level displayed by the Day-1 predictions fifty years ago), now. Some skill, of a modest level (about 15%), is displayed by the Day 8-10 predictions. As for minimum temperature, it may be shown that small positional and timing errors in the forecasting of major synoptic systems extract a penalty on account of errors in the prediction of the day-to-day fluctuations. To illustrate, for Day-1 predictions, the inter-diurnal component of the variance explained is less than 80%, whilst the total variance explained is greater than 85% - once again, not a great difference, but a difference, nevertheless. For longer lead times, the proportional difference grows, for Day-5 predictions, the respective components being also 50% and 65%. Also as for minimum temperature, by Day-10, almost none of the inter-diurnal component of the variance is explained.

     For amount of precipitation forecasts, an overall increase in accuracy is evident, albeit somewhat unsteady, with a peak shown during the very wet summer of 2010-2011 when some extreme events were well predicted. It may be shown that small positional and timing errors in the forecasting of major synoptic systems extract a far greater proportional penalty (than for temperature predictions) on account of errors in the prediction of the day-to-day fluctuations. To illustrate, for Day-1 predictions, the inter-diurnal component of the variance explained is about 50%, whilst the total variance explained is about 60%, a fairly great difference. For longer lead times, the proportional difference grows more rapidly (than for temperature predictions). By Day-5, only 5% of the inter-diurnal component of the variance is explained (reflected in the skill at predicting 'wet' days).

     Regarding probability of precipitation, improvement is evident for Day-1 and Day 2-4 predictions, but not for longer lead times. As for the amount of precipitation, it may be shown that small positional and timing errors in the forecasting of major synoptic systems extract a far greater proportional penalty (than for temperature predictions) on account of errors in the prediction of the day-to-day fluctuations. To illustrate, for Day-1 predictions, the inter-diurnal component of the variance explained is about 40%, whilst the total variance explained is about 55%, a fairly great difference. For longer lead times, the proportional difference also grows more rapidly (than for temperature predictions), by Day-5, only about 10% of the inter-diurnal component of the variance is explained. 

     To conclude, it is shown how one may quantify the extent to which positional and timing errors in the prediction of synoptic scale systems extract a penalty when traditional approaches to the verification of weather forecasts are applied. The penalty is shown to be proportionally greater for precipitation predictions than for temperature predictions. This may be due to the fact that whilst most day-to-day changes in temperature are gradual, not withstanding the impact of the occasional sharp changes associated with the passage of cold fronts, most significant precipitation events are over within a day or two. The relevance of the two different approaches to forecast verification, total variance and inter-diurnal variance, depends upon the needs of the client. The inter-diurnal approach is more relevant to those planning for a particular activity on a certain day, for example, a wedding or a sporting event. The total approach is more relevant to those planning for activities that stretch across a longer period, for example, hay making or an extended holiday.