The purpose of the current paper is to present comprehensive verification statistics for forecasts of weather elements, and to thereby document the accuracy, and trends in accuracy, of day-to-day medium range forecasts of weather for Melbourne. The forecasts are those prepared by operational meteorologists at the Australian Bureau of Meteorology's Victorian Regional Forecasting Centre. The data cover forecasts for minimum and maximum temperature since the 1960s, rainfall since the late 1990s, and qualitative descriptions of expected weather over the past year.
Until the 1980s, temperature forecasts in Australia were prepared for just the next 24 hours. At about that time, worded forecasts and predictions of maximum temperature out to four days were first issued to the public. From the late 1990s, this service was extended to minimum temperature. Experimental worded forecasts out to seven days, with corresponding predictions of minimum temperature, maximum temperature, and rainfall amount, were also commenced. Around 2000, these predictions were made available to special clients and, since early in 2006, they have been issued officially to the public.
Table 1 presents a summary of the current level of accuracy of Melbourne Day-1 to Day-7 forecasts (based on the most recent 12 months' of data). The percentage variance explained by the forecasts provides a measure of how successfully the predictions described the observed variations in the particular weather element. A perfect set of predictions explains 100% of the variance. By contrast, a set of predictions, that provides no better indication of future weather than climatology, explains 0% of the variance. The data therefore suggest that predictions of rainfall amount, minimum temperature, and maximum temperature all display positive skill out to Day-7. To verify forecasts of thunder and fog, the Critical Success Index (the percentage correct forecasts when the event is either forecast or observed) is used.
To verify predictions of precipitation, worded forecasts have been assigned to one of 5 categories. Figure 1 shows verification of Day-1 to Day-7 forecasts of measurable precipitation (0.2 mm or greater) over a 24-hour midnight-to-midnight period expressed as a probability. For category 1, “Fine or Fog then fine”, small probabilities indicate skilful forecasts. For category 5, “Rain or Thunder”, large probabilities indicate skilful forecasts.
Figure 1 also shows verification of Day-1 to Day-7 forecasts of precipitation expressed as amount of precipitation. It can be seen that a forecast of “Fine or Fog then fine” for Day-7 is associated with an average fall of only about 1 mm of rain, whilst a forecast of “Rain or Thunder” is associated with about 3.5mm of rain.
Figure 2(a) and Figure 2(b) show, respectively, 12-month running (calculated over the preceding 365 days) average errors of the minimum and maximum temperature forecasts, for which data back to the 1960s are available. The graphs show a clear long-term trend in the accuracy of theses forecasts. For example, Day-3 forecasts of minimum temperature in recent years (average error ~ 1.6°C) are as skilful as Day-1 forecasts of minimum temperature were in the 1960s and 1970s, whilst Day-4 forecasts of maximum temperature in recent years (average error ~ 2.0°C) are more skilful than Day-1 forecasts of maximum temperature were in the 1960s and 1970s.
Figure 2(c) compares the 12-month running (calculated over the preceding 365 days) average error of a Day-1 forecast of maximum temperature based upon the assumption of persistence (that is, the average inter-diurnal change in maximum temperature), with the actual forecasts. A number (but not all) of the troughs and peaks in the two graphs correspond. This indicates that, when the day-to-day variability in maximum temperature is high, the actual forecasts have slightly reduced skill, but the variability in the accuracy of the official forecasts is much less than the variability in the accuracy of the persistence forecasts.
Figure 3 shows time series of verification of Quantitative Precipitation Forecasts (QPFs) over the past 8 years for Day-1 to Day-7 forecasts. Since the year 2000, the percent inter-diurnal variance explained by the forecasts has increased from about 20% to 30% at Day-1, and from close to zero to about 15% at Day-7.
The author thanks colleagues Noel Davidson and Mark Williams for encouraging this work, reviewers Bob Seaman, Tony Bannister and Evan Morgan, for their helpful comments, and Terry Adair and Robert Dahni, for their development of the forecast verification data sets.
Table 1: The current level of accuracy of Melbourne's Day-1 to Day-7 forecasts.
| Element | Verification Parameter | Day-1 | Day-2 | Day-3 | Day-4 | Day-5 | Day-6 | Day-7 |
| √ (Rain Amount) | % Variance Explained | 39.8 | 36.2 | 30.6 | 25.0 | 18.4 | 10.9 | 6.6 |
| Min Temp | % Variance Explained | 74.8 | 59.0 | 53.7 | 47.4 | 28.3 | 23.0 | 13.9 |
| Max Temp | % Variance Explained | 79.7 | 71.7 | 62.4 | 55.7 | 40.6 | 31.1 | 20.5 |
| Thunder | Critical Success Index (%) | 27.9 | 26.1 | 25.6 | 16.3 | 11.6 | 9.5 | 5.1 |
| Fog | Critical Success Index (%) | 35.3 | 28.9 | 15.9 | 11.4 | 4.9 | 2.4 | 0.0 |
Figure 1 The probability and amount of precipitation occurring following the use of various phrases in Melbourne's forecasts (to view this and other Figures, click on the small images below the captions).
Figure 2(a) Trend in the accuracy of Melbourne's minimum temperature forecasts for Day-1, Day-2, … Day-7.
Figure 2(b) Trend in the accuracy of Melbourne's maximum temperature forecasts for Day-1, Day-2, … Day-7.
Figure 2(c) Trend in the average inter-diurnal change in Melbourne's maximum temperature.
Figure 3: Trend in the accuracy of the quantitative precipitation forecasts.
Go to http://www.weather-climate.com/abstracts.html for link to complete paper.
Extended Abstract (1.1M)
Recorded presentation