Evaluating Weather Forecasts in Terms of Two Measures of How Accurately a Set of Future Events Have Been Predicted—Intensity and Timing

Two approaches to forecast evaluation are examined. The first approach explored, addresses the capability of a forecast system to correctly indicate the intensity of a set of weather events. This approach has as its focus the percent departure from the seasonal normal in the observations that is, the actual variance, which is explained by the forecasts. The second approach explored, addresses the capability of a forecast system to correctly time a set of events. This approach has as its focus the percent of the day to day changes in the observations, that is, the inter-diurnal variance, which is explained by the forecasts. Two sets of forecasts are evaluated utilising these approaches:

· A set of the official forecasts (out to Day-7) for the Australian city of Melbourne; and,

· A set of forecasts (also for the Australian city of Melbourne) generated automatically via an algorithm (out to Day-14) interpreting the GFS NWP model forecasts.

There are six weather elements, whose forecasts are evaluated:

· Minimum temperature;

· Maximum temperature;

· Amount of precipitation;

· Probability of precipitation;

· 9am wind vector (wind direction and speed); and,

· 3pm wind vector (wind direction and speed).

An illustration of the two approaches shall now be presented, by referencing the amount of precipitation forecasts. Fig 1 depicts trends in the accumulated skill at predicting precipitation intensity for Day-1, Day-2, - - - Day-7 (for the official predictions) and for Day-8, Day-9, Day-10 (for the GFS NWP model-based predictions). It demonstrates an overall increase in skill for most lead times, the Day-5 official predictions being now as skilful as the Day-1 official predictions were less than two decades ago. Some limited skill is shown to be emerging for GFS predictions out to Day-10.

Fig 2 depicts trends in the accumulated skill at predicting precipitation timing for Day-1, Day-2, - - - Day-7 (for the official predictions) and for Day-8, Day-9, Day-10 (for the GFS NWP model-based predictions). As for Fig 1, Fig 2 also demonstrates an overall increase in skill for most lead times, albeit at a somewhat lower performance level.

An application of how the two approaches to forecast verification (intensity and timing) may be applied in evaluating the predictions of individual weather events. To illustrate, several significant rainfall events from mid-November 2017 to late-December 2017 are examined (refer, for example, to Stern, 2018):

· 18 Nov-2017: 24.8 mm

· 1-4 Dec-2017: 73.4 mm

· 7-8 Dec-2017: 25.2 mm

· 19-20 Dec-2017: 20.8 mm

Fig 3 depicts trends in the accumulated skill at predicting precipitation intensity for Day-1 during this period. A sharp decline in accumulated skill is noted for the 18-Nov event, a sharp increase for the 1-4 Dec event, with little change for the other two events. From Fig 3 it may be therefore concluded that the intensity of the 18-Nov event was very poorly forecast, the intensity of the 1-4 Dec event was very well forecast, whilst there was nothing special that may be concluded regarding how well the intensity of the 7-8 Dec event and the intensity of the 19-20 Dec event were forecast.

Fig 4 depicts trends in the accumulated skill at predicting precipitation timing for Day-1 during this period. A sharp decline in accumulated skill is noted for the 18-Nov event, a slight decline for the 1-4 Dec event, with a modest increase for the other two events. From Fig 4 it may be therefore concluded that the timing of the 18-Nov event was very poorly forecast, the timing of the 1-4 Dec event was slightly disappointing, whilst the timing of the 7-8 Dec event and the timing of the 19-20 Dec event were both reasonably well forecast.

Reference

Stern H 2018 Evaluating the accuracy of weather predictions for Melbourne leading up to the heavy rain event of early December 2017. Australia & New Zealand Disaster and Emergency Management Conference, Gold Coast, QLD, 21-22 May 2018.