23rd Conference on IIPS

2B.1

INCREASING FORECAST ACCURACY BY MECHANICALLY COMBINING HUMAN AND AUTOMATED PREDICTIONS USING A KNOWLEDGE BASED SYSTEM

Harvey Stern, BOM, Melbourne, Vic., Australia

The author has previously presented work providing an analysis of the relative skill displayed by human predictions (the current official forecasts), and by automated forecasts generated by a knowledge based system. The work suggested that adopting a strategy of mechanically combining human and automated predictions has the potential to lift the percentage variance explained by human predictions by 7.9%. Whilst encouraging, this suggestion needed to be validated by a fresh real-time trial conducted on a new set of independent data.

Forecast verification data from a new 100-day real-time trial, conducted from August to November 2005, on a fresh set of independent data, was analysed. For this new trial, the system was modified to mechanically combine human and automated predictions. The trial confirmed the previous work, the percentage variance explained being lifted by 7.7%.

In the light of the results of the 100-day trial, a number of minor modifications were made to the computer system (for example, from 27 December 2005, Day-1 wind forecasts were generated by a combining procedure; previously, they had been automated). The real-time trial was then continued. After 300 Day-1 to Day-7 Melbourne forecasts from 20 August 2005 to 15 June 2006, inclusive, that is, 2100 individual predictions, the average lift in percentage variance of weather (rainfall amount, sensible weather, minimum temperature, and maximum temperature) explained was 8.1% over that explained by the current official forecasts (refer to Figure and Table), the addition of more independent data further affirming the result suggested by the earlier work.

There is increasing interest in the question of what might be the appropriate future role for the human in the forecast process. The results presented here show that automated forecasts are unable (by themselves) to fully replicate the decision making processes of human forecasters. Similarly, the results also show that human forecasters (by themselves) are unable to optimally integrate into the forecasting process, guidance from automated predictions.

Two or more inaccurate but independent predictions of the same future events may be combined in a very specific way to yield predictions that are, on the average, more accurate than either of them taken individually. This is because, when combined mechanically, each "brings to the table" a different knowledge set.

That the combined forecasts are more accurate than individual currently available predictions taken separately, provides companies involved in forecasting with a potential competitive advantage over their peers should they choose to adopt a strategy of mechanically combining predictions.

With automated forecasts unable to fully incorporate human forecasters' valuable domain and contextual knowledge, there should be a need for the human forecaster well into the future. This future role may be as an input to a system that mechanically combines human predictions with automated forecasts.

 

Weather

Element

Verification

Parameter

Official

Forecasts

Combined

Forecasts

Comment

All Elements

% Variance Explained

35.1

43.2

Combined forecasts, are, overall, more accurate

Rain[1] or No Rain

% Correct

71.4

78.7

Combining forecasts yields more correct forecasts of rain occurrence

√(Rainfall Amount)

% Variance Explained

20.1

25.5

Combined rainfall forecasts are more accurate

...

RMS Error (mm0.5)

1.06

0.99

Combined rainfall forecasts are more accurate

...

Forecast Volatility[2] (mm0.5)

0.67

0.44

Combined rainfall forecasts are more consistent

Sensible Weather[3]

% Variance Explained

26.1

37.1

Combined forecasts of sensible weather are more accurate

...

Forecast Volatility (%)

19.4

11.7

Combined forecasts of sensible weather are more consistent

Min Temp

% Variance Explained

42.3

48.8

Combined minimum temperature forecasts are more accurate

...

RMS Error (ºC)

2.46

2.32

Combined minimum temperature forecasts are more accurate

...

Forecast Volatility (ºC)

1.44

1.25

Combined minimum temperature forecasts are more consistent

Max Temp

% Variance Explained

51.6

61.4

Combined maximum temperature forecasts are more accurate

...

RMS Error (ºC)

3.00

2.64

Combined maximum temperature forecasts are more accurate

...

Forecast Volatility (ºC)

2.03

1.48

Combined maximum temperature forecasts are more consistent

Thunder[4]

Critical Success Index (%)

19.8

22.8

Combined thunderstorm forecasts are (overall) superior …

Probability of Detection  (%)

23.2

37.5

With more “hits” …

False Alarm Ratio (%)

42.9

63.2

But, at a price of more “false alarms”

Fog[5]

Critical Success Index (%)

15.5

19.1

Combined fog forecasts are (overall) superior …

Probability of Detection  (%)

18.4

29.1

With more “hits” …

False Alarm Ratio (%)

50.7

64.1

But, at a price of more “false alarms”

Wind Speed[6]

% Variance Explained

46.7

53.5

Combined forecasts of wind speed are more accurate

Wind Direction

% Correct Within Half-Octant

65.9

70.0

Combined forecasts of wind direction are more accurate

 



[1]The official Amount of Precipitation forecasts are expressed in terms of rainfall ranges and, for verification purposes, the Amount of Precipitation forecast is taken to be the mid-point of the range forecast:

    Range 0 = No precipitation; Range 1 = 0.2 mm to 2.4 mm (1.3 mm); Range 2 = 2.5mm to 4.9mm (3.7 mm); Range 3 = 5.0mm to 9.9mm (7.5mm); Range 4 = 10.0mm to 19.9mm (14.9mm); Range 5 = 20.0mm to 39.9mm (29.9mm); Range 6 = 40.0mm to 79.9mm (59.9mm); and, Range 7 = 80.0mm or more (119.9mm). 

[2]RMS inter-diurnal change from Day-7 to Day6 to … to Day-1 forecast.

[3]Implied by probability of precipitation estimates.

[4]For verification purposes, it is said that there has been a thunderstorm in the metropolitan area during a particular day when at least one of the 0300, 0600, 0900, 1200, 1500, 1800, 2100, or 2400 Melbourne CBD and/or Melbourne Airport observations include a report of cumulonimbus with an anvil and/or lightning and/or funnel cloud and/or thunder (with or without precipitation).

[5]For verification purposes, it is said that there has been fog in the metropolitan area during a particular day when at least one of the 0300, 0600, 0900, 1200, 1500, 1800, 2100, or 2400 Melbourne CBD and/or Melbourne Airport observations include a report of fog (including shallow fog) and/or distant fog.

[6]Wind forecasts are verified only on the 171 sets of twice-daily (9am and 3pm) Day-1 forecasts from 27 December 2005, when combined wind forecasts were first generated.

extended abstract  Extended Abstract (324K)

wrf recording  Recorded presentation

Supplementary URL: http://www.weather-climate.com

Session 2B, International Applications - Part II (The last seven papers in this session are "overflow" papers from the joint session on Global Earth Observations with IOAS–AOLS)
Monday, 15 January 2007, 1:30 PM-5:30 PM, 217A

Next paper

Browse or search entire meeting

AMS Home Page