Combining human and computer generated forecasts using a knowledge based system

It was with the following words that Stern began his 2005 presentation, "Defining cognitive decision making processes in forecasting: a knowledge based system to generate weather graphics":

"'Combining forecasts by mathematically aggregating a number of individual forecasts increases the reliability of forecasts (Kelley, 1925; Stroop, 1932) and averages out unsystematic errors (but not systematic biases) in cue utilization. A common method for combining individual forecasts is to calculate an equal weighted average of individual forecasts' (Armstrong, 2001). The work to be presented here represents a vindication of Armstrong and others, who have long advocated combining forecasts. It suggests adopting a strategy, in the context of predicting Melbourne's weather, that has the potential to bring about forecasts that are substantially more accurate than those currently issued officially."

What Stern had set out to achieve was to attempt to objectively describe these decision-making processes via a computerised weather forecasting system (the knowledge based system) that operates in a manner to replicate the processes currently used by human forecasters in their work. He saw, in this context, that a test of how successfully the said processes are being replicated by the computerised weather forecasting system lies in the extent to which:

- The system's output on an individual day-to-day basis; and,

- The system's overall accuracy across an extended period;

mimic the day-to-day output and overall accuracy of the human (officially issued) product.

That some success was achieved in this attempt to replicate the cognitive decision making processes in forecasting was confirmed by the closeness of the overall percentage variances of the observed weather explained by the two sets of forecasts during a 100-day trial. However, less than half of the variance of the human forecasts was explained by the system's forecasts. This indicates that, on a day-to-day basis, there are significant aspects of the processes employed in deriving the human forecasts that are not taken into account by the computer forecasts, and vice versa.

Regarding the two sets of forecasts as partially independent and utilising linear regression to optimally combine the estimates of minimum temperature, maximum temperature, precipitation amount, and precipitation probability, lifted the overall percentage variance of observed weather explained to well above that explained by either the official forecasts or the system's forecasts. This suggests that adopting such a strategy of optimally combining the human and computer generated predictions has the potential to deliver a set of forecasts that are substantially more accurate than those currently issued officially.

It is the purpose of the present work to present the results of a new trial, conducted with a fresh set of independent data, whereby:

(1) Human and computer generated forecasts are combined automatically

(via http://www.weather-climate.com/graphicgenerator_sevendays_version3jul2005.html)

to yield another set of predictions; and,

(2) The accuracy of the new set of combined predictions are evaluated.

REFERENCES

Stern H (2005a) Defining cognitive decision making processes in forecasting: a knowledge based system to generate weather graphics. 21st Conference on Weather Analysis and Forecasting/17th Conference on Numerical Weather Prediction. American Meteorological Society, Washington, 1-5 August, 2005.

Stern H (2005b) Using a knowledge based forecasting system to establish the limits of predictability. 21st Conference on Interactive Information and Processing Systems, San Diego, California, USA 9-13 Jan., 2005.

Stern H (2004a) Using a knowledge based system to predict thunderstorms. International Conference on Storms, Storms Science to Disaster Mitigation, Brisbane, Queensland, Australia 5-9 Jul., 2004.

Stern H (2004b) Incorporating an ensemble forecasting proxy into a knowledge based system. 20th Conference on Interactive Information and Processing Systems, Seattle, Washington, USA 11-15 Jan., 2004.

Stern H (2003) Progress on a knowledge-based internet forecasting system. 19th Conference on Interactive Information and Processing Systems, Long Beach, California, USA 9-13 Feb., 2003.

Dawkins S S (2002) A web-based swell and wind forecasting tool. 9th Conference of the Australian Meteorological and Oceanographic Society, Melbourne, Australia, 18-20 Feb., 2002.

Stern H and Parkyn K (2001) A web-based Melbourne Airport fog and low cloud forecasting technique. 2nd Conference on Fog and Fog Collection, St John's, New Foundland, Canada 15-20 Jul.,2001.

Armstrong J S (2001) Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic Publishers.

Stroop J R (1932) Is the judgment of the group better than the average member of the group? Journal of Experimental Psychology, 15, 550-560 (as quoted by Armstrong, 2001).

Kelley T L (1925) The applicability of the Spearman-Brown formula for the measurement of reliability. Journal of Educational Psychology, 16, 300-303 (as quoted by Armstrong, 2001).

WEB SITE

http://www.weather-climate.com/graphicgenerator_sevendays_version3jul2005.html