Extended Abstract (224K)2.2
Assessing the Skill of Yes/No Forecasts for Markov Observations
William M. Briggs, Weill Cornell Medical School, New York, NY; and D. Ruppert
Mozer and Briggs (2003) and Briggs and Ruppert (2003) recently introduced a new, easy-to-calculate economic skill score for usein yes/no forecast decisions, of which precipitation forecast decisions are an example. The advantage of this new climate skill score is that the sampling distribution is known, which allows one to perform hypothesis tests on collections of forecasts and to say whether a given skill score is significant or not.
Skill, as ever, is defined as improvement over an optimal naive prediction. We show that the optimal naive prediction depends on both the base rate (the climatology) of the event being forecasted, and the loss one would incur if one were to make an incorrect decision based on the forecast.
Here, we take the climate skill score and extend it to the case where the predicted series is first-order Markov in nature, of which, again, precipitation occurrence series are an example. We show that Markov skill is different and more demanding than is persistence skill. Persistence skill is defined as improvement over forecasts which state that the next value in a series will equal the present value. We also define the optimal naive prediction in the Markov case.
Surprisingly, it turns out that the form of the Markov skill score is identical to the climate skill score, making calculations simple. The distribution of the Markov skill is more complex than is the distribution of the climate skill score, however. The distribution for the Markov skill score is presented, and examples of hypothesis testing for precipitation forecasts are given. We graph these skill scores for a wide range of forecast-user loss functions, a process which makes their interpretation simple.
Supplementary URL: http://wmbriggs.com/public/weathermarkov.pdf
Session 2, Forecast Evaluation (Room 3A)
Tuesday, 13 January 2004, 8:30 AM-2:30 PM, Room 3A
Previous paper Next paper