The Forecast Skill Card and the Diagonal Score

A general framework for the definition of proper user-oriented scores is set up using the concept of elementary scoring rules (Ehm et al. 2016). Generalizing the concept of Murphy diagrams, a new diagnostic tool is introduced that provides a complete overview of forecast skill. This so-called forecast skill card displays the forecast value as a function of two decision parameters, the probability level and the event threshold, simultaneously. Standard scores such as Brier score, quantile score and continuous ranked probability score can directly be related to the skill card. This provides a consistent interpretation of each score as the overall value to a particular set of forecast users.

Based on this new diagnostic skill card, a new score is proposed for the assessment of univariate probabilistic forecasts. The so-called diagonal score emerges as a summary score focusing on the diagonal of the skill card; that is by fixing the relationship between event base rate and forecast probability threshold. Properties and possible interpretations of the new score are discussed and results are presented based on forecasts of the operational ECMWF ensemble forecasts.

Ehm W., T. Gneiting, A. Jordan, F. Krueger, 2016. Of quantiles and expectiles: Consistent scoring functions, choquet representations, and forecast rankings. J.R. Statist. Soc. B 78: 1–29.