An Extension of the Reliability Diagram for 3-Category Probability Forecasts

Full exposition of the performance of a set of forecasts requires examination of the joint frequency distribution of those forecasts and their corresponding observations. In settings involving probability forecasts this joint distribution has a high dimensionality, and communication of its information content is often best achieved graphically. This paper describes an extension of the well-known reliability diagram, which displays the joint distribution for probability forecasts of dichotomous events, to the case of probability forecasts for three disjoint events, such as "below-", "near-", and "above-normal". The resulting diagram, called the calibration simplex, graphs the calibration-refinement factorization of the joint distribution on the 2-simplex, which is an equilateral triangle. Characteristics and interpretation of the calibration simplex are illustrated using both idealized verification data sets, and the 6 to 10- and 8 to 14-day temperature and precipitation forecasts produced by the U.S. Climate Prediction Center.