Verification Diagrams Based on Forecast Value

Analyzing forecast performance in terms of the types of decision-makers that can benefit from using the forecast would help improve our understanding of the usefulness of the information. Forecasts are considered to have positive “value” when they result in beneficial decision-making by users (Murphy 1993). Several studies have shown that forecast value can be estimated using a simple 2x2 decision problem (e.g., Thompson and Brier 1955; Robber and Bosart 1996; Katz and Murphy 1997; Richardson 2000; Wandishin and Brooks 2002). For this simple decision problem, forecast value is expressed in terms of the benefit/utility of taking preventative action relative to the losses endured when adverse weather events occur. Decision-makers without access to forecast information are assumed to make a uniform “reference” decision (either “always protect” or “never protect”) based on the climatological event frequency. A “utility ratio” (the ratio of the cost of “false alarms” to the costs of “missed events” and “false alarms”) can be used to quantify a user’s sensitivity to forecast/misclassification errors in the 2x2 situation. Wandishin and Brooks (2002) showed that the range of users that can potentially realize positive value from the forecast had utility ratios ranging from the “success ratio” at the upper bound to the “miss ratio” at the lower bound. Verification diagrams can be created using normalized upper/lower bounds of utility ratios as the coordinate system. Other measures of accuracy, association, bias, and skill can be derived in terms of these dimensions, allowing for the display of the geometric relationships between these aspects of performance on a single diagram. At the conference, these issues will be discussed and various examples will be presented.