Showing Value in Classic ROC Performance...Diagrams

Evaluating forecasts of dichotomous events is a common problem in meteorology and other fields with decision making under uncertainty. Forecasts ranging from convective outlooks down to warnings, or numerical model forecasts of convective events using parameters such as updraft helicity all fall into this realm. Often, forecasts made from some continuous variable must be translated into dichotomous forecasts by applying a threshold. Typically, there is a tradeoff between kinds of errors (missed detections and false alarms) and the threshold can, at least conceptually, be based on an assessment of the net costs of those errors For a relatively simple, but signficant, decision problem, the net costs of different kinds of errors can be described by the misclassification cost ratio (MCR).

Assessing forecast quality by considering single metrics such as the probability of detection (POD), probability of false detection (POFD), or false alarm ratio (FAR) can lead to ambiguity. As a result, the visualization of two quantities, one focused on missed detections and the other on false alarms, has become more popular in the recent past. Specifically, relative operating characteristic (ROC) diagrams consider POD vs. POFD. A simple measure of the overall quality of the forecast system, based on signal detection theory as a model for the decision making problem, is d’, the separation of the distribution of the forecast variable associated with yes events and the distribution associated with the no events. Varying thresholds for the same forecast system (and subsequently the same d’) will lie along the same curve and the appropriate threshold that maximizes user value can be set. In the absence of complete knowledge of user costs, a chosen threshold can be associated with the MCR of the user(s) with a decision problem that matches the threshold. Similarly, performance diagrams plot POD vs. the Success Ratio, which is 1-FAR. Although there is no concept directly related to d’ on the performance diagram, it is possible to plot d’ curves on the performance diagram and compare them to the Critical Success Index (CSI), a quantity that considers the tradeoff of missed detections and false alarms as the intersection of the forecast and events over the union of the forecasts and events.

On both ROC and performance diagrams, each point is associated with a unique combination of d’ and implied MCR of the decision problem. By a coordinate transformation into d’/MCR space, we can separate the forecast system performance into descriptors of quality and potential value. This can help improve interpretation of forecast systems and provide insight into how thresholds should be set. We will provide examples of the use of this approach from convective outlooks, convective warnings, and high-resolution numerical model forecasts.