Extensions and alternative formulations of the ROC curve

The Receiver Operating Characteristic (ROC) curve is a commonly used measure of a forecast system's discrimination ability, that is, the ability to distinguish between an event of class 0 and an event of class 1. The ROC curve takes a paired series of probability of detection and probability of false detection values (obtained by varying the critical threshold probability) and plots the former against the latter, with the area beneath this curve being the measure of forecast discrimination.

Of course, this is not the only way this information can be displayed or the limit of how this information can be used. Alternative formulations of the ROC curve include the transformation into likelihood ratio coordinates. The POD and POFD data can be use in a straightforward manner to calculate positive and negative likelihood ratios. A positive likelihood ratio is the extent to which a forecast of ‘yes' increases, from the prior probability (or climo), the probability of a ‘yes' event occurring, and similarly for the negative likelihood ratio. One advantage of this transformation is that it does not display the asymptotic behavior of the ROC curve, where ‘good' curves will crowd the upper left corner of the plot making comparison difficult.

A commonly seen extension of the ROC curve is the relative-value curve. This graph uses the same POD, POFD pairs along with the prior probability of the class 1 event (climatology) to plot the value of a set of forecasts as a function of cost-loss ratio (i.e., the cost incurred by taken preventative action divided by the loss incurred if such action is not taken and the event occurs). Other measures conveying information of forecast utility include normalized-expected-cost curves and prevalence-value-accuracy curves.]

These extensions and alternative presentations of the traditional ROC curve will be presented along with an examination of their utility.