Warnings (Tornado) Facts: Considerations, Reflections, and Perspectives on 30 Years of Warning Performance Metrics in the NWS

One of the most important and visible forecast products issued by National Weather Service (NWS) Forecast Offices is the tornado warning. Understanding how the quality of those warnings has changed over time is of interest both from an academic standpoint, as well as from the perspective of making realistic policy goals for the warning program. At a basic level, a variety of metrics, such as the probability of detection (POD), false alarm ratio (FAR), critical success index (CSI), and probability of false detection (POFD) can be derived from the 2x2 contingency table. In addition, the time between the issuance of a warning and the occurrence of the tornado (the so-called “lead time”), as well as descriptors of the warning polygons in terms of area and length of time, can be calculated. We have analyzed warnings from 1986-2015, using the archive at the NWS Performance Management. This period is of interest because of the changes in warning procedures associated with radar developments, the transition from county-based warnings to storm-based warnings, and a growing emphasis on user response. The analysis builds on previous work that the lead author participated in that looked at the 2x2 table elements (Brooks, Bull. Amer. Meteorol. Soc., 2004) and the lead time (Erickson and Brooks, AMS Severe Local Storms Conference, 2006).

From 1986-2011, the annual national POD increased from ~0.30 to ~0.75 without significant changes in the FAR. Over that time period, the mean annual lead time for tornadoes that had warnings issued prior to the tornado was effectively constant at 18.5 minutes, as was reported by Erickson and Brooks (2006). Large changes took place beginning in 2012. The POD dropped by ~0.15 and the FAR dropped by ~0.06. Lead time for warned tornadoes decreased to approximately 15 minutes. Coupled with this, warning duration and warning area decreased. These changes in performance, at a time when no official change in warning procedures took place, are consistent with a raising of the threshold for the weight of evidence to issue a warning (Brooks 2004). Warning forecasters have been behaving in a manner that seems to emphasize limiting false alarms. Because of the relative rarity of tornadoes, changes in the FAR are associated with changes in the POD that are almost twice as large. Thus, large decreases in the POD are concomitant with small decreases in the FAR.

We have also looked at warning performance as a function of the damage rating and length of time the tornado lasted. In general, performance is better for stronger, longer-lasting tornadoes. Paradoxically, the answer to the question of whether official performance goals are met is dominated by performance associated with weak, short-lived tornadoes.