This study examines a new forecast verification system for NWS warnings that is based on the AWIPS WARNGEN polygon. A comparison of verification statistics is made between the current (county based) system and one using the polygons to see what differences might be expected in skill scores. Warnings and reports from the Kansas City/Pleasant Hill Missouri NWS Forecast Office (2001) were analyzed and statistics generated using both the current NWS verification system and a system based on WARNGEN polygons.

Preliminary data suggests that the use of polygons can improve verification false alarm scores. An examination of just the multi-county severe thunderstorm and tornado warnings (40 in all covering 86 counties) were analyzed. A warning was considered verified if a report of any type of severe weather was received within the warning time period and within the indicated area (either polygon or county). The office achieved a FAR of 0.38 using the current county verification system. Using the polygon verification system for the same warnings and reports, the FAR dropped to 0.15. An important assumption was made here - that for the counties warned for, a polygon exactly matched the county outlines so that any reports occurring in any county in the multi-county warning would verify the entire warning. A similar look at nine (9) multi-county flash flood warnings showed a county FAR of 0.50 (22 counties) while the polygon assumption produced a FAR of 0.22. Although the improvement indicated in the FAR is encouraging, a detailed analysis of the FAR and POD for all warnings is necessary, since some reports may have occurred outside the warning polygon, but served to verify the warning in the county verification system. This analysis is nearing completion with final results being presented in the paper. Our expectations are that the statistics for FAR will improve while the Probability of Detection (POD) will be reduced since some warnings may have been verified by other storms that enter the county outline during the warning time, but remain outside of the polygon.

The county warning forecast verification system actually plays a role in how warnings are issued. Forecasters may choose to leave a portion of a county out of the warning in an effort to reduce the FAR since the likelihood of getting a report is small (in rural areas and where a weak spotter network may exist), even though a threat of severe weather exists. If county boundaries were removed from the process, the forecaster can focus solely on scientific reasoning for defining the warning area and more clearly define the severe weather threat area. For extreme severe events, longer warnings can be issued covering areas further downstream, ahead of the severe weather. The effect of today’s county warning system introduces hesitation in the warning process as the forecaster waits to be more certain that severe weather will affect the downstream counties.

This paper recommends that the polygon verification system be adopted even if little change (or initial degradation) is observed in the statistics. The system provides a better measure of the service provided, verifying where the forecaster expected the severe weather threat without concern for geopolitical boundaries.