Testing the Distance-Based Spatial Forecast Verification Methods on a New More Rigorous Set of Contrived Cases

In order to obtain more useful verification information for increasingly high-resolution forecasts, especially those such as quantitative precipitation forecasts (QPF), or cloud amount forecasts, where double penalties caused by small displacement/timing errors are incurred, numerous new methods were introduced in the last decade. Each approach focused on different aspects of issues related to verifying the newer and generally "better" forecasts. Some were aimed at the need for more diagnostic information, others attempted to provide similar measures as those previously used while accounting for spatial nuances, and others targeted specific issues, such as displacement errors. One subset of methods were primarily introduced within the context of more complicated ones, but have ultimately proven to be useful in and of themselves. These methods are specifically aimed at informing about the similarity/difference between the spatial patterns of forecasts when applied to the entire domain, or specific storm areas when applied in the context of a feature-based approach. Included in the spatial pattern information are spatial displacements. For example, some methods only inform about spatial displacement, such as the centroid distance. Since their introduction, issues with many of these methods have come to light, such as sensitivities to the domain, to the relative position of shapes to each other, the placement of two spatial features within the domain, edge effects, and more. While some methods were tested on a set of geometric cases, which proved invaluable in the assessment of their behavior, it became clear that a new set of cases was necessary to fully vet each approach, and inform potential users about the particular properties, as well as the pros and cons, of each method. The cases can also be used for additional methods that could be proposed in the future, thereby supplying a standard set of cases to be employed. This talk will introduce the cases while demonstrating the properties of several of the techniques with them.