Spatial Forecast Verification: Putting Location-Based Measures to the Test with a New Set of Geometric Cases

A major issue when verifying high-resolution forecasts concerns the double penalty problem associated with an otherwise very good forecast that is displaced a bit in space, which can lead to poor verification measures when using traditional methods. Moreover, information about size, shape and displacement of forecast event areas (e.g. size, shape and palcement of a storm area) need to be diagnosed. Therefore, many new measures of such errors based strictly on events (e.g., exceeding a threshold) have been proposed in the literature. Some are applied to an entire field and some are applied within an object- or feature-based framework on to one or a small cluster of connected components within the field. However, because most of the measures have been previously proposed for other purposes (such as image analysis), it is unclear what properties are important in the verification setting, and how each method describes various errors. For example, the centroid distance is a true mathematical metric, and therefore contains good properties from a theoretical point-of-view. However, it is a metric based on the centroid of a feature. In the attached image, the large ring around the circle has a perfect score for this measure (zero), whereas the right circle is much closer in size and shape, but slightly displaced yielding a positive centroid distance. Learning about such properties for different measures will aid in the interpretation and use of each one. This exhaustive study introduces a new set of geometric test cases in order to gauge the properties of these methods. Several such methods are also employed.