Challenges of incorporating the event-based perspective into verification techniques

Standard verification is built around the idea of forecast and observation pairs, whether continuous (e.g., today's high temperature will be X, the observed temperature was Y) or binary ('yes' it will snow tonight, 'no' snow was observed). A key feature of these verification data sets is regular forecast issuance and regular observation times. However, some forecast scenarios do not lend themselves to this standard approach. For example, forecasts of high-impact weather are issued only when the weather event threatens; in this case the null event is a default state. From both the forecast and observation perspective, events are positively identified while null events are implicit by the lack of an identified event. This can result directly from the nature of the event or from the way in which the forecast is used.

A number of questions arise in considering "event-based" verification. For example, if the null events are indicated only implicitly, then the number of null events is indeterminate. Without a proper count of null events (particularly, the 'correct no' quarter of the contingency table) many verification scores such as Peirce Skill Score, ROC curves, and Extreme Dependency Score cannot be calculated. Another question is how forecast and observed events are paired in order to calculate verification scores. Events could be matched based on the time of onset of the event, the time of the midpoint of the event, or on a completely different characteristic (e.g., maximum strength, event duration, etc.). Furthermore, the forecast and observed events may exist at different scales, so that several distinct observed events may occur within the time frame of a single forecast event. The choice made in how to account for this or any of the questions listed here will impact the resulting skill measures.

Questions such as these will be considered in two contexts: convective weather impacting the airspace in the vicinity of an airport, and turbulence along a flight path. These two contexts involve very different spatial scales, which will allow for an examination of the sensitivity different verification approaches have to the nature of the weather event being examined.