A Method for the Verification of Probabilistic Tropical Cyclone Wind Speed Forecasts

Five day probabilistic wind speed forecasts have been provided by the National Hurricane Center and private forecast companies during the past several hurricane seasons to denote the likelihood that winds above a certain threshold (i.e. 34, 50, or 64 kt) will affect a location. However, there have yet to be any extensive verification techniques used to evaluate these forecasts. It is important to develop a verification technique in order to quantify the extent in which these forecasts accurately represent the level of uncertainty as to whether or not a certain location will receive winds of a certain threshold. Accurately depicting this level of uncertainty will allow interests that are being threatened by a tropical cyclone (TC) to better understand the level of risk that they face. This would allow those in the potential path to understand what exactly a 50 percent chance of tropical storm or hurricane force winds means. Does it mean half the area that has a 50 percent chance of hurricane force winds will receive hurricane force winds for this particular storm? Are areas that have a 50 percent chance affected more than 50 percent of the time? Understanding this would allow them to take the necessary actions to be better prepared.

Probabilistic weather forecasts are traditionally verified by comparing a large number of forecasts for a given location to the observed occurrences of the predicted event. An example would be verification of rainfall probabilities. This is not possible for tropical cyclones due to the relative rare occurrence of such events. Many seasons would be required in order to have a large enough dataset for meaningful verification. A new technique is thus required. The proposed new technique seeks to verify each 5 day wind speed forecast against a single set of observations. This is possible as TCs are not point locations. Instead, they cover a large area. Thus, for the TC case, the probabilistic forecast is defined as the percentage of a given area that is expected to receive winds of a certain threshold. 5 day wind speed probabilities will be interpolated onto a .1o by .1o grid. This generates several hundred locations at which verification can be made. The forecasts that will be used in this study are the ImpactWeather tropical cyclone wind speed probabilities. These are available every 6 hours for the length of a TC forecast, up to 120 hours. The observations that will be used will consist of H-WIND data from the Hurricane Research Division of NOAA. The verification will be made at each data point, allowing for a sufficiently large number of forecasts for verification to be meaningful.

Reliability Diagrams (Kay and Brooks, 2000) are often used to qualitatively verify probabilistic forecasts. A perfect probabilistic forecast in a reliability diagram is denoted as one that lines along the y = x line. Over predictions lie to the right of the line while under predictions lie to the left. The Brier Score (Brier, 1950) is often used for quantitative verification. Upon careful consideration, it has been determined that the Brier Score should not be used for the TC wind probabilistic verification because it punishes forecasts that do not indicate a 0 or 100 percent chance of the event occurring. For a set of 10 forecasts that indicate a 50 percent chance of an event occurring, the Brier Score would be .25 regardless as to the number of verifying cases. Therefore, an alternative technique is required. An alternative technique that would remedy this problem is the mean absolute error. It is defined as

MAE = (∑n|p_i – o_i|) / N

where P_i is the predicted probability, O_i is the observed occurrence rate for the given predicted probability, N is the overall number of grid points, and n is the number of grid points for a predicted probability. Thus, in the case of 10 forecasts that indicate a 50 percent of a certain wind speed threshold occurring 5 times, the MAE would be 0, providing a more accurate quantification of the forecast skill than does the Brier Score. The viability of this technique will be demonstrated by using ImpactWeather TC wind speed probabilities from 2006-2013 for all storms where there are at least 5 consecutive days with H-WIND data. Preliminary results indicate that this technique is able to assess the accuracy of probabilistic TC wind speed forecasts. The full results will be presented at the conference.