Fire Weather Impact Based Performance Metric

Verification of fire weather forecasts produces a multitude of benefits. Forecast verification assists in assessing the need for model advancements and opportunities for human forecaster improvements. Verification information contributes to building trust with customers in the use of weather products and services. Finally, verification is often used as a metric to assess the value of forecasts and a benchmark to create future goals. It is for these reasons that the National Weather Services (NWS) has established core verification metrics that have been computed and tracked for a variety of forecast products.

As the NWS progresses towards the initiative of a Weather Ready Nation, largely based on societal impacts, it naturally makes sense that verification methodologies would follow suit. Impacts are established by investigating the hazard and vulnerability of the person(s) or place(s) being affected. If impacts are known, then (theoretically) the requirements for weather forecasts (i.e. accuracy, lead time, etc.) could be discovered. In practice, this idea proves quite difficult to implement for a variety of reasons, the largest of these being that weather may not be the only hazard at play. Looking at fire weather, for example, weather is only part of the hazard. It is the interaction of weather with an ignition or ongoing fire, combined with a community's vulnerability, which produces an impact. The intent of an impact based fire weather verification methodology is to develop a quantitative measure of customer satisfaction using non-meteorological parameters and fire based impacts. A prototype impact based performance fire weather metric with a couple years of data for sample locations was developed to illustrate how to potentially meet this intent.

The NWS produces a next day National Fire Danger Rating System (NFDRS) forecast. These forecasts aid Land Managers in making fire management decisions on a daily basis. The National Fire Danger Rating System (NFDRS) is based on weather and non-weather variables as inputs. The NFDRS model calculates fuel moistures and generates fire danger model outputs. All fire management decisions in a Fire Danger Rating Area (an area of land management responsibility) are based on NFDRS outputs. These decisions, if documented, can be found in a Fire Danger Operating Plan. Staffing Level and Adjective Rating are commonly used by fire managers to make fire management decisions including how land management resources are to be allocated. Staffing Level is used for internal Land Management Agency resource allocation, while Adjective Rating is used to communicate fire risk to the public and limit public impacts on fire risk. Staffing Level and Adjective Rating are computed using a combination of NFDRS outputs. Based on location (i.e. Fire Danger Rating Area), the combination of NFDRS outputs used may vary. Regression analyses of NFDRS outputs and fire business and/or climatological percentiles are used to determine decision breakpoints for fire management decisions thus creating decision categories.

Categories of fire management decisions for both forecasted and actual weather variables are compared to see if forecast error would have resulted in a different decision for the customer. The numbers of “Categories of Departure” between forecast and observed decision categories provide the “Error Value." The lower the “Error Value” the more successful the fire weather forecast impact on fire operations. This process is repeated using persistence data in place of forecast data to create a Persistence Error to compare to the Forecast Error as a frame of reference. For this prototype, a Persistence Forecast is defined as the actual observed conditions for the previous day. Persistence is used by fire managers if/when an NWS forecast is not available. Depending on resource needs that are associated with different levels of fire management decisions, a monetary estimation can be assigned to the Error Value. This enables the verification metric to quantify the value of fire weather forecast based on customer impacts.

Based on the Northern Utah Fire Danger Operating Plan and the Great Basin wildland fire incident cost calculator, an average cost of $5,000 to the Land Management Agencies can be assessed for each “Category of Departure” in the “Error Value.” In the prototype, the Mean Absolute Error (MAE) for NWS NFDRS Forecasts of Staffing Level was .10. Therefore, 1 out of every 10 days, NWS based Staffing Level predictions were 1 decision category off (Decision Error). If we use an average of $5k per Decision Error, then once every 10 days, NWS forecast error cost the Land Management Agencies $5,000. This equals approximately $182,500 per year. Persistence based Staffing Level Decision Error would result in an annual error cost of $273,750. Therefore, we can conclude that NWS NFDRS forecasts could save Northern Utah $91,000 per year for fire management Staffing Level decisions.

Through this illustration of an impact based performance metric of this fire weather prototype, we are able to investigate the strengths and limitations of such a metric. Further discussion may lead to universal methodologies that could be adjusted and applied to different weather hazards and user groups. There are plans for this fire weather prototype to be run in a real-time analysis during the 2015 summer fire season.