NOAA Local 3-Month Temperature Outlook Performance Evaluation

NOAA's National Weather Service (NWS) released the operational Local 3-Month Temperature Outlook (L3MTO) in January 2007 providing users with 3 month outlooks at the local level. Local climate forecast has been identified as a critical need by decision makers whose lives and livelihoods are sensitive to seasonal and inter-annual climate variation. In addition, this product is enhanced with online presentation features intended to mitigate “barriers” for climate forecast uses identified by Hartmann, et al, in 2002. The web functionality of the L3MTO offers multiple presentation formats for different user groups, interactive product interpretations, and long-term forecast evaluations.

Murphy (1993) introduces three measures for outlook performance: consistency, quality, and value. L3MTO performance evaluation utilizes all three measures. To ensure outlook consistency, the L3MTO uses the National Climatic Data Center (NCDC) homogenized serially completed data set. This dataset uses a linear regression technique with trend adjustment and a two-tiered quality control process. L3MTO quality, or correspondence between forecasts and observations, is measured using three verification skill scores: Heidke Skill Score (HSS), Continuous Rank Probability Skill Score (CSPSS), and Reliability diagrams. Each score assesses different, specific characteristics of the product's formats. L3MTO value or user appreciation is measured using a 2009 NWS customer satisfaction survey conducted by the Claes Fornell International (CFI) group using the American Customer Satisfaction Index (ACSI), an economic indicator based on modeling customer evaluations of the quality of products and services.

L3MTO's 2009 verification techniques using 1994-2008 re-forecast data demonstrated statistically significant improvement of forecast skill over the climatological reference period (1971-2000) for 60 to 90 percent of stations. The verification techniques used were the HSS and CRPSS. Winter and summer seasons indicate the greatest skill overall, peaking November-December-January through January-February-March (nearly 90% or greater number of sites passing all score thresholds). Scores decline, however, remaining skillful, during spring and fall (over 60% sites passing all score thresholds). Geographically, the four-corner states (UT, CO, AZ, and NM) continually prove skillful across all score analyses. The areas with the poorest forecast performance are the great lakes and central regions of the U.S. The eastern seaboard (north of Georgia) proves skillful primarily during winter and summer months. The Florida peninsula skill scores appear to peak in summer months. Conversely, Texas is skillful year-round with the exception of a slight reduction in skill during summer months. Reliability diagrams provide more information about long-term performance verification for each site and forecast 3-month period than a single score.

Overall, the L3MTO ACSI is 75, which is above the 2008 Federal Government ASCI of 68.9 and the 2009 NOAA Climate Services (including L3MTO) ASCI of 70. Survey results prove that the general public is more satisfied than the technical users of the L3MTO. L3MTO navigation (ASCI of 77) scores higher than the products functionality (ASCI of 74) and comprehension (74). The three different formats of L3MTO score high: pie chart (ASCI of 76), temperature range (ASCI of 76), and probability of exceedance (ASCI of 77).