Quantitative precipitation forecast verification comparison between the Global Forecast System and North American Mesoscale operational models

In order to quantify the differences in quantitative precipitation forecasts (QPF) produced by two model systems that vary significantly in horizontal resolution, the Developmental Testbed Center (DTC) performed an extensive evaluation utilizing the Global Forecast System (GFS) and the North American Mesoscale (NAM) operational models. Traditional verification metrics were computed for each model; however, such metrics may unfairly penalize higher resolution models with finer detail. Thus, more advanced spatial verification techniques were also applied in an attempt to associate QPF differences with the different horizontal scales of each model.

Accumulation periods of 3-h and 24-h were evaluated out to 84-h for model output initialized at 00 UTC daily between 18 December 2008 – 15 December 2009. For the 12-km NAM, QPF output was interpolated to both a 15-km and 60-km verification domain, while the half-degree GFS QPF output was interpolated to the 60-km domain only. The National Centers for Environmental Prediction (NCEP) Stage II precipitation analyses for the 3-h accumulation and the NCEP/Climate Prediction Center (CPC) precipitation analyses for the 24-h accumulations were also interpolated to the same 15- and 60-km verification domains, and then compared to the forecasts.

Objective model verification statistics were generated using version 3.0 of the Model Evaluation Tools (MET) software package, which offers a wide variety of state-of-the-art verification methods. For this test, grid-to-grid comparisons were performed. The traditional metrics computed included frequency bias and Equitable Threat Score (ETS), while the advanced techniques included an object-based verification approach - Method for Object-based Diagnostic Evaluation (MODE) - and a neighborhood method - Fraction Skill Score (FSS).