Application of MODE to GFS (T574) and GFS (T1534) Precipitation Verification

Forecast verification is a key step to improving model performance. Traditional verification measures offer forecast quality assessment scores, but do not provide diagnostic information about why forecast skill was high or low and penalize forecasts twice for missing observed precipitation and for giving a false alarm. Additionally, measures-based verification statistics “tend to favor smoother forecast fields of coarser-resolution models” and therefore do not provide useful information on the benefits of high resolution models. Object-based precipitation verification statistics, unlike traditional skill scores, offer spatial information on how close the precipitation forecasts are to the observations in location, size, orientation, and intensity. The Method for Object-Based Diagnostic Evaluation (MODE), developed by the National Center for Atmospheric Research (NCAR) Developmental Testbed Center (DTC), is an example of a verification tool that provides information about the spatial differences between forecast and observation objects. Within MODE, precipitation objects are defined in both forecasts and observations based on two main parameters, an accumulation threshold and a smoothing radius.

MODE precipitation verification was applied to two versions of the NCEP global numerical weather prediction model – the GFS (T574, 27km resolution) model and the GFSX (T1534, 13km resolution) model – to the 2014 seasons. The T1534 GFSX was implemented for operation on January 14, 2015 and replaced the previous operational T574 GFS. Multiple thresholds combined with a fixed smoothing radius were used for the MODE identification of objects. The 00Z forecasted 24-hour precipitation accumulations ending on the 36, 60, 84, 108, 132, 156, and 180 forecast lead times from the models were compared to Climatologically Calibrated Precipitation Analysis (CCPA) observations. The Median of the Maximum Interest with respect to observation objects (MMIO) is an output summary statistic generated for each MODE run, and is useful when aggregated across multiple cases to assess overall model performance. Larger values of MMIO indicate higher interest values which imply better matches across forecast and observation fields. For all four seasons in 2014, the GFSX generally has larger MMIO values than the GFS during most forecast lead times. In the summer for both model versions, the MMIO values are lowest compared to other seasons, and there seems to be a larger drop in MMIO values with forecast lead time. Both the GFS and GFSX tend to overestimate light rain and underestimate heavy rain, but the GFSX is frequently closer to observations compared to the GFS. The GFSX is also closest to observations in total object count, especially in summer where the GFS largely underestimates the number of precipitation objects. In winter and spring of 2014, the GFSX overestimates total object count.