Quantifying the effect of radar observational uncertainty on the verification of kilometer-scale NWP precipitation forecasts using the Fractions Skill Score

Through advances in computational power and improvements in numerical weather prediction (NWP) models, forecasts at kilometer-scale resolution are now becoming routine. While providing precipitation forecasts that are deemed more realistic (particularly in the intensity and spatial characteristics of convective precipitation), these high-resolution models can be penalised by traditional verification metrics for failing to exactly match the observed rainfall at the (inherently unpredictable) grid scale. For this reason, a number of new spatial verification methods have been developed to better assess the true quality of forecasts. The Fractions Skill Score (FSS) is one such metric, which assesses the fractional coverage of precipitation at a variety of spatial scales, from the grid scale to the whole domain.

When performing forecast verification, the observed values are commonly treated as "truth", and differences between the forecast and observed fields are generally attributed to model error. However, observational uncertainty can be large, particularly in complexly-derived fields such as the Quantitative Precipitation Estimates (QPE) from radar. These are typically used for computing the FSS. In this study, we make use of a new radar ensemble product that has been developed for the UK radar network, which accounts for the effects of random errors in the vertical profile of reflectivity (VPR) on the QPE derivation, yielding an ensemble of estimated rainfall rates. We use this radar ensemble in the precipitation verification of the operational Met Office UKV model, which provides deterministic forecasts for the UK at 1.5 km resolution. The range in FSS (rFSS) across the ensemble varies with the spatial scale and accumulation threshold considered, but early estimates indicate that it may be ~10% of the traditional “single radar field” score, and substantially larger at high accumulation thresholds. This evidence suggests that the (usually unaccounted for) effect of radar observational uncertainty on NWP verification metrics can be relatively large, and should be taken into account when assessing the relative performance of forecasts from different modelling systems, especially when using absolute accumulation thresholds.