Adding machine-learned uncertainty quantification to the National Blend of Models

The National Blend of Models (NBM) is a nationally consistent and skillful suite of calibrated forecast guidance based on a blend of both NWS and non-NWS numerical weather prediction (NWP) model data and post-processed model guidance. Target variables include near-surface sensible weather conditions, such as 2-m temperature, 2-m dewpoint, 10-m zonal wind, 10-m meridional wind, and 10-m gust speed. We are developing machine-learned uncertainty quantification (ML-UQ) tools to transition deterministic forecasts of these five variables – which are crucial for defining fire-weather conditions – into calibrated, probabilistic forecasts of the same. Our model evaluation and workflow emphasizes performance on extreme fire weather (e.g., forecast skill for extreme hot/dry/windy conditions), while ensuring that performance in a global sense (across all situations, including, e.g., winter weather) does not deteriorate.

Our specific deep-learning architecture is a U-net++ (Zhou et al. 2019) with time-series components (Chiu et al. 2020), allowing for the optimal use of “video” data (time series of spatial grids). The U-net++ model’s output is a prediction of the five target fields at the 48-hour lead time; in a future version we will predict all NBM lead times, out to 384 hours. The predictors for the U-net++ include data from four sources. The first source is NWP forecasts of multiple weather variables at multiple lead times – including both the five target variables and auxiliary variables such as precipitation, MSL pressure, and upper-level fields between the surface and 500 hPa. We use 10 different NWP models here: the HRRR, WRF-ARW, NAM Nest, gridded GFS MOS, gridded LAMP, RAP, NAM, GFS, GEFS, and ECMWF. The second predictor source is lagged truth fields – i.e., the five target fields at several recent times in the past, according to the Unrestricted Real-time Mesoscale Analysis (URMA), which we take as truth. The third source is lagged model biases – i.e., biases of the 10 NWP models at several recent times in the past, where bias is computed as forecast minus URMA. The fourth and last source is time-constant fields, e.g., orographic height and the land/sea mask.

To meet the needs of this meteorological application, we modify the U-net++ architecture of Chiu et al. in five ways. First, we use a patchwise training approach, as in Lagerquist et al. (2021). The NBM grid is quite large (2345 x 1597), and full-grid training leads to memory crashes; thus, we train on patches of size 432 x 432. To produce full-grid predictions, we stitch together patchwise predictions from all locations within the NBM grid. Second, we use memory-efficient fusion of data from multiple resolutions. The 10 NWP models used have resolutions varying from 2.5 km to 0.5 deg (55.6 km meridionally), and the naïve upsampling approach (interpolate all data to 2.5-km resolution) leads to extra memory use and makes training cumbersome. Instead, we interpolate each NWP model to a 2.5-km version (every grid point), 10-km version (every 4th grid point), 20-km version (every 8th grid point), or 40-km version (every 16th grid point) of the NBM. The target grid is chosen to produce as little upsampling as possible – e.g., GEFS data (0.5-deg native resolution) are interpolated to the 40-km NBM grid; GFS data (0.25-deg native resolution) are interpolated to the 20-km NBM grid; and HRRR data (3-km native resolution) are interpolated to the 2.5-km NBM grid. We then introduce data from each resolution at the appropriate level in the U-net++ architecture. For example, 2.5-km data are introduced right away, but 10-km data are introduced only after the first two pooling layers (which downsample the original 2.5-km data to 5- and then 10-km resolution). Third, we use residual prediction, where the U-net++ is tasked only with predicting the departure from the raw NWP ensemble (RNE) forecast to the URMA truth, where the RNE is a simple average of the 10 NWP models. Fourth, our loss function is the dual-weighted mean squared error (DWMSE) skill score, where the “dual-weighted” part emphasizes extreme values and the baseline for the skill score is the RNE forecast valid at the same time. Hence, our loss function emphasizes improvement over the RNE baseline while emphasizing extreme values pertinent to fire weather (high temperature, low dewpoint, high wind speed). The loss function also emphasizes the opposite thermodynamic extremes (low temperature and high dewpoint), keeping in mind the goal of improving NBM performance across all situations and not just for fire weather. Fifth, our ML-UQ method is a hybrid between the multi-model ensemble (MME) and continuous ranked probability score (CRPS) approaches, allowing us to capture the two components of uncertainty: epistemic and aleatory. The MME approach is achieved by training multiple U-net++ models with different random seeds; the CRPS approach is achieved by training each model to produce its own ensemble, constrained with a loss function that combines the CRPS and DWMSE skill score, the former encouraging good ensemble calibration.

In addition to our application-specific modifications of deep-learning tools, we will focus on the performance of our U-net++ model – using both the RNE and operational NBM as baselines – and the importance of different data sources. Specifically, we will show how model performance changes when lagged truth fields, lagged model biases, time-constant fields, certain NWP models, or certain weather variables (e.g., upper-level information) are removed from the U-net++ predictors.

Chiu, H., E. Adeli, and J. Niebles, 2020: “Segmenting the future.” IEEE Robotics and Automation Letters, 5 (3), 4202–4209, URL https://doi.org/10.1109/LRA.2020.2992184.

Lagerquist, R., J. Stewart, I. Ebert-Uphoff, and C. Kumler, 2021: "Using deep learning to nowcast the spatial coverage of convection from Himawari-8 satellite data." Monthly Weather Review, 149 (12), 3897-3921, URL https://doi.org/10.1175/MWR-D-21-0096.1.

Zhou, Z., M. Siddiquee, N. Tajbakhsh, and J. Liang, 2019: “Unet++: Redesigning skip connections to exploit multiscale features in image segmentation.” IEEE Transactions on Medical Imaging, 39 (6), 1856–1867, URL https://doi.org/10.1109/TMI.2019.2959609.