Are the Discrimination Ability and Maximum Economic Value of a Forecast System Affected by Forecast Biases?

Most forecast systems possess systematic biases because of the crude representation of model physics and dynamics to the real atmosphere. Focusing on short-range probabilistic quantitative precipitation forecasts (PQPFs) for typhoons, this study explores the effect of calibration on the discrimination ability and maximum economic value (EV_max) based on the Local Analysis and Prediction System (LAPS) ensemble prediction system (EPS) operated at Central Weather Bureau in Taiwan.

Results show that the discrimination ability, which can be assessed by relative operating characteristic (ROC), and EV_max of an EPS are insensitive to forecast biases. This implies that improving reliability via calibration cannot increase the discrimination and EV_max of a forecast system. However, the optimal probability threshold (P_t) for users to take preventive action and obtain their EV_max is different when adopting calibrated or uncalibrated forecasts. In other words, biased forecasts will not prevent users from obtaining their EV_max if an appropriate P_t is adopted. When uncalibrated forecasts are adopted, the optimal P_t should be determined based on the past long-term statistics of EV.

Experiments have been conducted to verify that ROC is insensitive to the linear property of a calibration method or even the accuracy of the calibration results. That is, the discrimination ability almost remains the same after a linear or nonlinear calibration, even though forecast biases cannot be properly corrected during the calibration process. This is because calibration only corrects the precipitation amount instead of modifying the rainfall pattern, which is controlled by the model physics/dynamics processes and is associated more with the discrimination of the forecasts.