Choosing a scoring rule for verification of forecast probability distributions: Continuous ranked probability score or ignorance score?
Insight into this question is gained by considering how CRPS and IGN score imperfect forecasts relative to each other and relative to the perfect forecast (true distribution). Under the assumptions of a Gaussian forecast distribution and a standard Gaussian true distribution, expected CRPS and expected IGN are plotted as a function of forecast mean and variance. For values of the forecast variance greater than the true variance, the expected CRPS and expected IGN fields are qualitatively similar, despite the radically different functional forms of CRPS and IGN. However, for values of the forecast variance less than the true variance, the expected score fields differ substantially. Relative to CRPS, IGN is expected to assign a very harsh penalty to a forecast with an erroneously low variance. Because of this property, it is argued that IGN verification is best-suited for probabilistic prediction applications in which it is of paramount importance to avoid underprediction of the true variance.