Understanding Forecast Verification from a Design of Experiments Perspective

In many situations, including those wherein one is verifying forecasts, there exists a desire to understand how various inputs to that forecast influence the forecast itself. One method partitions a metric, say model bias, into classes which include controllable and uncontrollable factors. Uncontrollable factors include the location where the model is forecasting as well as its initial conditions. Controllable factors include such things as model parameterizations and configurations. We use design of experiments to create an arrangement of model runs that allow us to separate controllable from uncontrolled effects. When coupled with other data, this allows an analyst to explore model errors from a variety of perspectives that may illuminate the factors which contribute to the bias and how to improve the forecast.