Lies, Damn Lies, and an Unethical Measure of Renewable Energy Predictability: The Case of Wind Energy Forecasting in Ireland

Wind energy has significant potential to mitigate climate change by eliminating society's dependence on fossil fuels. One recognized challenge with its use in the power grid is its meteorologically induced high degree of variability coupled with the grid's relatively stringent stability conditions in which system frequency, i.e., the level of voltage and current oscillations each second, is only in its normal range when electricity supply is well matched with demand. Supply being less than demand is particularly worrisome because if system frequency falls substantially, the entire high-voltage electricity system can crash, which can entail high societal costs. For this reason, wind energy predictability is recognized as essential to successfully integrating wind energy into the power grid.

Unfortunately, in reporting the summary measure of the forecast error as a percent, the wind energy forecast community has rejected an approach endorsed by the statistics community in which the mean level of generation weights the mean absolute error (MAE). Instead, it is strongly preferred that the wind energy forecast error be measured by dividing the MAE by the capacity of the wind energy turbines. System operators such as EirGrid in Ireland have accepted this recommendation.

This paper presents evidence that the capacity-weighted error approach can create an illusion of predictability that encourages complacency. This approach has been informally defended (there is no peer-reviewed statistical analysis to support this approach) based on the notion that all forecast error metrics are somehow “wrong” and that using a capacity-weighted error metric somehow resolves this alleged deficiency. With all due respect, the view here is that this justification is highly unethical because it bases the calculation method on the goal of reporting a “reasonable” error value instead of using a method endorsed by the statistics community and working dawn to dusk to improve accuracy if the reported error is too high. In short, the renewable energy forecast community’s strong preference for an error metric weighted by capacity appears to be based on the unethical desire to report that their predictions are accurate even when there is clear evidence that this is not the case.

The so-called capacity-weighted measure of Ireland's wind energy forecast error from Jan 1, 2019, through Dec 31, 2023, is a seemingly modest 3.5 % (Figure 1). Consistent with the large deviations away from the 45-degree line in Figure 1, forecasting skill does not drive this value. Instead, it is driven by the fact that the capacity-weighted error (CWMAE) is related to the capacity factor (CF), which measures the average generation level relative to potential generation. There is no rational basis for this stipulated relationship. Nevertheless, this stipulation can be seen by noting that basic algebra indicates that CWMAE = CF*WMAPE, where WMAPE is the weighted mean percentage error, an error metric accepted by the statistics community. In this case, the capacity factor is about 28.5 % while the WMAPE is about 12.3 %, and thus, it is no surprise that the CWMAE is only 3.5% (3.5 ~ 0.285*12.3). Given the CWMAE's indefensible dependence on the capacity factor, it is not a surprise that the statistics community has not endorsed the methodology. It should also be noted that the forecast skill score equals -27.9, which indicates that the forecast is vastly inferior to a persistence forecast that myopically presumes that the predicted outcome in period t equals the actual outcome in period t-1. Given these facts, claiming that the 3.5% CWMAE is evidence of forecasting excellence is unethical.

With the hope of improving upon this dismal situation, this paper presents a statistical methodology to improve the predictability of wind energy generation using data from the Irish power grid. The analysis begins by observing that the existing wind energy forecasts do not fully reflect expected meteorological conditions, a likely ominous result given that rising CO2 concentrations can be expected to make meteorological conditions more challenging. It is further noted that wind energy generation is highly volatile at times but also has a significant autoregressive pattern that can be exploited to improve predictability. An ARCH/ARMAX(autoregressive conditional heteroskedasticity/ autoregressive–moving-average with exogenous inputs) time series model is formulated based on these properties. One of the key modeling innovations is the implementation of ARCH-in-mean effects, which boosts predictive accuracy by capturing the information in the conditional variance. The model is estimated using 15-minute data from Jan 1, 2019, through Dec 31, 2022. The model is evaluated using out-of-sample data from Jan 1, 2023, to Dec 31, 2023. The period-ahead out-of-sample predictions have a WMAPE of about 5 %, substantially less than the approximately 11.5 % WMAPE associated with the wind energy forecasts used by the system operator over the same period. While this finding is unlikely to induce the apologists of capacity weighting to recant their unethical reporting approach, it may inform decision-makers at the American Meteorological Society and other organizations with a desire for a clean energy future with a stable power grid that the so-called capacity-weighted error metric strongly advocated by the renewable energy forecasters cannot be trusted.