Big Data and Machine Learning for Weather Forecasts: Applications to Forecasting Solar Power for Utility Operations

Applied weather prediction is an important real-time challenge and many applications rely on its accuracy. Predictions are generally post-processed using machine learning methods to blend as much of the data, model output, and statistical learning as possible to improve the deterministic forecast and to quantify the uncertainty. Additional observations, when available, can be used in the post-processing step to improve the forecast. Training these post-processing methods requires large amounts of both model and observational data. The best methods blend computational intelligence with the discretization of the physics and dynamics of the system. Such systems can be quite complex, and this is a problem in Big Data. Here we describe the Sun4Cast solar power forecasting system as an example of such a Big Data Weather system.

The system consists of several technologies to predict irradiance at differing timescales. In the short term (0-6 hrs), that involves improving various Nowcasting technologies. These technologies are based on current knowledge of the atmospheric state via observations, then use appropriate methods to predict near-term changes in that state. Each of these various approaches to Nowcasting have merit, and thus, are blended to produce a single best forecast for the end user. These technologies include: StatCast, TSICast, CIRACast, MADCast, and WRF-Solar-Now. In addition, forecasts from various numerical weather (NWP) models are blended via the Dynamic Integrated Forecast (DICast®) System. The Nowcast and DICast® systems are then used to produce the best solar irradiance forecast for industry partner solar array sites. Those irradiances are translated to power forecasts, the uncertainty is quantified using an analog ensemble (AnEn) approach, and provided to the industry partners for their real-time decision making.

The methods to make the forecast require a plethora of historical and real-time data from both models and observations, blended in real-time to provide a seamless forecast for use by decision makers. Thus, forecasting for variable renewable energy, here using solar power as an example, is a real-world Big Data problem. It includes data of large volume, velocity, variety, variability, veracity, and complexity. Each of these will be discussed in the presentation and recommendations will be made for best practices for such real-time forecasting systems.