Obviating the Challenges of Data Assimilation with Direct Observation Prediction

Machine Learned (ML) weather forecasts continue to challenge traditional approaches to numerical weather prediction (NWP), demonstrating competitive performance compared to physics-based models. Many of these data-driven systems have been trained to forecast future weather by learning from long historical records of past weather, typically provided by reanalyses such as ECMWF’s ERA5. These datasets have been made freely available to the wider research community, including the commercial sector, which has been a major factor in the rise of ML forecast systems and the levels of accuracy they have achieved.

However, both historical reanalyses used for training and real-time analyses used for initial conditions are produced by data assimilation, essentially an optimal blending of observations with a traditional physics-based forecast model. As such, many ML forecast systems have an implicit unquantified dependence on the physics-based models they seek to challenge and, furthermore, rely on the continued future development of these traditional physics-based systems and data assimilation systems.

Here we propose a new and radical approach to weather forecasting, training a neural network to predict future weather purely from historical observations with no dependence on a physics-based model or reanalysis datasets. We use raw observations to initialise a model of the atmosphere built in observation space, this model being learned directly from the observations themselves. Forecasts of crucial weather-related parameters (such as surface and upper air temperature and wind) are obtained by predicting weather parameter observations (such as SYNOP surface data and radiosondes) at future times and arbitrary locations.

We argue that this new approach, by staying purely in observation space avoids many of the challenges of traditional data assimilation, can exploit a wider range of observations and produces forecasts of the full Earth system (atmosphere, land, ocean and atmospheric composition) without any need for complex coupling techniques.