The Factors Driving the Development of Machine Learning at the UK Met Office

The traditional approach to weather forecasting of developing ever more complex dynamical models is being challenged by new data driven approaches. These two approaches solve different problems and there is likely to be significant benefit in combining them. Even with the major advances in science and technology that allows us to run global dynamical models at 10 km resolution and models for the UK at 1 km there is still an increasing need for localisation, assessment of impact and communication of risk. Data driven approaches are becoming widespread and can be very powerful tools to provide such information. However if the statistical technique is purely based on patterns in the data and statistical relationships it can lose or misrepresent physical linkages. This can result in a “black box” approach that is difficult to verify and may mask the value of the underlying forecasts. For example the “black box” approach may work well under normal circumstances but fail in unusual situations.

The Met Office has a track record in using data driven, statistical techniques combined with physical models and understanding. For example it has developed operational and planning tools for the wind energy sector combining data from our observations network, local observations from site managers, complex model simulations and reanalysis using statistical techniques in an 'intelligence function' to provide accurate wind assessments for new sites, even in regions with complex terrain. It has also used emulators in climate prediction. These emulate the behaviour of extensive complex models using statistical techniques. Multiple emulations were combined with a limited number of more complex model realisations to provide global information to UK government to support their climate policy development. More detailed UK projections were also published f in 2009 or everyone to access. They were used in the last 2 UK climate change risk assessments. Upgraded projections will be published in 2018.

The weather services sector needs to understand the strengths and weaknesses of different data driven different approaches and machine learning and whether or not they should link to physical models and understanding. This is becoming increasingly important as a wider range of suppliers are providing weather services, often with very little understanding of whether science, based on good data analytics capability. Building on its existing experience the Met office is well placed to test the value of data driven approaches for a wide range of applications.

This paper will set out the main drivers for exploring machine learning and some of the challenges that lie ahead in the context of what has already been achieved. In particular, it will ask the question about what the NMS role should be in providing quality standards for data driven approaches, or at the very least exemplars, in addition to their core activities. A related paper by Rachel Prudden will talk about some of the specific techniques being employed in weather forecasting and early results.