Weather Forecast Improvements at the UKMO: Responding to the Big Data Challenge

Weather services have always been at the forefront of handling big data. The UK Met Office faces a new and increasing challenge because of a step change in the high performance computing it has available to generate weather forecasts and climate prediction, thereby introducing a step change in the volumes of data that we produce. In particular forecasts will be run much more frequently and with multiple realisations allowing more continuous automated update of information and a risk-based approach to services. At the same time open data and big data initiatives are encouraging us to make all our data more accessible to users and other weather service providers. In order to maximise the socio-economic benefits of our observational and forecast data we intend to actively encourage innovation with our data both by our own staff by those developing external applications for users who are sensitive to the weather. The challenge is to make data available in a cost-effective meaningful way that balances compliance, cost recovery and reach. Users need to be able to extract the data they need whilst minimising the volume of extraneous data they don’t need.

 By 2017 the Met Office expects to produce 200 – 250TB per day. In order to make our Met Office data more useful, we are undertaking a ‘Big Data Drive’ that will ultimately transform how our data is being accessed, disseminated and stored. A number of major, planned changes are being researched, tested and implemented to ensure that we take the required steps towards our legal obligations and realise the opportunities that our data provides for the whole of the Met Office.  A particular challenge still to be addressed is how users will benefit from the provision of ensemble forecasts.

 The Met Office is creating a suite of API s as a “Service Hub”, which will provide access to a broad range of data and make it widely interoperable. This is a considerable task and there are a number of steps along the way that are needed to achieve this aim:

 Decoupler Project, whereby our data will be ‘detached’ from our weather and climate models.  This will allow more frequent increases in the resolution model and changes in grid structure without disrupting data based services. It will also help to manage the volumes of data being processed. This in turn will ensure that improved data and greater choice of data is available for dissemination to our customers.

API development and data migration to provide new data capability and tools which will make Met Office data easier and faster to use and is likely to significantly grow the volumes of data transmitted to customers.

The current Met Office portal for accessing data, DataPoint http://www.metoffice.gov.uk/datapoint is being developed further to provide a single user interface to access all Met Office Open and Managed public data sets. APIs will allow users to benefit from the richness of large datasets without actually having to move the whole dataset around. However, sometimes the data provided through APIs is not enough so we are also looking at the possibilities for bringing ‘problems to the data’ by providing a computational environment and in-transit encryption to protect IP of users. Current research and development is centred around encrypted container technologies along with some block-chain thinking so that users will be able to bring their own proprietary software onto Met Office systems to manipulate and process data without compromising their own intellectual property or the integrity of Met Office systems.

 The Met Office is actively encouraging open innovation with its data within and outside the organisation by introducing an Informatics Lab. This is focused on research, prototyping and collaborating on new types of data and demonstration projects http://www.informaticslab.co.uk/ .