Digital Data Services at the UK Met Office: From Open Data to Channel Partners

The Met Office produce ~400TBs of data daily, ranging from observations to predictions/projections, and from scientific research to operational services. We are developing a series of data services which will allow new users to get value from our data in new ways. These services have emerged from an objective analysis of the outcomes we prioritise, the market and user-stories, before we develop and test prototypes.

New services include:

• Provision of “no-frills” open data, which decreases the cost to consumer/producer and decreases the time to market. For example, weather forecast data on Amazon Sustainability Data Initiative.
• Provision of data through third party marketplaces, which aims to make our data readily available where consumers work. For example, data sales on Snowflake.
• Provision of data through platforms which facilitate consumers to use it more effectively. For example, the Met Office Climate Data Portal.

Note that these are not mutually exclusive.

The Met Office is measured on creation of socio-economic benefit (SEB). For instance, SEB realisation is used directly to justify the investment in our new supercomputing facility. Moreover, we are obliged to facilitate the use of some of our data due to various legislation or as conditions of funding. Finally, as a “Trading Fund”, the Met Office can sell products to generate revenue.

In recent years, our major data service has been Weather Data Hub. This is an advanced data web service and API, allowing users to relatively easily configure API queries which subset and deliver operational data. It allows us to meet our legislative requirements around data availability, while generating revenue and making our data more useable. This bespoke service is developed and run by the Met Office and subcontractors.

However, as customers have changed, Weather Data Hub has some downsides:

1. The extra functionality offered by Weather Data Hub requires complex engineering. This has become expensive to maintain, undermining Return on Investment. Moreover, it has increased time to market for new datasets.
2. It assumes that potential consumers will seek out the Met Office to consume data.
3. It assumes that consumers will use an Extract, Transform, Load (ETL) approach i.e. that they want to “download” data.
4. Finally, it conflates the need to meet legal data availability obligations, with the sales of data, the sales of services to make use of data, and making desired outcomes unclear.

The market has changed, offering new opportunities:

• Many power-users of Met Office data do not want any added value services at all, for instance, weather enterprise companies. They would rather consume raw data, cheaper and faster, which they ingest into their own data stack.
• Decision making in many industries is increasingly data driven. There has emerged a platform and middleware economy to enable this. For example, an analyst at a retail company analysing supply chain distribution is increasingly unlikely to consume from the Met Office directly. Instead, they log into their third-party data platform, and simply search for “weather” in the catalogue of pre-prepared data they can purchase.
• There are a range of platforms which specialise in offering advanced functionality such as generic AI, big data processing, and dashboard creation. These are things the Met Office cannot (and should not) offer. However, they allow users to add tremendous value. For instance, Esri is a market leader in no-code/low-code geospatial analytics. In the UK, 21 of 23 government departments use Esri, and the Ministry of Defence is the biggest Esri customer in the UK – these are all fundamental Met Office customers.

In general, integration of Met Office data into these platforms makes it easy to create value. At its most simple, it facilitates consumers to self-serve. However, it offers some interesting new opportunities beyond that.

1. The Met Office can use these services internally.
2. Met Office experts create digitized intellectual property which we can offer via platforms. E.g. we can sell access to algorithms to process data or sell a template dashboard design.
3. The Met Office can work in collaboration with end-users to create services. Crucially, the end-user is then responsible for maintaining (and funding) that service, reducing the burden on the Met Office.

These new types of data service are set to be a pivotal part of the Met Office in the future to generate both socio-economic benefit and revenue.