Extracting Value from Large Meteorological Datasets in a Cloud Delivery Environment Where Data Volume is Measured in Dollars

Background

The advent of large scale computing generates ever increasing volumes of post-processed geospatial datasets. Moving such large amounts of data across the internet is no longer feasible and alternative approaches to data sharing must be found. The adoption of cloud gives flexibility, scalability and availability and is rapidly being exploited for the generation and storage of a large scale meteorological models. With this shift, old business models are rapidly becoming unaffordable in the face of egress charging and data processing costs. With this in mind it is imperative that we move to a more flexible alternative to bulk transfer

This presentation will explore some of the findings that the Met Office has made during its exploration of this new paradigm and ask questions about the assumptions that appear to still bind us.

Data Challenges

Research by the Met Office identifies several key use cases.

1. Net importers of large-scale models which are then used for assimilation purposes to provide blended data sources
2. Net importers of large-scale models which are then used to derive consumer products
3. End users of smaller datasets that must extract these from larger datasets at point of use, in order to draw out the value
4. Data visualisers, who only need what they can see on a screen.
5. Ensemble users.

In investigations of the business requirements for these personas, several patterns start to emerge. These include;

Cases where bundled data is assimilated for blending where much of the assimilated data adds no value to an outcome because the contributing model is known to be less accurate in certain areas.

An inability to adjust spatial or temporal resolution across the same model aligned to decreasing value of data ‘further away’ from the focus business use case.

Significant challenges around the use of ensemble data.

As we start to dig into the requirement more closely, we have started to investigate the art of the possible and it is clear that much of what we are taking for granted, could be changed if there is a will to do it. This presentation looks to ask some ‘What if...’ questions around the nature of the data we move around the globe and the uses to which we put it.