Big Data is Critical for the Weather Ready Nation

The driver for the Weather Ready Nation is the ability to get the right information, to the right person, at the right time, in a form that can be easily understood and acted upon. We are quite good at creating information, but that doesn't translate to being good at adjusting what we do so that our data can easily be integrated with others - to tell the precise story the decision maker needs to hear. The recent technology revolution of the Big Data paradigm shift has a direct bearing on our ability to translate massive amounts of data from provider-centric creation to consumer-centric usage.

Big Data is the hot buzzword of the moment, and typical of all buzzwords, there are claims that it will magically solve all problems. Big Data is not the magic it is often described as, but there is a fundamental truth beyond the hype. Big Data represents a true phase change in data system capabilities that has opened up previously impossible or economically impractical capabilities. Twenty years ago the large-scale simulation community went through a transformation to scale out to massively parallel processors, rather than relying on continued scale-up to faster processors. Big Data represents this same transformation for the data processing community - horizontal scaling that fundamentally changes data access and analysis capabilities, and the economics of these capabilities. From a data consumer's point of view, the ability to quickly access just the data that is needed, when it is needed, with the ability to easily browse to other data related to the phenomenon and location of interest will be a game-changer.

So how will Big Data enable the Weather Ready Nation? Big Data will transform the weather enterprise by lowering the friction on the data. Friction is the appropriate term to describe the difficulty of getting data from where it is to where it is needed. Our current processes are optimized for those who produce data, but there are no additional optimization processes for facilitating the consumption of the data. We disseminate data to experts who know where the sites are that provide regularly updated zip files of data, what the origins are of that data, what the underlying science is that drove the production of the data, how the data was processed by others before they got it, how to parse and interpret the contents of the data files, and how to transform and integrate the data so they can then filter for what they need. This is a complicated process that prevents non-experts from deriving value from critical forecast and observation data.

We have built a data platform that lowers the friction for disseminating data with a consumer-oriented focus. This talk will describe the core Big Data technology that will drive the Weather Ready Nation, our data-as-a-service platform and how it will lower the friction, and our approach to make the provenance of the data accessible to non-experts.