Optimizing and Observing Model Data at Scale

Through our collaboration with NOAA IOOS on the project, “Reaching for the Cloud: Architecting a Cloud-Native Service-Based Ecosystem for DMAC”, we produced several prototype demonstrations of effective methods for interacting with cloud-based data repositories. This work has given us a clearer direction toward where we should concentrate development efforts toward stable, reliable, community-supported tools for serving environmental data. Ongoing work in the Pangeo community, which our project is aligned with, is also helping to drive design decisions that will ultimately support better access to cloud-based environmental data.

Now that we have experimented with these prototypes in live environments, we have noted several improvements to support overall functionality and usability for long-term success. These key areas are observability, scalability, and formalizing development processes.

Observability into infrastructure, data workflows, and data state itself will be needed to provide status updates and inform operators of workflow operations. Infrastructure monitoring helps ensure that the system is operating normally and using adequate resources while controlling costs. Workflow monitoring supports the data itself, providing domain-specific diagnostics such as potentially invalid or missing data at certain timestamps. This helps ensure that the data being produced is valid and alerts data product owners of any errors occurring in their data pipelines. It also gives general users a way to check what products are available and how new the data is. Finally, recording data state at a granular level simplifies reporting and diagnostics about the data repository. This also helps move toward further read optimization by providing an index to quickly scan, which is necessary to support multi-year data aggregations especially as used in machine learning applications.

Our cloud optimization workflows quickly grew from a collection of standard forecast models to an entire suite of models of all different shapes and sizes. Observability is needed to make sense of everything going on, and scalable design is also needed to reduce recurring computation costs and systematically organize the code to support common functionality across many dataset configurations. We are engineering scalability by controlling the code execution to reduce unnecessary computations (and therefore cost), using shared caches, and executing on scalable Lambda architecture.

Finally, we acknowledge that our success is dependent on the whole open-source community’s success. As a member of the open-source community and to extend this work further, we aim to improve our project stewardship of those repositories by improving documentation, adding automated testing and CI/CD, and regularly deploying versioned releases.