Enabling an Iterative Open Science Transformation to the Geoverse with a Federated Knowledge Mesh

Recent and rapid accelerations in both conceptual and technological advancements in the complimentary spaces of computational infrastructure (e.g. scalable object storage, server-less architecture components, cloud native file formats, machine learning), and data management (e.g. FAIR, CARE, TRUST, Open Science) have resulted in a superimposed state that has excited new interest in use of the now scalable semantic web as a means of capturing, defining, fusing, and delivering data in vastly more connected, valuable, and growth oriented ways. This interest is evident, in part, by the large number of fast maturing interagency and interdisciplinary digital twins pilots; large science agency movements toward ontological driven democratized governance; and integrated knowledge network efforts for implementing federated communications, exemplified by the National Science Foundation's Open Knowledge Network.

The July 2022 UN-GGIM Geoverse discussion paper describes the disruptive nature of the present condition as a macro evolutionary opportunity for orchestrating a step change to a largely automated and fully integrated virtual reality, providing guidance in the form of a globally-integrated holistic vision and classified leveling framework that employs a bootstrapping phase to step into a future characterized by vast interoperable machine to machine communications, international use driven federated digital twins, participatory domain vocabularies and standards, intrinsic data sovereignty and trust, 'credit where credit is due' cost models, and a steady state target of explicitly democratized wisdom, ostensibly for the purpose of general earth systems improvement. This vision plainly describes the exciting opportunities and clear measures that organizations may take to empower the next generation of scientists and technologists in improving the value of their own data and information, while simultaneously satisfying internal and external requirements.

Within this framing, we will provide conceptual grounding, implementation detail, and actual examples to describe an interoperable knowledge mesh developed and implemented at NOAA's National Centers for Environmental Information (NCEI), which is the officially designated repository responsible for archiving and ensuring democratic access to holistic NOAA data. The framework underlying the knowledge mesh, dubbed the 'virtual Archival Information Package’ (vAIP), was designed to foster iterative integration across the extraordinarily diverse domains at NOAA by enabling relatively fast baseline integration of legacy systems and their subsequent iterations toward full ‘5 star linked open data’, while automating aspects of critical requirements related to NARA, FAIR, CARE, and TRUST.

The vAIP framework exposes an object oriented API which provides full control and access to a process-oriented, standards based, cloud native knowledge graph and associated processing engine. The vAIP knowledge graph holds interoperable definitions and records of all integrated models, datasets, products, and access points. Employing a two-tier common reference model approach using RDF encodings of the OAIS (Open Archival Information System) Information Package and Information Object constructs, any user can leverage the vAIP API to self-define patterns, contextualize classified-task processes that leverage their own processing methods, and produce fully and automatically contextualized records in an event-driven manner.

Every definition and record held by the vAIP is stored as a fully denormalized and fully interoperable JSON-LD (JavaScript Object Notation for Linked Data) file in a dynamic hierarchy for maximal concurrency. These system features work together to enable open ended and equitable access patterns such as static web crawlers and Large Language Models, as well as combination and fusion of datasets to foster emergent multi-owner earth system digital twins and NOAA-wide foundation models.