Pivoting Towards the Future of Open Science as a Data Repository

There has been a growing desire among climate data users for the ability to access high throughput cloud optimized data as well as having the expectation for analysis-ready curation (ARCO). To satisfy this demand, the role of the data repository has needed to expand its functionality to accommodate these needs while maintaining existing workflows for its users..However, persistent questions remain about how best to optimize data and workflows to broadly reach both existing and new communities.

This talk will focus on how the Research Data Archive (RDA), hosted at the National Center for Atmospheric Research (NCAR), has been exploring and addressing issues related to the adoption of ARCO within our existing repositories. We will discuss recent work comparing different chunking strategies when creating a dataset and how this can affect local versus remote access to data. Additionally, we will consider whether there is an optimal chunking scheme or if we can instead find acceptable access times for a variety of workflows.

The RDA has a significant number of users that work adjacent to the data, so tests were performed to determine if ARCO formats have any utility when accessing over POSIX. Similarly, we have also surveyed when it is appropriate to create virtual vs real Zarr stores for existing datasets and analyze the tradeoffs and challenges between the formats for both remote and local access. Finally, we will present the effort of leveraging the Open Science Data Federation (OSDF) to optimize data transfers over HTTP to worldwide users.