Icechunk: An Open-Source, Transactional, Cloud-Native Storage Engine for Massive-Scale Array Data

Our community is still struggling to figure out how to best take advantage of the cloud to leverage our massive volume of environmental data. Many of the questions revolve around file formats. Agency-led efforts such as the NASA Earthdata Cloud and the NOAA Open Data Dissemination Program have used a “lift and shift” approach to migrate existing archival files (mostly NetCDF and GRIB) to cloud object storage, while data analysis platforms such as Google Earth Engine, Microsoft Planetary Computer, and Pangeo have emphasized a transformation of data to analysis-ready, cloud-optimized (ARCO) formats such as Zarr and Cloud-Optimized Geotiff. Both approaches have significant drawbacks. More recently, the Kerchunk project has suggested that its possible to unify these two approaches through virtualization, exposing archival formats via a more performant Zarr interface. But questions remain about how to operationalize this technology.

Within this landscape, we are excited to announce the release of Icechunk—a new open-source library and specification for the storage of multidimensional array (a.k.a. tensor) data in cloud object storage which resolves many of the challenges facing our community today. Inspired in part by the popular Apache Iceberg table format, Icechunk functions as a universal storage engine for Zarr, supporting both “native” Zarr data as well as virtual Zarr-compatible data stored in archival files. Additionally, Icechunk introduces database-style transactions, enabling safe, consistent updates to large datasets in an operational context. The core requirements behind Icechunk are as follows:

1. Object storage only - The full state of an Icechunk store is contained in cloud object storage, without the need for an external database.
2. Serializable isolation - Reads are isolated from concurrent writes and always use a committed snapshot. Writes are committed via a single atomic operation and are never partially visible. Readers do not acquire locks.
3. Chunk sharding and references - Chunk storage is decoupled from specific file names. Multiple chunks can be packed into a single object (sharding). Zarr-compatible chunks within other file formats (e.g. HDF5, NetCDF, GRIB) can be referenced.
4. Time travel - Previous snapshots of a store remain accessible after new ones have been written. Reverting to an earlier snapshot is trivial and inexpensive.
5. Schema Evolution - Arrays and Groups can be added, renamed, and removed from the hierarchy with minimal overhead.

This talk will present the Icechunk specification and describe our Rust-based implementation. We will demonstrate the integration with popular Python tools such as Xarray, Dask, and VirtualiZarr. And we will showcase real-world end-to-end workflows building Icechunk stores on top of NASA and NODD cloud data, illustrating the potential of Icechunk to unlock Petabytes of existing archival data to high-performance cloud-native analytics.

We’re optimistic that Icechunk may be the missing link that finally allows our community to efficiently store and share analysis-ready weather and climate data in the cloud, bridging the gap between traditional archival file formats and newer cloud-native formats through a unified interface. We welcome community feedback and involvement in the project.