zarrista

A fast, low-level Zarr API for Python, powered from Rust by Zarrs.

Very early benchmarking suggests potential 17-35x faster read throughput than Zarr-Python.

While Zarrista will exist as a standalone Python library and can be used directly, the goal is to integrate Zarrista directly into Zarr-Python so that existing users can get improved performance out of the box and to avoid fracturing the ecosystem.

This library is beta-quality. The underlying Zarrs library is reliable and broadly used. The way we expose a Python API from it may change in the future.

Documentation

Documentation website.

Features

• High-performance Rust core
  • Encoded chunk access allows explicitly managing IO-bound data access and CPU-bound decoding separately.
• Sync/Async support
  • Synchronous Array and Group for local file system access
  • Asynchronous counterparts for remote data access through Obstore
• NumPy integration
• Zero-copy data exchange via DLPack, the buffer protocol, and Arrow.
• Broad data type support including variable-length string/bytes and the machine-learning float and sub-byte integer types (integrating with ml_dtypes).
• Broad codec support, including all in the Zarr v3 spec.
• Full type hinting for all operations.
• Icechunk integration

Example

Open a store, then open an Array from it:

from zarrista import Array
from zarrista.store import FilesystemStore

store = FilesystemStore("data/example.zarr")
array = Array.open(store, path="/temperature")

Inspect the array's metadata:

array.shape
# [720, 1440]

array.dtype
# DataType(float32 / <f4)

array.dimension_names
# ["lat", "lon"]

Read a subset of the array. Indexing returns a Tensor, which converts to a NumPy array:

data = array[0:128, 0:128]
arr = data.to_numpy()
arr.shape
# (128, 128)

You can also read individual chunks by their grid index:

data = array.retrieve_chunk([0, 0])

AI Policy

This project is minimally vibe-coded.

Most of the library code was written by hand by @kylebarron, sometimes in conversation with Claude.

Documentation and type stubs are mixed. Much documentation is written by hand but Python type stubs are partially kept up to date via Claude.

Almost all current tests were written by Claude.