Add public synchronous API for concurrent group/metadata operations

[aladinor]

Motivation

From the discussion in #3835, downstream libraries like xarray need to open multiple zarr groups concurrently for performance. Currently, this requires either:

1. Using zarr's internal zarr.core.sync.sync() to bridge async→sync (not public API)
2. Managing a separate event loop (risky, conflicts with zarr's internal loop)

The consensus from the discussion is that zarr should provide proper public APIs that handle concurrency internally, rather than exposing sync() to downstream consumers.

Proposed APIs

1. Public synchronous store API

Zarr currently has no public synchronous API for store classes. PR #3638 added _get_bytes_sync, _get_json_sync as private methods on the Store ABC, with the intent to make them public once the API design matures (specifically the prototype parameter question). Promoting these to public would give downstream libraries a supported way to do concurrency-backed store operations.

2. Higher-level concurrent group opening

As @d-v-b suggested in #3835, a synchronous function that opens multiple groups concurrently, async internally:

# Opens all child groups/arrays concurrently, returns sync objects
children: dict[str, zarr.Array | zarr.Group] = zarr.open_members(group)

This directly addresses the xarray/xradar open_datatree use case — fetching metadata and coordinate arrays for N groups in parallel. Internally it would use AsyncGroup.members() and the async store APIs, but callers wouldn't need to touch the event loop.

3. Concurrent data access (future)

@ilan-gold raised more advanced use cases in #3835 involving concurrent reads across multiple arrays with interleaved computation. @d-v-b's lazy indexing prototype (#3678) may address some of these patterns. This is a separate concern from (1) and (2) and can be tracked independently.

Use cases

xarray / xradar (@TomNicholas, @aladinor):

• Fetch/set metadata of arbitrary groups/arrays concurrently
• Concurrently fetch/set data for coordinate arrays
• Mostly metadata operations and reading entire arrays, not lazy indexing
• Opening hierarchical group structures (e.g., multi-sweep radar volumes) concurrently is critical for cloud performance

General downstream (@BorisTheBrave, xarray PR #11171):

• Any library wanting zarr's async performance benefits through a sync API

References

• Discussion: Stabilize zarr.core.sync.sync() for downstream async integration #3835
• Prior sync store API (private methods): Add methods for getting bytes + json to store abc #3638
• Lazy indexing prototype: feat/lazy indexing #3678
• xarray async datatree PR: Implement async support for open_datatree pydata/xarray#10742
• xarray async strategy: How should Xarray control asynchronous calls? pydata/xarray#10622
• xarray sync-via-zarr-loop PR: PoC Set variables async pydata/xarray#11171

cc @d-v-b @ilan-gold @TomNicholas @shoyer @keewis @dcherian @BorisTheBrave

---

This is my understanding of the discussion so far. If I've misrepresented anything or missed important context, please correct me. Additional ideas and suggestions are very welcome.

[d-v-b]

Thanks for this great summary!

1. Public synchronous store API

Zarr currently has no public synchronous API for store classes. PR #3638 added _get_bytes_sync, _get_json_sync as private methods on the Store ABC, with the intent to make them public once the API design matures (specifically the prototype parameter question). Promoting these to public would give downstream libraries a supported way to do concurrency-backed store operations.

I think it would be great for these useful methods to be come public, and to add even more useful synchronous methods. These would be changes that are safe to make to our existing store classes. I think it would be even better if the synchronous methods were just called <method>, while the async methods were called <method>_async or equivalent, and if the store methods had useful return types (something more useful than a Buffer). Those changes are unsafe to make on our current store classes, and would require new store classes. The timeline for the additive, safe changes is much shorter than the timeline for a full refresh of our store API.

2. Higher-level concurrent group opening

As @d-v-b suggested in #3835, a synchronous function that opens multiple groups concurrently, async internally:

Opens all child groups/arrays concurrently, returns sync objects
children: dict[str, zarr.Array | zarr.Group] = zarr.open_members(group)

This directly addresses the xarray/xradar open_datatree use case — fetching metadata and coordinate arrays for N groups in parallel. Internally it would use AsyncGroup.members() and the async store APIs, but callers wouldn't need to touch the event loop.

IMO accessing members needs to go through an existing group for consolidated metadata to work properly, so we are looking at a new function / method on the Group class. Group.members is already in use, which is unfortunate because defining a NodeMembers object and using it e.g.Group.members.open_many(), Group.members.iter_arrays(), Group.members.iter_groups() would have had a nice shape. But we can settle for something here. One thing to fix first: we need to make separate group classes for v2 and v3 hierarchies. This will simplify internal sanity checks (to ensure that v3 groups never reveal v2 arrays). I would love to review a PR that adds this distinction.

[aladinor]

Thanks for the great discussion! After implementing the async open_datatree in xarray (pydata/xarray#10742), I want to share what we actually need from zarr and why higher-level APIs alone may not be sufficient.

Two bottlenecks, not one

Our xarray PR solves two distinct concurrency bottlenecks:

Bottleneck 1: Group discovery + dataset opening

• Uses async_root.members(max_depth=None) to walk the tree in one async pass
• Opens each group's dataset concurrently via asyncio.TaskGroup
• A zarr API like open_children() could help here

Bottleneck 2: Concurrent index creation (the bigger win)

• For each group, coordinate variables must be loaded to create indexes
• Each coordinate load triggers a store.get() — doing this sequentially across N groups × M coordinates = O(N×M) round trips
• We solve this by loading all coordinates across all groups concurrently via asyncio.gather + var.load_async()
• Open to suggestions on how zarr could better support this pattern — but the index creation logic itself is inherently xarray-specific (Datasets, CF decoding, coordinate handling)

Why sync() is the fundamental piece

Both bottlenecks are solved by async code that is deeply xarray-specific — creating Datasets, building indexes, decoding CF variables. This logic can't be pushed down into zarr. What we need is a way to run xarray's async code on zarr's event loop from xarray's sync open_datatree():

# xarray's open_datatree (sync) — submits async work to zarr's loop
from zarr.core.sync import sync as zarr_sync
return zarr_sync(self._open_datatree_from_stores_async(...))

# Concurrent index creation across all groups
results = zarr_sync(create_indexes_async())

Using zarr's event loop (rather than creating our own) is critical because:

• Avoids conflicting with zarr's internal loop management
• Shares zarr's thread pool and concurrency settings
• Is Jupyter-safe (zarr already handles the nested event loop case)

Community convergence on sync()

Multiple independent efforts have converged on needing sync():

• @TomNicholas separated the two questions clearly: the ideal future async interface vs. "Can xarray rely on zarr.core.sync.sync() so we can merge PRs that enormously speed up xarray today?"

• @BorisTheBrave built PoC Set variables async pydata/xarray#11171 — an independent PoC that uses zarr's sync() to parallelize variable writing, calling it "just lifting the [async] wrapper one level higher"

• @shoyer noted we'd still need a way to invoke zarr's event loop from inside xarray

Proposal

I'd love to see zarr.core.sync.sync() promoted to public API (e.g., zarr.sync()). This is the minimal change that unblocks both our xarray PR and @BorisTheBrave's approach.

Higher-level APIs like open_children() or public store methods are still valuable additions — but they complement sync(), they don't replace it.

cc @d-v-b @shoyer @TomNicholas @BorisTheBrave @ilan-gold