docs: add roadmap page outlining future plans by d-v-b · Pull Request #4149 · zarr-developers/zarr-python

d-v-b · 2026-07-15T20:08:18Z

Adds a Roadmap page to the documentation describing the goals for the
next major cycle of work ("v4"), the intended changes per theme, and
the three-stream release model (additive minors, deprecations, one
minimal removals-only major), with an option for a legacy module.

Assisted-by: ClaudeCode:claude-fable-5

Author attestation

I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

Adds a Roadmap page to the documentation describing the goals for the next major cycle of work ("v4"), the intended changes per theme, and the three-stream release model (additive minors, deprecations, one minimal removals-only major). Assisted-by: ClaudeCode:claude-fable-5

d-v-b · 2026-07-15T20:08:46Z

cc @ilan-gold @maxrjones, opening this as a draft for visibility. lmk if there's anything more or less we should say here

ilan-gold · 2026-07-16T15:08:29Z

This doesn't lead me where I thought it would, somehow linking to an old page

ilan-gold · 2026-07-16T15:10:56Z

+Zarr-Python around the stack, so that each level is something you can depend
+on, conform to, or replace, without buying every other level:
+
+- **A focused package per level** — `zarr-metadata`, `zarr-store`,


Would maybe note the zarrs example here

good point, will add something crediting zarrs with this layout

ilan-gold · 2026-07-16T15:16:05Z

+complementary rather than competitive: Zarr-Python aims to be the best
+pure-Python Zarr implementation *and* the best wrapper around the
+compiled-language implementations, so that users who need native throughput can


I'm really conflicted on this, because I think baking fragmentation into the ecosystem is not great, but the flipside is that languages may have different features/advantages. It would be great to have a concrete reason (or list of reasons) as to why we would design around this level of extensibility

good points, I'd say we want the backend to be an extension point for a few reasons:

we always want a traditional python impl. IMO it's our job as the python implementation, and it keeps the ecosystem healthy to have multiple implementations

we can't guarantee uniform, complete feature support across all implementations. As long as we allow people to hack on stuff in python, there will be things the python impl does that e.g. a rust impl doesn't support, and that's OK. For now, a compiled backend is effectively an external dep, and we simply can't promise what external deps will do in perpetuity.

IMO a full backend is just an refined version of our current model, which requires extenders manage buffer prototypes, data type implementations, codec implementations, and store implementations. Consolidating all of this pluggability on a single surface should make extending Zarr Python easier even for other python backends (like a cupy backend, or a pysparse backend)

maxrjones

This is awesome! thanks so much for your work on this @d-v-b! 🚀

maxrjones · 2026-07-17T14:57:56Z

+Make Zarr-Python's IO surfaces device-agnostic rather than adding GPU support
+as a bolted-on feature: stores and codecs grow APIs for writing into a
+caller-provided buffer (`read_into`, `decode_into`), and the `Array` facade
+returns array-like objects in the user's chosen Array API namespace. GPU
+support falls out once the assumption of CPU destinations is removed, and CPU
+paths get faster too, because pre-allocated output buffers eliminate per-chunk
+allocation.


@TomAugspurger it'd be great to get your thoughts on this component in particular.

maxrjones · 2026-07-17T15:24:43Z

This doesn't lead me where I thought it would, somehow linking to an old page

fyi @d-v-b this can be fixed by removing this redirect:

zarr-python/mkdocs.yml

Line 204 in 50b7e01

'roadmap.md': 'https://zarr.readthedocs.io/en/v3.0.8/developers/roadmap.html'

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>

codecov · 2026-07-17T15:31:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.12%. Comparing base (f4a239c) to head (a56b9ba).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4149   +/-   ##
=======================================
  Coverage   94.12%   94.12%           
=======================================
  Files          92       92           
  Lines       12826    12826           
=======================================
  Hits        12072    12072           
  Misses        754      754

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>

The mkdocs-redirects plugin was generating a redirect stub for roadmap.md pointing at the old v3.0.8 docs, which clobbered the new roadmap page added in this PR. The developers/roadmap.html redirect is kept so old links to the historical v3 design roadmap still resolve. Assisted-by: ClaudeCode:claude-fable-5

…into docs/roadmap

kylebarron · 2026-08-10T20:50:07Z

+Move configuration from "global mutable state read implicitly" to "typed data
+passed explicitly": a typed config object replacing the untyped global `donfig`
+dict, array-scoped runtime config passed at open time, a registry redesign that
+addresses implementations by stable identity and resolves plugin name-conflicts
+deliberately, and named profiles replacing global mutators.


FWIW I would like to prototype using a context manager in Zarrista for scope-local configuration.

See developmentseed/zarrista#153. In particular the idea is to borrow from rasterio.Env, which injects settings only for that local scope.

with rasterio.Env(GDAL_CACHEMAX=128000000) as env: # All drivers are registered, GDAL's raster block cache # size is set to 128 MB. # Commence processing... ... # End of processing. # At this point, configuration options are set to their # previous (possible unset) values.

This kind of thing relies on two routines or objects being the same, modulo the configuration. So why not make this explicit? for functions its easy, we declare their configuration in a parameter. for objects we can do something similar, by initializing the object with a configuration, and copying the object with a different configuration as needed, e.g. new_object = StatefulObject.with_config(new_config). IMO this is cleaner because it means runtime internals don't need to know about environment variables. they just inspect local data -- function parameters or the attributes of self -- for configuration info.

This kind of thing relies on two routines or objects being the same, modulo the configuration. So why not make this explicit?

We do currently make this explicit as a parameter into each function. The context manager would be an abstraction to automatically pass down the config into all zarrista calls within that scope.

I'm still not sure whether I like the idea though.

don't need to know about environment variables.

Oh but to be clear, I didn't have in mind environment variables. Despite the name of rasterio.Env, I don't think it's setting global environment variables, because that wouldn't be scope-local.

I had in mind a context manager that would hold config as a Python object, and that local config could be queried. But I don't think Zarrista should touch shell environment variables at all

that local config could be queried.

this is a pattern I don't really like. if an object like a function or class instance depends on some other data, then that should be formalized as part of the function signature or class structure. zarr-python today has a lot of places where code reaches for a free-floating config object, when IMO the simpler solution is to add parameters to functions / classes.

Co-authored-by: Kyle Barron <kylebarron2@gmail.com>

d-v-b · 2026-08-11T14:25:29Z

I gave the text a sweep, folks should have another look.

…into docs/roadmap

d-v-b · 2026-08-11T19:04:10Z

@zarr-developers/python-core-devs please have a look!

d-v-b · 2026-08-12T10:41:38Z

I'd like to get this merged soon, so I will self-merge today unless anyone raises objections to specific content here, in which case we can refine things and then merge soon. Otherwise I recommend we continue iterating on the roadmap in follow up PRs.

maxrjones

I agree with everything in this roadmap. I also support merging this since there's been a long window for comments, and folks can still open PRs against the roadmap.

maxrjones · 2026-07-21T23:43:03Z

+The 3.x redesign was carried out under hard backwards-compatibility
+constraints, and it inherited many structural patterns from the 2.x
+implementation it replaced. The library has never had a release cycle whose
+primary goal was the *shape* of the internals. The next body of work — which we
+call **"v4"** — is that overdue investment.


Suggested change

The 3.x redesign was carried out under hard backwards-compatibility

constraints, and it inherited many structural patterns from the 2.x

implementation it replaced. The library has never had a release cycle whose

primary goal was the *shape* of the internals. The next body of work — which we

call **"v4"** — is that overdue investment.

The 3.x redesign was carried out under hard backwards-compatibility

constraints, and it inherited many structural patterns from the 2.x

implementation it replaced. These structural patterns have made it

challenging to evolve the API for usability (e.g., many argument are

counter-intuitive and lead to user error) and provide excellent performance

across backends (e.g., local and hybrid systems suffer from

Python overhead with our current design). The goal of **v4** is

to provide a highly usable library that delivers first-class performance

across all backends. We recognize that accomplishing these goals is

only possible via improvements to Zarr Python's internals, and consider

the investment in the Zarr Python internals to be a crucial part of

the **v4*** effort.

I'd like this section to be explicitly motivated for reasons beyond just cleaning up the shape of the internals.

this suggestion was based off an earlier version of the PR; I didn't realize it'd still be included in my review. It probably isn't still necessary, but I'll leave it.

maxrjones · 2026-08-12T22:35:08Z

+need read-only array access; other tools need everything. We think of this as a
+"Zarr stack", from most abstract to most concrete:
+
+1. **Conventions** — application and/or domain-specific schemas built on top of Zarr (OME-NGFF,


I prefer keeping it as just "conventions" since specification may confuse folks who think of the core zarr spec

d-v-b added 2 commits July 15, 2026 09:56

docs: refine roadmap

9304a06

Merge branch 'main' into docs/roadmap

199b4e1

ilan-gold reviewed Jul 16, 2026

View reviewed changes

maxrjones reviewed Jul 17, 2026

View reviewed changes

d-v-b and others added 3 commits July 17, 2026 17:26

Update roadmap.md

bd0f1f1

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>

Update roadmap.md

641d5c0

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>

Update roadmap.md

649f60b

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>

d-v-b and others added 4 commits July 21, 2026 20:31

Update docs/roadmap.md

8072b42

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>

Merge branch 'main' into docs/roadmap

e4a49db

Rename +roadmap.doc.md to 4149.doc.md

c0e1f1f

maxrjones mentioned this pull request Jul 22, 2026

ODD PI 26.3 Objective 2: 🌳 Evolve the Zarr Ecosystem NASA-IMPACT/veda-odd#347

Closed

4 tasks

d-v-b mentioned this pull request Jul 30, 2026

fully separate sync and async internal APIs #4217

Open

d-v-b added 3 commits August 10, 2026 20:30

Merge branch 'main' of https://github.com/zarr-developers/zarr-python …

2fb5fa0

…into docs/roadmap

docs: update roadmap per feedback from review. it's simpler

69bb812

docs: lint

c2ce491

d-v-b marked this pull request as ready for review August 10, 2026 19:27

TomNicholas reviewed Aug 10, 2026

View reviewed changes

Comment thread docs/roadmap.md Outdated

kylebarron approved these changes Aug 10, 2026

View reviewed changes

d-v-b and others added 6 commits August 11, 2026 08:05

Update docs/roadmap.md

86c7ce8

Co-authored-by: Kyle Barron <kylebarron2@gmail.com>

Update docs/roadmap.md

e564b35

Co-authored-by: Kyle Barron <kylebarron2@gmail.com>

Update docs/roadmap.md

15c2307

Co-authored-by: Kyle Barron <kylebarron2@gmail.com>

Update docs/roadmap.md

ce5fea6

Co-authored-by: Kyle Barron <kylebarron2@gmail.com>

Update docs/roadmap.md

1f88796

Co-authored-by: Kyle Barron <kylebarron2@gmail.com>

docs: amend motivation

1b8093b

d-v-b added 5 commits August 11, 2026 14:43

docs: update section about coordination / synchronization during IO

68f964c

chore: lint

217249f

docs: update roadmap

7040e60

docs: prose / tightening

951535d

chore: lint

2b05103

kylebarron mentioned this pull request Aug 11, 2026

Support global configuration? developmentseed/zarrista#153

Closed

kylebarron approved these changes Aug 11, 2026

View reviewed changes

d-v-b added 3 commits August 11, 2026 17:56

docs: prose

dd85104

Merge branch 'main' of https://github.com/zarr-developers/zarr-python …

fa9b283

…into docs/roadmap

docs: not by whatever means necessary

2d92b87

d-v-b added 2 commits August 11, 2026 21:23

Merge branch 'main' into docs/roadmap

451e6e1

Merge branch 'main' into docs/roadmap

3410198

d-v-b mentioned this pull request Aug 12, 2026

Release Zarr-Python v3.3.1 #4256

Open

14 tasks

maxrjones reviewed Aug 14, 2026

View reviewed changes

Merge branch 'main' into docs/roadmap

a56b9ba

d-v-b merged commit 7431699 into zarr-developers:main Aug 14, 2026
27 checks passed

-The 3.x redesign was carried out under hard backwards-compatibility
-constraints, and it inherited many structural patterns from the 2.x
-implementation it replaced. The library has never had a release cycle whose
-primary goal was the *shape* of the internals. The next body of work — which we
-call **"v4"** — is that overdue investment.
+The 3.x redesign was carried out under hard backwards-compatibility
+constraints, and it inherited many structural patterns from the 2.x
+implementation it replaced. These structural patterns have made it
+challenging to evolve the API for usability (e.g., many argument are
+counter-intuitive and lead to user error) and provide excellent performance
+across backends (e.g., local and hybrid systems suffer from
+Python overhead with our current design). The goal of **v4** is
+to provide a highly usable library that delivers first-class performance
+across all backends. We recognize that accomplishing these goals is
+only possible via improvements to Zarr Python's internals, and consider
+the investment in the Zarr Python internals to be a crucial part of
+the **v4*** effort.

Uh oh!

Uh oh!

Conversation

d-v-b commented Jul 15, 2026

Author attestation

Uh oh!

d-v-b commented Jul 15, 2026

Uh oh!

ilan-gold commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

maxrjones left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maxrjones commented Jul 17, 2026

Uh oh!

codecov Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-v-b commented Aug 11, 2026

Uh oh!

d-v-b commented Aug 11, 2026

Uh oh!

d-v-b commented Aug 12, 2026

Uh oh!

maxrjones left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

ilan-gold commented Jul 16, 2026 •

edited

Loading

codecov Bot commented Jul 17, 2026 •

edited

Loading