Partial zarr download/upload (stage: Design + Implementation)#1816
Draft
yarikoptic wants to merge 3 commits intomasterfrom
Draft
Partial zarr download/upload (stage: Design + Implementation)#1816yarikoptic wants to merge 3 commits intomasterfrom
yarikoptic wants to merge 3 commits intomasterfrom
Conversation
yarikoptic
changed the title
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1816 +/- ##
==========================================
+ Coverage 75.12% 75.59% +0.46%
==========================================
Files 84 86 +2
Lines 11930 12225 +295
==========================================
+ Hits 8962 9241 +279
- Misses 2968 2984 +16
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
yarikoptic
added
enhancement
Covers five areas:
- --zarr TYPE:PATTERN filtering for download (glob, path, regex)
- URL parsing with zarr boundary detection (AssetZarrEntryURL)
- --zarr-mode {full, patch} for upload
- Checksums and manifests (per-directory checksums are NOT
persisted on the archive; legacy .checksum files exist on S3
under zarr-checksums/ for ~72% of older zarrs but are orphaned)
- dandi ls for zarr contents
Related: #1462, #1474
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --zarr CLI option for download to filter entries within zarr assets (glob/path/regex patterns with predefined 'metadata' alias), and --zarr-mode option for upload to support 'patch' mode (upload/update without deleting remote-only files). Key changes: - New dandi/zarr_filter.py: filter parsing, matching, and aliases - URL parsing: AssetZarrEntryURL for URLs pointing into zarr assets - Download pipeline: thread zarr_entry_filter through Downloader and _download_zarr, skip deletion and checksum when filter active - Upload pipeline: zarr_mode='patch' skips remote file deletion and client-side checksum verification - dandi ls: list zarr entries when URL points into a zarr Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Pre-compile regex patterns at ZarrFilter construction time, catching
invalid patterns early instead of on every matches() call (B1)
- Remove redundant get_asset_download_path override in AssetZarrEntryURL
that was identical to the inherited SingleAssetURL method (H1)
- Use Literal["full", "patch"] for zarr_mode parameter instead of bare
str to prevent silent misbehavior on invalid values (H2)
- Collapse consecutive ** glob segments to avoid exponential
backtracking in _glob_match_parts (H3)
- Simplify split_zarr_location to use str.split instead of
PurePosixPath (M1)
- Add explanatory comment for type: ignore in parse_zarr_filter (M2)
- Yield {"status": "done"} when zarr filter matches zero entries (M4)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
yarikoptic
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Design document for partial zarr download and upload support, addressing #1462, #1474, and related archive issues.
The design covers five areas:
--zarr TYPE:PATTERNfiltering fordandi download— glob, path, and regex filters for selecting entries within zarr assets, with ametadataalias for common zarr metadata filesAssetZarrEntryURLto handle URLs likedandi://dandi/000108/.../file.ome.zarr/0/0/0--zarr-mode {full, patch}fordandi upload— patch mode uploads changed files without deleting remote files absent locallyzarr_checksumlibrary but are NOT persisted (only the root digest is stored in the DB); legacy.checksumfiles exist on S3 atzarr-checksums/for ~72% of older zarrs but are orphaned since Dec 2022dandi lsfor zarr contents — listing files within a zarr assetKey findings from investigation
zarr_checksumalgorithm IS hierarchical (Merkle tree, bottom-up viaZarrChecksumTree)ingest_zarr_archivetask computes checksums entirely in memory and stores only the root digest.checksumfiles on S3 (zarr-checksums/prefix) were written byZarrChecksumFileUpdater, removed in dandi-archive PRs Always clear checksum files during zarr ingestion dandi-archive#1390 (Dec 2022). Legacy files remain for older zarrs but no API exposes themReview checklist
Please review the design at
doc/design/partial-zarr.mdand comment on:--zarr TYPE:PATTERNsyntax — is the filter approach right? Are glob/path/regex the right types?metadataalias expansion — doesglob:**/.z*+glob:**/zarr.json+glob:**/.zmetadatacover all cases?--zarr-mode patchsemantics — is "upload without deleting" the right default for patch? Should subtree cleanup happen?AssetZarrEntryURLwith zarr boundary detection the right approach?--syncinteraction, server-side glob)zarr-checksums/files on S3 be cleaned up as part of this or separately?TODO (post-review)
dandi/zarr_filter.py— filter parsing and matchingAssetZarrEntryURLandsplit_zarr_location()indandi/dandiarchive.py--zarroption todandi downloadCLI_download_zarr()for partial download support--zarr-modeoption todandi uploadCLIiter_upload()(dandi/files/zarr.py)zarr_modethroughdandi/upload.pydandi lszarr contents support (may be separate PR)🤖 Generated with Claude Code