Skip to content

feat(data): migrate zarr v2 → v3 (3.1.6) with zero-dead-space itar writes #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
janickm wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat(data): migrate zarr v2 → v3 (3.1.6) with zero-dead-space itar writes #68

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 1 addition & 10 deletions MODULE.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,7 @@ python.toolchain(
is_default = True,
python_version = "3.11",
)
python.toolchain(
python_version = "3.8",
)
use_repo(python, "python_3_11", "python_3_8", "python_versions")
use_repo(python, "python_3_11", "python_versions")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
Expand All @@ -63,12 +60,6 @@ pip.parse(
python_version = "3.11",
requirements_lock = "//deps/pip:requirements_3_11.txt",
)
pip.parse(
experimental_index_url = "https://pypi.org/simple", # download wheels via bazel downloader instead of pip
hub_name = "ncore_pip_deps",
python_version = "3.8",
requirements_lock = "//deps/pip:requirements_3_8.txt",
)
use_repo(pip, "ncore_pip_deps")

## rules_multitool (for ruff formatter)
Expand Down
15 changes: 0 additions & 15 deletions deps/pip/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ load("@rules_uv//uv:pip.bzl", "pip_compile")
# Export lock files for use as constraints by other modules
exports_files([
"requirements_3_11.txt",
"requirements_3_8.txt",
])

# Check that our compiled pip requirements are up-to-date
Expand Down Expand Up @@ -48,25 +47,11 @@ pip_compile(
requirements_txt = "requirements_3_11.txt",
)

pip_compile(
name = "requirements_3_8",
size = "medium",
args = PIP_COMPILE_ARGS,
data = [
":requirements_ncore.in",
":requirements_tests.in",
],
py3_runtime = "@python_3_8//:py3_runtime",
requirements_in = "requirements_3_8.in",
requirements_txt = "requirements_3_8.txt",
)

# Single command to update all requirements files (no order dependency)
multirun(
name = "update_all_requirements",
commands = [
":requirements_3_11",
":requirements_3_8",
],
keep_going = False, # Fail on error
)
25 changes: 0 additions & 25 deletions deps/pip/requirements_3_8.in
View file
Open in desktop

This file was deleted.

281 changes: 0 additions & 281 deletions deps/pip/requirements_3_8.txt
View file
Open in desktop

This file was deleted.

13 changes: 5 additions & 8 deletions deps/pip/requirements_ncore.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,15 @@ numpy
dataclasses_json>=0.2.12 # deserialization fails with earlier versions
pillow

# zarr version rationales:
# - 2.17.0 contains crucial perf improvements ["Cache result of FSStore._fsspec_installed()" https://github.com/zarr-developers/zarr-python/pull/1581]
# - 2.12.0 is the last version supporting python3.8 with old numpy versions, which we still want to support for now
# - we can't switch to zarr 3 yet unconditionally as it's not available for python3.8 anymore
zarr>=2.12.0,<3.0.0
# zarr version rationale:
# - 3.1.6 is the zarr3 release used for the zarr v2→v3 migration
# - Python 3.8 support was dropped; zarr3 requires Python >=3.11
zarr>=3.1.6

# cbor2 version rationale:
# - 5.9.0 fixes CWE-674 / CVE-2026-26209 (uncontrolled recursion DoS)
# - 5.8.0 fixes CVE-2025-68131 (shared value info disclosure)
# - 5.8.0+ dropped Python 3.8 support; last 3.8-compatible version is 5.7.0
cbor2>=5.9.0; python_version>="3.9"
cbor2; python_version<"3.9"
cbor2>=5.9.0
scipy
torch
typing-extensions
Expand Down
10 changes: 4 additions & 6 deletions ncore/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,18 +48,16 @@ py_wheel(
homepage = "https://github.com/NVIDIA/ncore",
license = "Apache-2.0",
platform = "any",
python_requires = ">=3.8",
python_requires = ">=3.11",
python_tag = "py3",
requires = [
"numpy",
# dataclasses_json version restriction rationale: deserialization fails with earlier versions
"dataclasses_json>=0.2.12",
"pillow",
# zarr version restrictions / suggestions rationale:
# - 2.17.0 contains crucial perf improvements ["Cache result of FSStore._fsspec_installed()" https://github.com/zarr-developers/zarr-python/pull/1581]
# - 2.12.0 is the last version supporting python3.8 with old numpy versions, which we still want to support for now
# - we can't switch to zarr 3 yet unconditionally as it's not available for python3.8 anymore
"zarr>=2.12.0,<3.0.0",
# zarr version rationale:
# - 3.1.6 is the zarr3 release used for the zarr v2→v3 migration
"zarr>=3.1.6",
"cbor2",
"scipy",
"torch",
Expand Down
4 changes: 1 addition & 3 deletions ncore/impl/common/transformations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@

from __future__ import annotations

import sys

from dataclasses import dataclass
from typing import TYPE_CHECKING, Dict, List, Optional, Set, Tuple, Union

Expand Down Expand Up @@ -62,7 +60,7 @@ def time_bounds(timestamps_us: List[int], seek_sec: Optional[float], duration_se
return start_timestamp_us, end_timestamp_us


@dataclass(**({"slots": True, "frozen": True} if sys.version_info >= (3, 10) else {"frozen": True}))
@dataclass(slots=True, frozen=True)
class HalfClosedInterval:
"""Represents a half closed interval [start, stop) of integers"""

Expand Down
Loading
Loading