Relax numpy version constraints #6137

JanuszL · 2025-12-17T06:52:14Z

Removes standalone numpy>=2 package entry
Removes numpy<2 dependency constraints from jupyter,
tensorflow, jaxlib, flax, and numba packages
Updates numba from 0.61.2 to 0.63.1

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

Removes standalone numpy>=2 package entry
Removes numpy<2 dependency constraints from jupyter,
tensorflow, jaxlib, flax, and numba packages
Updates numba from 0.61.2 to 0.63.1

Additional information:

Affected modules and functionalities:

setup_pacakges.py

Key points relevant for the review:

NA

Tests:

Checklist

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: N/A

JIRA TASK: N/A

dali-automaton · 2025-12-17T06:54:20Z

CI MESSAGE: [40348302]: BUILD STARTED

greptile-apps · 2025-12-17T06:55:39Z

Greptile Summary

This PR relaxes numpy version constraints to support both numpy 1.x and 2.x across the DALI codebase.

Key changes:

Removed standalone numpy>=2 package entry from package list
Removed numpy<2 dependency constraints from jupyter, tensorflow, flax, and clu packages
Updated numba from 0.61.2 to 0.63.1 (which supports numpy 2.x) for Python 3.10+
Fixed multiple test files to handle numpy 2.x behavior changes (type coercion, overflow handling, deprecated constants)
Added backward compatibility testing with numpy<2 to ensure both versions work

Packages still constrained to numpy<2:

paddlepaddle-gpu (2.6.1) - external dependency limitation
jax (0.4.16) - old version with comment "ToDo update to newer JAX version some day"
numba (0.59.1) for Python 3.9 only
numba-cuda (all versions) - external dependency limitation

The test fixes address numpy 2.x stricter type checking and behavioral changes around integer overflow, type coercion in array creation, and deprecated constants.

Confidence Score: 4/5

This PR is safe to merge with minor considerations around JAX version constraints
The changes are well-structured and address numpy 2.x compatibility systematically. All test fixes follow correct patterns for handling numpy 2.x behavioral changes. The backward compatibility testing is a good addition. Score reflects one potential improvement: JAX 0.4.16 still has numpy<2 constraint and there's a TODO comment suggesting it should be updated
Pay attention to qa/setup_packages.py - verify that keeping numpy<2 for JAX 0.4.16 is intentional given the TODO comment about updating JAX

Important Files Changed

Filename	Overview
qa/setup_packages.py	Removed standalone numpy>=2 package entry and numpy<2 constraints from jupyter, tensorflow, flax, clu packages; updated numba 0.61.2 to 0.63.1 with numpy<2 constraint removed
dali/test/python/operator_1/test_cast.py	Fixed numpy 2.x compatibility by adding explicit float64 cast after rounding and replacing np.Infinity with np.inf
dali/test/python/operator_2/test_reduce.py	Fixed overflow handling by using int64 intermediate type to prevent numpy 2.x integer overflow behavior change
qa/TL0_python-self-test-core/test_body.sh	Added new test_different_numpy_versions function to verify compatibility with numpy<2 by running core pipeline tests

Sequence Diagram

sequenceDiagram
    participant Dev as Developer
    participant Setup as setup_packages.py
    participant Tests as Test Suite
    participant Numpy as NumPy (1.x/2.x)
    
    Dev->>Setup: Update package constraints
    Setup->>Setup: Remove standalone numpy>=2 entry
    Setup->>Setup: Remove numpy<2 from jupyter, tensorflow, flax, clu
    Setup->>Setup: Update numba 0.61.2 → 0.63.1
    Setup->>Setup: Keep numpy<2 for paddlepaddle, jax, numba-cuda
    
    Dev->>Tests: Fix numpy 2.x compatibility issues
    Tests->>Tests: Update test_cast.py (float64 cast, np.inf)
    Tests->>Tests: Update test_inflate.py (int64 intermediate)
    Tests->>Tests: Update test_python_function.py (numpy.float64)
    Tests->>Tests: Update test_reduce.py (overflow handling)
    Tests->>Tests: Update test_backend_impl.py (float intermediate)
    Tests->>Tests: Update test_random_resized_crop.py (explicit casts)
    
    Dev->>Tests: Add backward compatibility tests
    Tests->>Tests: Add test_different_numpy_versions()
    Tests->>Numpy: Install numpy<2
    Tests->>Tests: Run core pipeline tests
    Numpy-->>Tests: Verify compatibility
    
    Tests->>Numpy: Also test with numpy>=2
    Numpy-->>Tests: Verify compatibility

greptile-apps

Additional Comments (1)

qa/setup_packages.py, line 680-682 (link)

style: numba 0.59.1 for Python 3.9 still has numpy<2 constraint - consider updating if this version supports numpy 2

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

dali-automaton · 2025-12-17T09:35:34Z

CI MESSAGE: [40356541]: BUILD STARTED

dali-automaton · 2025-12-17T20:43:57Z

CI MESSAGE: [40348302]: BUILD FAILED

dali-automaton · 2025-12-17T22:31:30Z

CI MESSAGE: [40394275]: BUILD STARTED

dali-automaton · 2025-12-18T00:25:44Z

CI MESSAGE: [40394275]: BUILD FAILED

dali-automaton · 2025-12-18T06:15:47Z

CI MESSAGE: [40415495]: BUILD STARTED

dali-automaton · 2025-12-18T07:56:04Z

CI MESSAGE: [40415495]: BUILD FAILED

dali-automaton · 2025-12-18T12:15:25Z

CI MESSAGE: [40432108]: BUILD STARTED

dali-automaton · 2025-12-18T12:21:51Z

CI MESSAGE: [40432108]: BUILD FAILED

- Removes standalone numpy>=2 package entry - Removes numpy<2 dependency constraints from jupyter, tensorflow, jax, jaxlib, flax, and numba packages - Updates numba from 0.61.2 to 0.63.1 Signed-off-by: Janusz Lisiecki <[email protected]>

dali-automaton · 2025-12-18T12:33:27Z

CI MESSAGE: [40432727]: BUILD STARTED

dali-automaton · 2025-12-18T14:43:08Z

CI MESSAGE: [40432727]: BUILD FAILED

dali-automaton · 2025-12-18T18:09:05Z

CI MESSAGE: [40432727]: BUILD PASSED

greptile-apps

Additional Comments (1)

dali/test/python/operator_1/test_cast.py, line 34 (link)

syntax: typo: "sing" should be "sign"

_{7 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Janusz Lisiecki <[email protected]>

dali-automaton · 2025-12-19T08:23:10Z

CI MESSAGE: [40489685]: BUILD STARTED

dali-automaton · 2025-12-19T10:12:02Z

CI MESSAGE: [40489685]: BUILD FAILED

Signed-off-by: Janusz Lisiecki <[email protected]>

dali/test/python/operator_2/test_reduce.py

greptile-apps

Additional Comments (3)

qa/setup_packages.py, line 640 (link)

logic: PR description claims to remove numpy<2 dependency constraints from jax, but the constraint remains here
qa/setup_packages.py, line 619 (link)

style: paddlepaddle-gpu still has numpy<2 constraint - verify if this should also be removed per the PR objectives
qa/setup_packages.py, line 701-728 (link)

style: numba-cuda still has numpy<2 constraints on all versions (Python 3.9 and 3.10+, CUDA 120 and 130). Verify if these should also be removed.

_{7 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

Signed-off-by: Janusz Lisiecki <[email protected]>

dali-automaton · 2025-12-19T11:21:52Z

CI MESSAGE: [40497112]: BUILD STARTED

greptile-apps

Additional Comments (2)

qa/setup_packages.py, line 640 (link)

logic: JAX 0.4.16 is incompatible with numpy 2.x - it was released before numpy 2.0 (Feb 2024 vs June 2024) and uses deprecated APIs removed in numpy 2.0 like numpy.Infinity and numpy.float_. This will break all JAX plugin tests.

Per JAX maintainers (GitHub Issue #18779), numpy 2.0 support requires JAX 0.4.25+. Either:
- Upgrade JAX/jaxlib to 0.4.25+ and update flax/clu constraints accordingly
- Keep "numpy<2" in dependencies for JAX 0.4.16
Test files that will fail: dali/test/python/jax_plugin/test_jax_operator.py
qa/setup_packages.py, line 685-688 (link)

style: The comment "Free-threaded Python build is incompatible with numpy<2" is now misleading after removing the numpy<2 constraint. If numpy 2.x is now allowed, but python_free_threaded=False blocks free-threaded builds, the constraint relationship is unclear.

Update comment to clarify what's actually being blocked (e.g., "numba 0.63.1 free-threading support is experimental" or similar)

_{7 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

dali-automaton · 2025-12-19T11:37:27Z

CI MESSAGE: [40497490]: BUILD STARTED

greptile-apps

Additional Comments (5)

qa/setup_packages.py, line 640 (link)

logic: JAX 0.4.16 was released in mid-2024, around the same time as numpy 2.0. This old version likely lacks proper numpy 2.x support and may have compatibility issues. The PR description removes numpy<2 constraints, but this specific JAX version may not be fully compatible.

Check if JAX 0.4.16 actually supports numpy 2.x, or consider either:
- Keeping "numpy<2" in dependencies here
- Updating to a newer JAX version with verified numpy 2.x support
qa/setup_packages.py, line 619 (link)

logic: Paddlepaddle-gpu still has "numpy<2" constraint while the PR aims to relax numpy version constraints. This creates an inconsistency where some packages support numpy 2.x while paddlepaddle forces numpy<2.
qa/setup_packages.py, line 701-708 (link)

logic: All numba-cuda versions still have "numpy<2" constraints. This contradicts the PR's goal to relax numpy version constraints and creates dependency conflicts when other packages now allow numpy 2.x.
qa/setup_packages.py, line 680 (link)

logic: Numba 0.59.1 for Python 3.9 still has "numpy<2" constraint. This creates inconsistency where Python 3.10+ gets numba 0.63.1 with numpy 2.x support, but Python 3.9 is restricted to numpy<2.
dali/test/python/operator_1/test_cast.py, line 117 (link)

style: np.Infinity was deprecated and removed in numpy 2.x, replaced with np.inf

_{9 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}

dali-automaton · 2025-12-19T11:56:11Z

CI MESSAGE: [40497919]: BUILD STARTED

dali-automaton · 2025-12-19T12:05:16Z

CI MESSAGE: [40489685]: BUILD PASSED

dali-automaton · 2025-12-19T13:44:12Z

CI MESSAGE: [40497919]: BUILD FAILED

Signed-off-by: Janusz Lisiecki <[email protected]>

dali-automaton · 2025-12-19T13:48:53Z

CI MESSAGE: [40501672]: BUILD STARTED

dali-automaton · 2025-12-19T16:38:42Z

CI MESSAGE: [40507599]: BUILD STARTED

dali-automaton · 2025-12-19T17:47:58Z

CI MESSAGE: [40501672]: BUILD FAILED

dali-automaton · 2025-12-19T18:18:32Z

CI MESSAGE: [40501672]: BUILD PASSED

dali-automaton · 2025-12-19T18:22:01Z

CI MESSAGE: [40507599]: BUILD PASSED

dali-automaton · 2025-12-19T18:25:50Z

CI MESSAGE: [40497490]: BUILD FAILED

greptile-apps bot reviewed Dec 17, 2025

View reviewed changes

dali-automaton assigned szkarpinski and stiepan Dec 17, 2025

JanuszL marked this pull request as draft December 17, 2025 21:05

JanuszL force-pushed the relax_numpy branch from 80bdc47 to bbaf606 Compare December 17, 2025 22:28

JanuszL force-pushed the relax_numpy branch from bbaf606 to 1e68f24 Compare December 18, 2025 12:14

Relax numpy version constraints

119e9dc

- Removes standalone numpy>=2 package entry - Removes numpy<2 dependency constraints from jupyter, tensorflow, jax, jaxlib, flax, and numba packages - Updates numba from 0.61.2 to 0.63.1 Signed-off-by: Janusz Lisiecki <[email protected]>

JanuszL force-pushed the relax_numpy branch from 1e68f24 to 119e9dc Compare December 18, 2025 12:32

JanuszL marked this pull request as ready for review December 18, 2025 18:21

greptile-apps bot reviewed Dec 18, 2025

View reviewed changes

Fix

bbb8c60

Signed-off-by: Janusz Lisiecki <[email protected]>

JanuszL force-pushed the relax_numpy branch from a0fa924 to bbb8c60 Compare December 19, 2025 08:21

Fix

5cc4932

Signed-off-by: Janusz Lisiecki <[email protected]>

github-advanced-security bot found potential problems Dec 19, 2025

View reviewed changes

dali/test/python/operator_2/test_reduce.py Fixed Show fixed Hide fixed

greptile-apps bot reviewed Dec 19, 2025

View reviewed changes

Reert change

d180ad9

Signed-off-by: Janusz Lisiecki <[email protected]>

greptile-apps bot reviewed Dec 19, 2025

View reviewed changes

szkarpinski approved these changes Dec 19, 2025

View reviewed changes

greptile-apps bot reviewed Dec 19, 2025

View reviewed changes

JanuszL force-pushed the relax_numpy branch from 25f205f to 45976b1 Compare December 19, 2025 11:55

mzient approved these changes Dec 19, 2025

View reviewed changes

mzient assigned mzient and unassigned stiepan Dec 19, 2025

More tests

52a1dcb

Signed-off-by: Janusz Lisiecki <[email protected]>

JanuszL force-pushed the relax_numpy branch from 45976b1 to 52a1dcb Compare December 19, 2025 13:47

JanuszL merged commit cb65d67 into NVIDIA:main Dec 19, 2025
6 of 7 checks passed

JanuszL deleted the relax_numpy branch December 19, 2025 18:28

Relax numpy version constraints #6137

Relax numpy version constraints #6137

Uh oh!

Conversation

JanuszL commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

greptile-apps bot commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 18, 2025

Uh oh!

dali-automaton commented Dec 18, 2025

Uh oh!

dali-automaton commented Dec 18, 2025

Uh oh!

dali-automaton commented Dec 18, 2025

Uh oh!

dali-automaton commented Dec 18, 2025

Uh oh!

dali-automaton commented Dec 18, 2025

Uh oh!

dali-automaton commented Dec 18, 2025

Uh oh!

dali-automaton commented Dec 18, 2025

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

dali-automaton commented Dec 19, 2025

Uh oh!

dali-automaton commented Dec 19, 2025

Uh oh!

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (3)

Uh oh!

dali-automaton commented Dec 19, 2025

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (2)

Uh oh!

dali-automaton commented Dec 19, 2025

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

JanuszL commented Dec 17, 2025 •

edited

Loading

greptile-apps bot commented Dec 17, 2025 •

edited

Loading

greptile-apps bot left a comment •

edited

Loading

greptile-apps bot left a comment •

edited

Loading

greptile-apps bot left a comment •

edited

Loading

greptile-apps bot left a comment •

edited

Loading

greptile-apps bot left a comment •

edited

Loading