Skip to content

feat(pypi): add native local wheel override support to bzlmod pip.parse#3768

Open
meteorcloudy wants to merge 7 commits into
bazel-contrib:mainfrom
meteorcloudy:local-wheel-override
Open

feat(pypi): add native local wheel override support to bzlmod pip.parse#3768
meteorcloudy wants to merge 7 commits into
bazel-contrib:mainfrom
meteorcloudy:local-wheel-override

Conversation

@meteorcloudy
Copy link
Copy Markdown
Contributor

@meteorcloudy meteorcloudy commented May 11, 2026

This feature allows developers to easily test locally built Python wheels (e.g., wheels built locally before running integration tests) without needing to publish them to an index or manually edit requirements_lock.txt.

This is a proper implementation of the local_wheel_dist_folder feature used in JAX/TF/XLA.

How to Use

In your MODULE.bazel file, configure pip.parse with the new local_wheels dictionary mapping:

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")

pip.parse(
    hub_name = "pip",
    python_version = "3.15",
    requirements_lock = "//:requirements_lock.txt",
    experimental_index_url = "https://pypi.org/simple",
    local_wheels = {
        "my_package": "dist/my_package-0.1.0-py3-none-any.whl",
        "libtpu": "dist/libtpu-*.whl",
    },
)

Details

  • Maps individual package names to local wheel paths relative to your workspace root. Supports middle wildcards (e.g. dist/libtpu-*.whl) and automatically selects the newest version if multiple matching wheels exist. Silently falls back to remote PyPI if the local file is missing.
  • Discovery the workspace root via @@//:MODULE.bazel
  • Effective only when pip.parse is evaluated by the root module.
  • Effective only when index_url is used (which is now by default)

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for overriding PyPI dependencies with local wheels located in a specified distribution folder, restricted to the root module. It adds new attributes to the pip extension for specifying the distribution folder and inclusion/exclusion patterns, implements logic to collect and filter local wheels based on ABI markers, and updates the wheel repository generation to use local file URLs. Feedback from the review highlights potential issues with ABI marker generation when patch versions are present and warns against using lexicographical sorting to resolve multiple matching wheels, suggesting instead to fail on ambiguity.

Comment thread python/private/pypi/hub_builder.bzl Outdated
Comment thread python/private/pypi/hub_builder.bzl Outdated
@meteorcloudy meteorcloudy marked this pull request as draft May 11, 2026 15:09
@meteorcloudy meteorcloudy force-pushed the local-wheel-override branch 4 times, most recently from 30e1a50 to 7083382 Compare May 12, 2026 12:48
@meteorcloudy meteorcloudy force-pushed the local-wheel-override branch from 7083382 to ace01f7 Compare May 12, 2026 13:48
@meteorcloudy meteorcloudy force-pushed the local-wheel-override branch from d8b2229 to 8310448 Compare May 12, 2026 13:54
@meteorcloudy meteorcloudy marked this pull request as ready for review May 12, 2026 14:16
Copy link
Copy Markdown
Collaborator

@aignas aignas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some concerns about this short-circuiting the wheel selection algorithm used in parse_requirements so left a few ideas.


Overrides apply when bazel downloader is used and only take effect in the root module.

:::{versionadded} 2.1.0
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
:::{versionadded} 2.1.0
:::{versionadded} VERSION_NEXT_FEATURE

dirname = "/".join(path.split("/")[:-1]),
_path = path,
)
path_str = str(path)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great to have, but having a few comments would be good.

self._get_index_urls.get(python_version) != None,
)

def _collect_local_wheels(module_ctx, pip_attr, is_root = False):
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a docstring with typing information

if local_wheel:
repo_name += "_local_override"
path_str = local_wheel._path if hasattr(local_wheel, "_path") else str(local_wheel)
args["urls"] = ["file://" + path_str]
Copy link
Copy Markdown
Collaborator

@aignas aignas May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Is this path_str absolute? We could as well use whl_file attribute and pass a label if we were ever able to construct one.

Note to myself - because we are not getting this from pypi_cache.bzl, this will never go into the lock file, which means that whether it is absolute or not does not matter that much.

if not candidate.basename.endswith(".whl"):
continue
if _wildcard_match(candidate.basename, pattern):
if not matched_wheel or matched_wheel.basename < candidate.basename:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relies on lexicographic sorting, but it may be better to:

  1. parse the wheel name (with //python/private:parse_whl_name.bzl or similar)
  2. get the version component
  3. parse the version component (with //python/private:version.bzl)
  4. Do the version comparison.

What do we do if we match multiple abis, platforms? Should we use the select_whl function from parse_requirements.bzl to actually match the right wheel based on target platform?

Maybe the right way would be to actually override the get_index function in case it is a root module to replace the wheels found on the internet with the wheels found locally and then let the wheel selection algorithm do the right thing instead of doing the wild card matching here and this comparison.

copybara-service Bot pushed a commit to jax-ml/jax that referenced this pull request May 13, 2026
On some CI jobs, we need to install our custom wheels for given pip
package. Those were supported via local_wheel_dist_folder in
python_init_repositories.

bazel-contrib/rules_python#3768 implements this
feature in a better way with Bzlmod.

PiperOrigin-RevId: 914756253
copybara-service Bot pushed a commit to jax-ml/jax that referenced this pull request May 13, 2026
On some CI jobs, we need to install our custom wheels for given pip
package. Those were supported via local_wheel_dist_folder in
python_init_repositories.

bazel-contrib/rules_python#3768 implements this
feature in a better way with Bzlmod.

PiperOrigin-RevId: 914810693
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants