Skip to content

Linux wheel ships build-machine absolute paths in _portable_lib RUNPATH #21611

Description

@shoumikhin

The Linux wheel's _portable_lib extension ships with absolute paths from the machine that built it in its runtime search path, including a path that exists on no user's machine, plus two Windows directories.

What a user gets

From a pip install executorch on Linux x86_64:

$ readelf -d .../executorch/extension/pybindings/_portable_lib.cpython-312-x86_64-linux-gnu.so
  RUNPATH  [$ORIGIN/../../../torch/lib
            /lib/intel64
            /lib/intel64_win
            /lib/win-x64
            /__w/_temp/conda_environment_21402829909/lib/python3.12/site-packages/torch/lib]

Four of those five entries are not intended:

  • /__w/_temp/conda_environment_<number>/... is an absolute path to the conda
    environment on the build machine. It cannot exist for a user, and the number
    changes from build to build, so it also makes the artifact non-reproducible.
  • /lib/intel64_win and /lib/win-x64 are Windows layouts, in a Linux wheel.
  • /lib/intel64 is a bare absolute path that resolves to whatever the user
    happens to have there.

Only the first entry, $ORIGIN/../../../torch/lib, is deliberate and relocatable.

Where they come from

Torch's exported CMake package creates an MKL::MKL imported target, and linking
torch brings it in even though ExecuTorch never asks for MKL. The linker records
the directories it found MKL in. Packaging then copies the built extension out of
the build tree rather than running an install step, so whatever the linker
recorded ships as-is.

The build log makes the discovery visible:

-- MKL_ROOT /opt/conda/envs/py_3.10
-- MKL_ARCH: None, set to ` intel64` by default
-- MKL_INTERFACE_FULL: None, set to ` intel_ilp64` by default

Why it matters beyond untidiness

A build that resolves a dependency through one of those absolute entries is
resolving it through the layout of the build environment, not through anything
the wheel controls. That works on a machine whose filesystem happens to match and
fails elsewhere, and the failure appears at import time:

ImportError: libmkl_intel_ilp64.so.2: cannot open shared object file

So the paths are simultaneously useless to users and load-bearing for the build
environment, which is the worst combination: they cannot be removed without
noticing what was relying on them, and they cannot be relied on either.

Suggested direction

Two parts, and they are separable:

  1. Do not ship a runtime search path a user cannot use. An entry that is absolute
    or names a foreign platform should not survive packaging. This is checkable:
    read RUNPATH/RPATH on every shipped object and reject any absolute entry.
  2. Resolve MKL through something the wheel or its declared dependencies own,
    rather than through a directory the build machine happened to have, so the
    extension does not depend on the build environment's layout at load time.

Reproduction

pip install executorch
python - <<'EOF'
import executorch, pathlib, subprocess
p = next(pathlib.Path(executorch.__path__[0]).glob("extension/pybindings/_portable_lib*.so"))
print(subprocess.run(["readelf", "-d", str(p)], capture_output=True, text=True).stdout)
EOF

Any entry in the printed RUNPATH that starts with / is a path from the build
machine.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions