fix(pyzes): resolve libze_loader via portable path fallback (fixes non-Debian Linux)#496
Open
saforem2 wants to merge 1 commit into
Open
Conversation
pyzes _LoadZeLibrary hardcoded the Debian/Ubuntu multiarch path
/usr/lib/x86_64-linux-gnu/libze_loader.so.1 on Linux. On non-Debian
systems (RHEL/SUSE/Fedora, and HPC images such as ALCF Aurora/Sunspot)
the loader lives in /usr/lib64, so `import pyzes` -- and therefore
torch.xpu.is_available() / device_count() which import it -- failed with:
OSError: /usr/lib/x86_64-linux-gnu/libze_loader.so.1:
cannot open shared object file: No such file or directory
Give the Linux branch the same possible_paths fallback the Windows branch
already uses: try the bare soname first (so the dynamic linker resolves it
via LD_LIBRARY_PATH/ld.so.cache/RUNPATH -- works on every distro,
including Debian), then fall back to the Debian multiarch dir, /usr/lib64,
and /usr/lib. Also drop the now-redundant trailing linux CDLL(libName).
Verified on ALCF Sunspot (Intel Max 1550, RHEL-family, oneAPI 2026.1.0):
torch.xpu.is_available() -> True, device_count() -> 12 after the fix.
Signed-off-by: Sam Foreman <saforem2@gmail.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes pyzes failing to import on non-Debian Linux distributions by making Level Zero loader resolution rely on the platform dynamic linker first, then falling back to common distro-specific library locations.
Changes:
- On Linux, attempt to load
libze_loader.so.1by soname first (lettingLD_LIBRARY_PATH/ld.so.cache/ RUNPATH resolve it). - Add explicit fallback locations (
/usr/lib/x86_64-linux-gnu,/usr/lib64,/usr/lib) and aggregate per-path load errors on failure. - Remove the now-redundant unconditional Linux
CDLL(libName)call.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+97
to
+101
| if not library_loaded: | ||
| raise Exception( | ||
| "Failed to load Level Zero loader on Linux. " | ||
| f"Tried paths: {possible_paths}. Errors: {load_errors}" | ||
| ) |
saforem2
added a commit
to saforem2/torchtitan
that referenced
this pull request
Jul 14, 2026
…ath bug pyzes (imported by torch.xpu) hardcodes /usr/lib/x86_64-linux-gnu/libze_ loader.so.1; on Sunspot/Aurora (RHEL, loader in /usr/lib64) import pyzes -> OSError -> torch.xpu crashes the whole job. A direct site-packages patch is wiped on any venv rebuild (hit exactly that: the 2026.1.0 venv refresh at 13:17 reverted the morning patch). This idempotent script rewrites the hardcoded path to the bare soname (dynamic linker resolves via /usr/lib64). Locates pyzes.py BY PATH via site.getsitepackages() -- NOT by importing it, since buggy pyzes crashes on import (chicken-and-egg). Re-run after activating the venv; no-op if already patched or pyzes absent. Verified: restores xpu True/12 after a simulated refresh. Upstream fix in flight: oneapi-src/level-zero#496. Usage: source venvs/<ver>/bin/activate && bash torchtitan/experiments/ezpz/pyfix/fix_pyzes.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #485.
Problem
pyzes._LoadZeLibraryhardcodes the Debian/Ubuntu multiarch path on Linux:On non-Debian systems (RHEL/SUSE/Fedora, and HPC images such as ALCF
Aurora/Sunspot) the Level-Zero loader lives in
/usr/lib64, soimport pyzes—and therefore
torch.xpu.is_available()/torch.xpu.device_count(), whichimport it — fails with:
Fix
Give the Linux branch the same
possible_pathsfallback that the Windows branchalready uses. Try the bare soname first so the dynamic linker resolves it via
the standard search path (
LD_LIBRARY_PATH/ld.so.cache/ RUNPATH) — thisworks on every distro, including Debian (where
/usr/lib/x86_64-linux-gnuis already on the default search path). Then fall back to explicit locations:
the Debian multiarch dir (preserves the old behavior),
/usr/lib64(RHEL/SUSE/Fedora), and
/usr/lib. Collect and report per-path errors on totalfailure, matching the Windows branch. The now-redundant trailing
if linux: gpuLib = CDLL(libName)is removed.Why not
ctypes.util.find_library("ze_loader")(suggested in #485)find_libraryworks but has drawbacks on the exact systems that hit this bug:it invokes
gcc/objdump/ldconfigat runtime (slow, and often unavailable onminimal HPC compute images), and its results for versioned
.so.1sonames areinconsistent across platforms. Handing the bare soname
libze_loader.so.1directly to
CDLLlets the loader do the resolution with no subprocess and noextra toolchain dependency, and the explicit fallbacks cover the non-default
locations.
Testing
ALCF Sunspot (Intel Data Center GPU Max 1550, RHEL-family, oneAPI 2026.1.0),
inside a compute-node allocation:
import pyzes->OSError(path above);torch.xpu.is_available()->False,device_count()->0.torch.xpu.is_available()->True,torch.xpu.device_count()->12.+30 / -4, no API change.pyzes.pyis hand-written (not.mako-generated),so no template regeneration is required. Commit is DCO signed-off.