Build and publish CUDA wheels

[shoumikhin]

The problem

The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says:

with-cuda: disabled

So a user who wants GPU support still clones the repository and builds from source, which is the thing
the shipped libraries were supposed to remove. Nothing publishes them because nothing builds them, and
anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on.

The change

Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel, calling
the same shared matrix generator with CUDA turned on, then narrowing the result.

.github/workflows/build-wheels-cuda-linux.yml            x86_64
.github/workflows/build-wheels-cuda-aarch64-linux.yml    aarch64

architecture	CUDA	Python
x86_64	12.6, 13.0, 13.2	3.10 to 3.13
aarch64	12.6, 13.0, 13.2	3.10 to 3.13

The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather than by
what is convenient to verify. A delegate built against one CUDA version needs an ExecuTorch wheel for that
same version, and a missing version leaves that consumer with nothing to pair with, which fails for
whoever installs the pair rather than for the row that omitted it. 13.2 is published for that reason even
though no machine on hand can execute it.

Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason, so a GPU
row would inherit a known-broken build. A pull request builds the newest CUDA version among the rows on
offer, chosen by preference rather than by naming an exact combination: an exact request degrades quietly,
and asking for a python version the generator does not emit once left a pull request building the oldest
CUDA version while still reporting success.

Jetson devices need their own row, with a different container and a single Python and CUDA version,
because they cannot take a generic aarch64 wheel. That row is in the filter but empty: published PyTorch
stopped shipping device code for those GPUs, so a Jetson row today would build a wheel whose PyTorch
dependency cannot execute on the device. Windows is recorded the same way, because the separate libraries
this wheel exists to ship are Linux only, so a Windows GPU wheel would carry a delegate a C++ application
still could not link.

Which GPUs each wheel carries code for. GPU code is compiled per hardware generation, so a wheel built
by detection alone carries code for whichever GPU the builder happened to have. It then installs on every
machine the row claims and fails the moment a model runs on a different generation, with an error that
looks like a model problem rather than a packaging one. So each row states its architectures:

row	architectures
x86_64 CUDA 13.0 and 13.2	7.5, 8.0, 8.6, 8.9, 9.0, 10.0, 12.0
x86_64 CUDA 12.6	7.5, 8.0, 8.6, 8.9, 9.0
aarch64 CUDA 13.0 and 13.2	8.0, 9.0, 10.0, 11.0, 12.0
aarch64 CUDA 12.6	8.0, 9.0

Both lists are read from the published PyTorch CUDA library for that architecture rather than chosen by
judgement. A delegate is only useful where PyTorch already runs, so an architecture PyTorch supports and
this wheel omits gives a user a wheel that installs and then fails at the first kernel launch. Checking
that way found two omissions worth having: the GPU on the runner that tests these wheels, and a common
desktop card. The newest architecture in each row also gets a portable form, so a GPU newer than the list
can still run by compiling that form at load time, and a CUDA version with no list fails the build rather
than falling back to detection.

The CUDA build reads that list and converts it, because it previously enabled the CUDA language without
naming any architectures and used whatever CMake defaults to. On one device that default is older than the
intrinsics these sources use, and the compile failed with an undefined identifier that reads as a source
problem.

One dependency stops following the CUDA channel. The requirements installer points both PyTorch and
torchao at a CUDA-specific package channel when it detects a toolkit. For PyTorch that is the point. For
torchao it makes the build impossible on aarch64, because that channel publishes no aarch64 build:

channel	files published	usable on aarch64
CUDA-suffixed	2	0
plain nightly	16	6

So torchao resolves from the plain channel, which serves every architecture, while PyTorch keeps its CUDA
channel. Nothing in the wheel links or bundles torchao, and a CUDA wheel built by hand on an aarch64 GPU
machine used the non-CUDA torchao build without issue.

Before and after

BEFORE                                  AFTER

pip install executorch                  pip install executorch
  CPU only, always                        CPU by default
                                          GPU wheels published per CUDA version

GPU support means cloning the           a published wheel carries the delegate
repository and building

a delegate built for CUDA 13.2          every consumer CUDA version has a
has no ExecuTorch wheel to pair with    matching ExecuTorch wheel

Test plan

A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU:

• the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate
• the CUDA runtime is declared, so a user has something to resolve it from
• each CUDA library reaches its runtime through a relative path, not the builder's toolkit directory,
  which resolves on the builder and nowhere else
• the wheel carries device code for every architecture its row claims, read from the built libraries
  rather than assumed. Searched across every shipped library, because the kernels are compiled into their
  own library rather than into the delegate, and which one holds them is an internal detail.

Then everything a CPU wheel is already held to: one owner per component, no build-tree search paths, and a
C++ application outside the wheel able to link what it ships.

Each check was run against a wheel that should fail it as well as one that should pass, because a check
that cannot fail is worse than no check. All of them reject a CPU wheel and accept a CUDA wheel. One did
not fail at first: it looped over libraries that were not there and reported a pass having inspected
nothing, so it now requires at least one to be present.

The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is treated as
not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed run rather than a clear
error. The checks now assert the accepted spelling. The existing rows say "disabled", which happens to
produce the intended result because anything that is not "enable" disables, so they are left alone. The rows also keep whichever builder image the release
provides, and the aarch64 workflow passes the architecture that workflow needs, without which it prepares
an x86_64 job and the first build step fails on a missing conda.

The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to 24, a
pull request narrowed to 1 and to the newest CUDA version even when only one python version is on offer,
and an empty result treated as a failure rather than passed through, since a workflow with no build job
reads as a green check for a build that never happened.

Verified against wheels the release workflow built: both CUDA rows produce a wheel carrying both CUDA
libraries with the runtime declared, the relative search path resolves to the directory those runtime
wheels install into from every depth the wheel ships a library at, and the device code covers the row.

[shoumikhin]

Stack from ghstack (oldest at bottom):

• -> Build and publish CUDA wheels #21668
• Ship the CUDA delegate in the wheel #21645
• Ship the quantized kernels as their own library #21642
• Ship a C++ SDK in the wheel #21639
• Split the runtime out of the Python bindings extension #21610

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21668

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 262 Pending

As of commit 43c89b5 with merge base 48741ac ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.