Ship a C++ SDK in the wheel

[shoumikhin]

The problem

The previous change split the runtime, kernels, delegate, thread pool and profiler out of the
Python extension into separate shared libraries. The wheel ships them, but nothing outside Python
can use them: the installed CMake package names none of the five, so a C++ application has no way
to link them without hard-coding paths into the wheel's private layout.

The headers have the same gap. The package installs the subset a custom-operator build needs,
which leaves out extension/module, the entry point the documentation tells a C++ application to
use. So the wheel ships the libraries to load and run a program and no way to call them.

The change

Names each shipped library as a CMake component, adds a version file so version requests are
checked, and ships the headers a caller needs.

find_package(executorch 1.5 REQUIRED COMPONENTS kernels_optimized)
target_link_libraries(my_app PRIVATE executorch::runtime
                                     executorch::kernels_optimized)

component	library it resolves to
executorch::runtime	libexecutorch.so
executorch::kernels_optimized	libexecutorch_kernels_optimized.so
executorch::backend_xnnpack	libexecutorch_backend_xnnpack.so
executorch::threadpool	libexecutorch_threadpool.so
executorch::etdump	libexecutorch_etdump.so

The names are namespaced because a name containing :: must be an alias or imported target, so
CMake reports a missing one while configuring and says which. A bare name becomes -lexecutorch
and fails later with a worse message, or resolves to an unrelated system library. That matters
more for a wheel than a source build: the wheel's contents depend on how it was built, so a
consumer asking for a delegate it does not carry should be told during configuration.

Each component also carries the retention its library needs. A registration-only library has no
symbol the application references, so the default --as-needed drops it and its static
initializer never runs, leaving a delegate that is linked and unregistered.

Before and after

BEFORE                                  AFTER

the libraries are shipped               find_package(executorch REQUIRED)
but nothing names them                  target_link_libraries(app PRIVATE
                                          executorch::runtime)

find_package(executorch 1.5)            the version is actually checked
accepts any version

#include <executorch/extension/         the header ships
  module/module.h>   not shipped

The runtime component carries the thread pool as well as the compile definition that needs it. That
definition switches a header from an inline serial helper to a bare declaration, and the thread pool
library holds the only definition of what it declares, so a consumer linking only the runtime failed
to link with an undefined reference.

The version file reports two different things, so they are filled separately. CMake compares the
numeric release, and the full version, including any prerelease suffix and local label, is what a
consumer pins against when an exact build pairing is required. Both were filled from the same
placeholder, so the full version came out truncated and a consumer comparing it would pass against a
different wheel. Deriving no numeric version now fails the build rather than writing 0, which would
otherwise satisfy every version request.

Test plan

A standalone application built outside the wheel, exporting a real model, running it from C++
through Module, and comparing against eager PyTorch, because a model that returns wrong numbers
without erroring satisfies every other check:

• the version file carries the numeric release for CMake to compare AND the full build version
  separately, checked against several realistic version strings including local labels
• find_package accepts the installed version and an older request, and rejects a newer one
• the C++ example in the documentation compiles against the wheel, extracted from the page rather
  than copied, so the two cannot drift
• linking only the runtime loads a program and reports every operator missing, which is the split
  working rather than a defect, and fails if the runtime starts carrying kernels again
• adding the kernels component runs the model and matches eager PyTorch
• adding the delegate runs a delegated model and matches
• the same delegated program fails in an application that linked the kernels but not the
  delegate, which shows the component is what registers it
• the application still runs after being copied away from the wheel with the absolute search path
  removed, so the package is relocatable
• a consumer linking only the runtime compiles and links, which is what the thread pool being
  carried with its compile definition is for
• one registry serves the process: an application linking five components sees exactly one more
  backend than one linking two

Ran against an installed wheel on Linux x86_64 and aarch64. The C++ output matches eager PyTorch
to 2.4e-07 in every executing case.

[shoumikhin]

Stack from ghstack (oldest at bottom):

• Build and publish CUDA wheels #21668
• Ship the CUDA delegate in the wheel #21645
• Ship the quantized kernels as their own library #21642
• -> Ship a C++ SDK in the wheel #21639
• Split the runtime out of the Python bindings extension #21610

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21639

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 455 Pending, 2 Unrelated Failures, 1 Unclassified Failure

As of commit c6bd747 with merge base 48741ac ():

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

• Build macOS Wheels / pytorch/executorch / build-wheel-py3_14-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  pyarrow

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Test RISC-V Backend / test-riscv (llama2, true, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (mv2, true, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers

This comment was automatically generated by Dr. CI and updates every 15 minutes.