Ship the quantized kernels as their own library

[shoumikhin]

The problem

The wheel now ships the runtime, the CPU kernels, the XNNPACK delegate, the thread pool and the
profiler as separate shared libraries a C++ application can link. The quantized kernels are the
one piece from that set still missing.

They exist today only inside libquantized_ops_aot_lib.so, which sits beside the Python bindings
and is built for ahead-of-time export. A C++ application that loads a quantized .pte has no
library to link, so it fails at run time:

Missing operator: [0] quantized_decomposed::dequantize_per_tensor.out

The change

Builds the quantized kernels as a shared library, ships it beside the others, and names it as a
component.

executorch/lib/libexecutorch_kernels_quantized.so

find_package(executorch REQUIRED COMPONENTS kernels_quantized)
target_link_libraries(my_app PRIVATE executorch::runtime
                                     executorch::kernels_quantized)

Named after what it provides rather than after the target that produces it, matching
libexecutorch_kernels_optimized.so next to it.

Two things this needed that were not obvious. The library is named explicitly in the wheel's
build target list, because a C++ application is its only consumer, and without naming it the
target is generated and never built. And the single-owner checks skip the ahead-of-time
libraries, which deliberately carry their own copy because export runs in a process that never
loads the runtime libraries a C++ application links. They are recognised by linking torch rather
than by name, so a rename does not silently drop the exclusion.

Before and after

BEFORE                                  AFTER

a quantized .pte from C++               a quantized .pte from C++
  Missing operator:                       runs, matching eager PyTorch
  quantized_decomposed::
  dequantize_per_tensor.out

nothing to link                         target_link_libraries(app PRIVATE
                                          executorch::kernels_quantized)

Nothing changes for a Python user. The export path keeps using the ahead-of-time library.

Test plan

• exactly one shipped object defines the quantized operators, and it is this library
• a C++ application linking executorch::kernels_quantized exports a quantized model, runs it,
  and matches eager PyTorch

The comparison tolerance is per model rather than one number for all: a correct int8 run differs
from eager PyTorch by about one quantization step, which for this model is 5e-3, while a float32
model matches to within rounding. Using the looser number for both would stop the float path
catching a real regression.

Ran on Linux x86_64 and aarch64 against a wheel built from source and installed into a clean
environment. The quantized model matches eager PyTorch to 4.8e-03, one quantization step.

[shoumikhin]

Stack from ghstack (oldest at bottom):

• Build and publish CUDA wheels #21668
• Ship the CUDA delegate in the wheel #21645
• -> Ship the quantized kernels as their own library #21642
• Ship a C++ SDK in the wheel #21639
• Split the runtime out of the Python bindings extension #21610

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21642

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 2 Pending, 22 Unrelated Failures, 22 Unclassified Failures

As of commit 85164ea with merge base 48741ac ():

NEW FAILURES - The following jobs have failed:

• pull / test-lora-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2f55ac51ab44acdb2394ce107348a882844e157c731023815c2ce2f28ebf220d /exec failed with exit code 1
• pull / test-lora-multimethod-linux / linux-job (gh)
  RuntimeError: Command docker exec -t acfd9645407fa22d487d1d0dd3db17b1f70968e240ab3930f05e250a0960af94 /exec failed with exit code 1
• pull / unittest / linux / linux-job (gh)
  extension/llm/modules/test/test_turboquant_kv_cache.py::TestCodebook::test_rotation_orthogonal

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Aarch64 Linux Wheels / pytorch/executorch / build-wheel-py3_14-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  pyarrow
• Build Aarch64 Linux Wheels / pytorch/executorch / upload / upload-wheel-py3_14-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux Wheels / pytorch/executorch / build-manywheel-py3_14-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  pyarrow
• Build Linux Wheels / pytorch/executorch / upload / upload-manywheel-py3_14-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_14-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  pyarrow
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_14-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Windows Wheels / pytorch/executorch / build-wheel-py3_14-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  pyarrow
• Build Windows Wheels / pytorch/executorch / upload / upload-wheel-py3_14-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Test CUDA Builds / export-model-cuda-artifact (google, gemma-3-4b-it, non-quantized) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (google, gemma-3-4b-it, quantized-int4-tile-packed) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, non-quantized) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-tile-packed) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-weight-only) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-tile-packed) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (openai, whisper-large-v3-turbo, quantized-int4-tile-packed) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (openai, whisper-small, non-quantized) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (Qwen, Qwen3-0.6B, non-quantized) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (Qwen, Qwen3-0.6B, quantized-int4-tile-packed) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• Test CUDA Builds / export-model-cuda-artifact (SocialLocalMobile, Qwen3.5-35B-A3B-HQQ-INT4, quantized-int4-tile-packed) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test CUDA Builds / unittest-cuda / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  backends/cuda/tests/test_output_stride_rearrange.py::TestOutputStrideRearrange::test_slow_path_sparse_bool_mask
• Test Vulkan Backend (specialized runners) / test-vulkan-nvidia / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t ac2520de2702d8793079e8465dd97c3ed6159d1e5e0da6a304b05a835b273d3b /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  extension/llm/modules/test/test_turboquant_kv_cache.py::TestCodebook::test_rotation_varies_with_seed
• pull / unittest-nxp-neutron / linux-job (gh) (trunk failure)
  test_concat_cluster_quantization__conv_7_sigmoid
• Test RISC-V Backend / test-riscv (add, false, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (add, true, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (add, true, true) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (llama2, false, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (llama2, true, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (llama2, true, true) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (mobilebert, false, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (mobilebert, true, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (mobilebert, true, true) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (mv2, false, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (mv2, true, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (mv2, true, true) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (resnet18, false, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (resnet18, true, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (resnet18, true, true) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (yolo26, false, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test RISC-V Backend / test-riscv (yolo26, true, false) / run / linux-job (gh) (trunk failure)
  pytorch_tokenizers
• Test WebGPU Native (Dawn) / test-webgpu-native / linux-job (gh) (trunk failure)
  [ FAILED ] WebGPUNative.RopeHfDynamicSequenceReusedGraph
• trunk / test-arm-backend-ethos-u (test_deit_e2e_ethos_u) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  extension/llm/modules/test/test_turboquant_kv_cache.py::TestCodebook::test_rotation_varies_with_seed

This comment was automatically generated by Dr. CI and updates every 15 minutes.