From ab9bc1ea1b05ef8dc1141c492f08a6938eb88114 Mon Sep 17 00:00:00 2001 From: shoumikhin Date: Mon, 10 Aug 2026 19:56:25 -0700 Subject: [PATCH] Ship the quantized kernels as their own library A quantized model uses smaller numbers than a normal one, so the tensors take less memory. Running one needs the quantized operator kernels. The only copy the wheel shipped is the one torch loads to export a model, which a C++ application cannot use. Such an application links the runtime, loads a quantized model, and the model fails at run time with a missing operator, which looks like a model problem rather than a packaging one. Build the quantized kernels as their own shared library and name it as a CMake component, the same way the other kernel sets are named. ```cmake find_package(executorch REQUIRED COMPONENTS kernels_quantized) target_link_libraries(my_app PRIVATE executorch::runtime executorch::kernels_quantized) ``` The wheel now ships `lib/libexecutorch_kernels_quantized.so`. Note that the wheel also ships a second copy of these kernels, inside the library torch loads when you export a model. That copy is built into the plugin rather than resolved from the shared library, so a process holding both registers the same operators twice, and the runtime treats that as fatal: ``` Re-registering quantized_decomposed::add.out ``` This affects only a process that does both, for example an application that embeds a Python interpreter. A plain C++ application can link the component freely. Because of that, this is the one component `EXECUTORCH_LIBRARIES` does not include, so an application that links whatever the package offers cannot end up in that position without asking. A consumer that wants the quantized kernels names the component, or on CMake older than 3.28, where no component targets exist, links `EXECUTORCH_QUANTIZED_KERNELS_LIBRARY` as well. That variable is now populated on both CMake routes, so a consumer that adopts the older-CMake recipe and later upgrades keeps the library on their link line instead of silently losing it. Built the wheel, installed it into a clean environment, and: - exported a quantized model and ran it from Python, matching eager PyTorch to within the quantization step (measured worst difference 0.0048 against a tolerance of 0.02). - built a C++ application that links `executorch::kernels_quantized`, ran the same program, and got the same output as Python, byte for byte. - confirmed the Python extension does not depend on the run-time copy, and that a process holding the shipped library and the export plugin aborts in either load order. - checked every shipped library the same way, to establish that this is the only pair that collides: the CPU kernels, the delegate, the thread pool, the profiler and the runtime all coexist with both the extension and the export plugin. - an application linking only `EXECUTORCH_LIBRARIES` does not depend on the quantized library while still depending on the CPU kernels, on CMake 3.28 and on real CMake 3.24. A new check asserts this, and it fails on the previous behaviour. - `EXECUTORCH_QUANTIZED_KERNELS_LIBRARY` resolves to the shipped library on both the modern-CMake route (as the imported target) and the pre-3.28 route (as a file path). - a missing quantized library now fails the checks instead of skipping them. The preset that builds the wheel enables these kernels unconditionally, so their absence is a regression rather than a configuration to tolerate, and both the ownership table and the C++ check previously treated it as an acceptable state and reported coverage they had not run. Ran on Linux x86_64 and aarch64. ghstack-source-id: 1579b2423c8f6a25b34ee796ae935033e5bdf4b8 ghstack-comment-id: 5217087046 Pull-Request: https://github.com/pytorch/executorch/pull/21642 --- .ci/scripts/wheel/test_cpp_sdk.py | 169 ++++++++++++++++++++- .ci/scripts/wheel/test_shared_libraries.py | 109 ++++++++++++- docs/source/using-executorch-cpp.md | 27 +++- kernels/quantized/CMakeLists.txt | 21 ++- setup.py | 23 +++ tools/cmake/executorch-wheel-config.cmake | 116 +++++++++++--- 6 files changed, 429 insertions(+), 36 deletions(-) diff --git a/.ci/scripts/wheel/test_cpp_sdk.py b/.ci/scripts/wheel/test_cpp_sdk.py index 68fd067411b..97b6266b134 100644 --- a/.ci/scripts/wheel/test_cpp_sdk.py +++ b/.ci/scripts/wheel/test_cpp_sdk.py @@ -57,6 +57,25 @@ def forward(self, x, image): with torch.no_grad(): expected = model(*example) +if mode == "quantized": + # Quantize with the same flow the documentation shows, so the exported program + # references the quantized operator set rather than the plain one. + # Importing this loads the ahead-of-time library, which is what registers the out + # variants of the quantized operators with torch. Without it the export fails with + # "Missing out variants: quantized_decomposed::quantize_per_tensor", because the + # lowering step has no out variant to select. + import executorch.kernels.quantized # noqa: F401 + from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import ( + get_symmetric_quantization_config, + XNNPACKQuantizer, + ) + from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e, prepare_pt2e + + quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config()) + prepared = prepare_pt2e(torch.export.export(model, example).module(), quantizer) + prepared(*example) + model = convert_pt2e(prepared) + partitioners = [] if mode == "delegate": from executorch.backends.xnnpack.partition.xnnpack_partitioner import ( @@ -85,6 +104,11 @@ def forward(self, x, image): "expected": expected.flatten().tolist(), "delegated": mode == "delegate", "has_xnnpack": b"XnnpackBackend" in bytes(buffer), + # Whether the program actually carries quantized operators. The numeric comparison alone + # cannot tell: an unquantized export of the same model produces a closer match than the + # tolerance a quantized one needs, so it would pass while proving nothing about the + # quantized kernels. + "has_quantized": b"quantized_decomposed" in bytes(buffer), } ) ) @@ -102,6 +126,7 @@ def forward(self, x, image): #include #include +#include #include #include #include @@ -196,8 +221,14 @@ def forward(self, x, image): } worst = std::fmax(worst, diff); } - if (worst > 1e-4) { - std::printf("output differs from eager PyTorch by %g\n", worst); + // Passed in rather than fixed, because the acceptable difference depends on the + // model. A float32 model should match to within rounding, while an int8 quantized one + // legitimately differs by about one quantization step, and using the looser number + // for both would stop the float path catching a real regression. + const double tolerance = argc > 7 ? std::atof(argv[7]) : 1e-4; + if (worst > tolerance) { + std::printf( + "output differs from eager PyTorch by %g, tolerance %g\n", worst, tolerance); return 1; } @@ -335,8 +366,16 @@ def _build_consumer(work_dir: Path, name: str, components) -> Path: return consumer -def _run_consumer(consumer: Path, model: Path, reference, work_dir: Path) -> str: - """Run the application and require it to match eager PyTorch.""" +def _run_consumer( + consumer: Path, model: Path, reference, work_dir: Path, tolerance: float = 1e-4 +) -> str: + """Run the application and require it to match eager PyTorch within `tolerance`. + + The tolerance is a parameter because the acceptable difference depends on the model. + A float32 model should match to within rounding, while an int8 quantized one + legitimately differs by about one quantization step, and using the looser number for + both would stop the float path catching a real regression. + """ inputs = reference["inputs"] shape_a, data_a = _write_tensor(work_dir, "a", inputs[0]) shape_b, data_b = _write_tensor(work_dir, "b", inputs[1]) @@ -358,6 +397,7 @@ def _run_consumer(consumer: Path, model: Path, reference, work_dir: Path) -> str str(shape_b), str(data_b), str(expected), + str(tolerance), ], capture_output=True, text=True, @@ -1212,6 +1252,125 @@ def test_pre_3_28_route_builds_a_consumer_through_variables(work_dir: Path) -> N ) +def test_quantized_kernels_component_runs_a_model(work_dir: Path) -> None: + """A C++ application can run a quantized model using the shipped quantized kernels. + + Before the quantized kernels became their own library they existed only inside the + ahead-of-time extension beside the Python bindings, so a C++ application loading a + quantized program had nothing to link and failed at run time with the operators + reported missing. + + A missing library is a failure rather than a skip. The preset that builds the wheel + always enables the quantized kernels, so their absence is a regression in packaging + or in the build, not a configuration this suite has to tolerate. Skipping there + reported the whole check as coverage while running none of it. + """ + package_dir = _installed_package_dir() + # Globbed for the same reason the profiler check is: the library carries a version suffix outside a + # wheel build, and an exact name would skip this silently there rather than running it. + shipped = sorted((package_dir / "lib").glob("libexecutorch_kernels_quantized.so*")) + assert shipped, ( + "the wheel ships no quantized kernels library. The preset that builds it enables " + "them unconditionally, so this is a packaging or build regression rather than an " + "unsupported configuration." + ) + + model, reference = _export(work_dir, "quantized") + # The export has to have produced a quantized program, or the rest of this proves nothing about the + # quantized kernels. The numeric comparison cannot tell the difference: an unquantized export of the + # same model lands well inside the tolerance a quantized one needs, so it would pass while linking a + # library it never exercised. + assert reference["has_quantized"], ( + "the quantized export produced a program with no quantized operators, so this check would " + "prove nothing about the quantized kernels" + ) + consumer = _build_consumer( + work_dir, + "with-quantized", + ["runtime", "kernels_optimized", "kernels_quantized"], + ) + # One int8 quantization step over this model's output range is about 5e-3, so a + # float32 tolerance cannot be met by a correct quantized run. + output = _run_consumer(consumer, model, reference, work_dir, tolerance=2e-2) + print( + f"✓ a C++ app linking executorch::kernels_quantized runs a quantized model " + f"({output})" + ) + + +def test_aggregate_variable_excludes_the_quantized_kernels(work_dir: Path) -> None: + """`${EXECUTORCH_LIBRARIES}` must not drag in the quantized kernels. + + The export-time plugin that `executorch.kernels.quantized` loads carries its own + copy of those kernels rather than depending on the shipped library, so a process + holding both registers the same operators twice and the runtime stops on the + second one. An application that links whatever the package offers by default + would inherit that, so the component is defined but held out of the aggregate and + a consumer that wants it names it. + + Checked by reading the link line rather than by running, because the failure is a + process-wide abort that needs a Python interpreter in the same process to trigger. + What this owns is the packaging decision: is the library on the link line at all. + """ + package_dir = _installed_package_dir() + # Fatal for the same reason the check above is: the preset that builds the wheel + # always enables these kernels, so their absence is a regression rather than a + # configuration to tolerate, and skipping would report this as coverage. + assert sorted( + (package_dir / "lib").glob("libexecutorch_kernels_quantized.so*") + ), "the wheel ships no quantized kernels library, so this check cannot run" + + source_dir = work_dir / "aggregate-only" + source_dir.mkdir(parents=True, exist_ok=True) + (source_dir / "consumer.cpp").write_text(_CONSUMER_SOURCE) + # No COMPONENTS and no named target, which is the shape the older-CMake route + # forces and the documentation offers as the general case. + (source_dir / "CMakeLists.txt").write_text( + "cmake_minimum_required(VERSION 3.28)\n" + "project(consumer CXX)\n" + "find_package(executorch REQUIRED)\n" + "add_executable(consumer consumer.cpp)\n" + "target_link_libraries(consumer PRIVATE ${EXECUTORCH_LIBRARIES})\n" + ) + build_dir = work_dir / "aggregate-only-build" + config = package_dir / "share" / "cmake" / "executorch-config.cmake" + for command in ( + [ + _tool("cmake"), + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_PREFIX_PATH={config.parent}", + ], + [_tool("cmake"), "--build", str(build_dir)], + ): + result = subprocess.run(command, capture_output=True, text=True, check=False) + assert result.returncode == 0, ( + "an application linking only ${EXECUTORCH_LIBRARIES} could not be built:\n" + f"{result.stdout[-2000:]}\n{result.stderr[-2000:]}" + ) + + consumer = build_dir / "consumer" + dependencies = subprocess.run( + ["readelf", "-d", str(consumer)], capture_output=True, text=True, check=True + ).stdout + assert "libexecutorch_kernels_quantized" not in dependencies, ( + "an application that linked only ${EXECUTORCH_LIBRARIES} depends on the " + "quantized kernels. That library collides with the export-time plugin, so it " + "has to be opted into by name rather than handed to every consumer." + ) + # The rest of the aggregate still has to be there, or this would pass by shipping + # nothing at all. + assert "libexecutorch_kernels_optimized" in dependencies, ( + "the aggregate no longer carries the CPU kernels, so an application linking it " + "would fail at run time with the operators reported missing" + ) + print( + "✓ ${EXECUTORCH_LIBRARIES} carries the CPU kernels and not the quantized ones" + ) + + def run_tests(work_dir: Path) -> None: test_find_package_honours_a_version_request(work_dir) test_profiler_component_is_usable(work_dir) @@ -1221,6 +1380,8 @@ def run_tests(work_dir: Path) -> None: test_runtime_alone_links_but_cannot_compute(work_dir) test_kernels_component_runs_a_model(work_dir) test_pre_3_28_route_builds_a_consumer_through_variables(work_dir) + test_quantized_kernels_component_runs_a_model(work_dir) + test_aggregate_variable_excludes_the_quantized_kernels(work_dir) test_delegated_model_needs_the_delegate_component(work_dir) test_consumer_is_relocatable(work_dir) test_one_registry_in_the_cpp_process(work_dir) diff --git a/.ci/scripts/wheel/test_shared_libraries.py b/.ci/scripts/wheel/test_shared_libraries.py index fa6e8681345..1589b30e3f2 100644 --- a/.ci/scripts/wheel/test_shared_libraries.py +++ b/.ci/scripts/wheel/test_shared_libraries.py @@ -52,6 +52,14 @@ # the operators are registered twice, which aborts at startup. _KERNEL_SYMBOLS = ("torch::executor::native::abs_out",) +# The quantized kernels, whose own library the wheel ships when they are built. +# A separate group because they have a separate owner, and because a wheel built +# without them ships neither the library nor these symbols. +_QUANTIZED_KERNEL_SYMBOLS = ( + "torch::executor::native::quantize_per_tensor_out", + "torch::executor::native::dequantize_per_tensor_out", +) + # The registry entry points, kept separate from the kernel implementations above. # A library that carries its own copy of these has its own registration code, which # is what this split is meant to prevent: one owner of the operator table. Checking @@ -280,7 +288,37 @@ def _defines_symbol(library: Path, symbol: str) -> bool: return False -def _assert_single_definer(symbols, what: str, owner: str | None = None) -> None: +def _is_export_only(library: Path) -> bool: + """Whether a library exists to export a model rather than to run one. + + The ahead-of-time operator libraries register kernels into torch so a model can be + exported, and they link torch to do it. They deliberately carry their own copy of + the kernels, because the copy a C++ application links is registered into a table + those libraries never read. + + Named by the caller per component rather than excluded everywhere. Counting them for + the component they duplicate would report a duplicate that is not one, and excusing + them for every component would stop this catching a second registry hiding inside + one of them. + + Matched on both the torch dependency and the name marker, because either alone + misfires: several shipped libraries link torch without being export-side, and a + name check alone would accept a runtime library that adopted the suffix. + """ + if _tool("readelf") is None: + return library.name.endswith("_aot_lib.so") + dynamic = subprocess.run( + [_tool("readelf"), "-d", str(library)], + capture_output=True, + text=True, + check=False, + ).stdout + return "libtorch.so" in dynamic and "_aot_lib" in library.name + + +def _assert_single_definer( + symbols, what: str, owner: str | None = None, allow_export_copy: bool = False +) -> None: """At most one shipped library may define each of `symbols`. The owner is named where one is expected, because counting definers alone does @@ -291,11 +329,25 @@ def _assert_single_definer(symbols, what: str, owner: str | None = None) -> None A component the wheel does not ship at all is a valid configuration, not a fault. Delegates and kernel sets are build options, so a wheel built without one has zero definers and is reported as such. What must never happen is two. + + `allow_export_copy` excuses the export-side libraries for one component only. The + quantized kernels genuinely exist twice, once in the runtime library and once in the + library torch loads at export time, because each side registers into a table the + other never reads. Loading both into one process does abort on the second + registration, so what this check enforces for that component is one owner among the + runtime libraries, not the absence of the export copy. Excusing every component + would disarm the check where duplication is a real fault: two of these libraries + defined the backend registry symbols in one released wheel and not in the release + before it, so the duplication this catches does happen. """ assert _tool("nm") is not None, "nm is required to inspect the wheel" package_dir = _installed_package_dir() - libraries = _shipped_shared_objects(package_dir) + libraries = [ + library + for library in _shipped_shared_objects(package_dir) + if not (allow_export_copy and _is_export_only(library)) + ] assert libraries, f"no shared libraries found under {package_dir}" # Every symbol is resolved before anything is reported, so a component that is only @@ -368,6 +420,12 @@ def _assert_single_definer(symbols, what: str, owner: str | None = None) -> None "libexecutorch_kernels_optimized.so", False, ), + ( + "set of quantized kernels", + _QUANTIZED_KERNEL_SYMBOLS, + "libexecutorch_kernels_quantized.so", + True, + ), # The third-party code these libraries bundle, checked separately from the # wrappers above. A wrapper can have a single owner while the implementation # underneath it is bundled into two of these, which is two real thread pools or @@ -394,6 +452,15 @@ def _assert_single_definer(symbols, what: str, owner: str | None = None) -> None ) +# The one component that legitimately exists twice. The quantized kernels are compiled into the runtime +# library and again into the library torch loads at export time, because each side registers into a +# table the other never reads, so a second definer there is expected rather than a fault. A process +# that loads both does abort on the second registration, which is why this is named per component and +# the check stays armed for the other ten, where a second definer means two registries or two thread +# pools in one process. +_COMPONENTS_WITH_AN_EXPORT_COPY = frozenset({"set of quantized kernels"}) + + def test_each_component_has_one_owner() -> None: """No component may be defined by more than one library the wheel ships. @@ -411,7 +478,12 @@ def test_each_component_has_one_owner() -> None: f"the wheel ships no {owner}, which owns the {what}. Either packaging " "dropped it or the build did not produce it." ) - _assert_single_definer(symbols, what, owner if present else None) + _assert_single_definer( + symbols, + what, + owner if present else None, + allow_export_copy=what in _COMPONENTS_WITH_AN_EXPORT_COPY, + ) def test_python_extensions_import() -> None: @@ -1212,17 +1284,43 @@ def test_extension_contains_no_component() -> None: ).stdout.splitlines() if "NEEDED" in line } + # Only the libraries whose code the extension used to contain. Those are the ones + # this split moved out of it, so the extension must now resolve them from outside or + # a retention option silently failed. + # + # Not every shipped library serves Python. The quantized kernels and the CUDA + # delegate exist for a C++ application: Python registers quantized operators through + # the torch-linked ahead-of-time library at export time, and never loads the CUDA + # delegate from this extension at all. Requiring a dependency on those would demand + # the extension link code it has no use for. shipped = {path.name for path in _shipped_runtime_libraries(package_dir)} assert shipped, ( f"the wheel installed no runtime libraries under {package_dir / 'lib'}, so this check would " "compare the extension against nothing and pass" ) - unused = sorted(shipped - needed) + expected = { + name + for name in shipped + if not any( + marker in name + for marker in ("kernels_quantized", "backend_cuda", "extension_cuda") + ) + } + unused = sorted(expected - needed) assert not unused, ( f"the wheel ships {unused} but {extension.name} does not depend on them, so " "either they are dead weight or a retention option did not hold" ) + # Two shipped libraries register the same quantized operators, one for export and one for a C++ + # application, and the runtime treats a repeat registration as fatal. Reaching both from one process + # aborts it, and the only thing preventing that is this extension not depending on the run-time one. + assert not any("kernels_quantized" in name for name in needed), ( + f"{extension.name} depends on the run-time quantized library, which registers the same operators " + "as the export-time one it already loads. The runtime aborts on a repeat registration, so " + "importing this extension would kill the process." + ) + # Positive proof that the extension resolves these from elsewhere, rather than # only the absence of a visible definition. A hidden or local copy would not # appear in the dynamic symbol table at all, so "defines nothing" on its own is @@ -1248,7 +1346,7 @@ def test_extension_contains_no_component() -> None: print( f"✓ {extension.name} ({extension.stat().st_size // 1024} KiB) contains no " f"component, imports the runtime symbols it uses, and depends on all " - f"{len(shipped)} shipped libraries" + f"{len(expected)} shipped libraries it used to contain" ) @@ -1292,6 +1390,7 @@ def test_shipped_library_names_are_expected() -> None: known = ( "libexecutorch", "libexecutorch_kernels_optimized", + "libexecutorch_kernels_quantized", "libexecutorch_backend_xnnpack", "libexecutorch_threadpool", "libexecutorch_etdump", diff --git a/docs/source/using-executorch-cpp.md b/docs/source/using-executorch-cpp.md index 40c8e1eef96..e50148a8075 100644 --- a/docs/source/using-executorch-cpp.md +++ b/docs/source/using-executorch-cpp.md @@ -108,6 +108,7 @@ reported while CMake configures, rather than failing later at link time. | --- | --- | | `executorch::runtime` | the program loader and executor. Always present. | | `executorch::kernels_optimized` | CPU operator kernels. Needed for any operator a delegate does not claim. | +| `executorch::kernels_quantized` | quantized operator kernels, for a quantized model. Link it only when you need it: see the note below. | | `executorch::backend_xnnpack` | the XNNPACK delegate. | | `executorch::threadpool` | the shared thread pool. | | `executorch::etdump` | the profiler. | @@ -117,6 +118,22 @@ kernels a model computes with, so a model that is not fully delegated needs a ke component too. Linking a delegate is what registers it: a program delegated to XNNPACK fails to load in an application that did not link `executorch::backend_xnnpack`. +#### The quantized kernels are opt in + +`executorch::kernels_quantized` is the one component that `${EXECUTORCH_LIBRARIES}` does +not include, so you have to name it. The reason is a conflict with the Python side: +`executorch.kernels.quantized` loads a plugin that carries its own copy of the same +kernels, and the runtime stops when the same operator is registered twice: + +``` +Re-registering quantized_decomposed::add.out +``` + +That only affects a process holding both, for example an application that embeds a +Python interpreter. A plain C++ application can link this component freely. It is kept +out of the default set so that linking whatever the package offers cannot put you in that +position by accident. + To require a minimum version, pass it to `find_package`: ```cmake @@ -149,8 +166,14 @@ set_property(TARGET my_app PROPERTY CXX_STANDARD_REQUIRED ON) ``` `EXECUTORCH_LIBRARIES` names the runtime and every component the wheel shipped, so you -cannot choose components on this route. Upgrade to CMake 3.28 and link the specific -targets you need instead. +cannot choose components on this route. The quantized kernels are the exception described +above, offered as `EXECUTORCH_QUANTIZED_KERNELS_LIBRARY` for a consumer that wants them: + +```cmake +target_link_libraries(my_app PRIVATE ${EXECUTORCH_QUANTIZED_KERNELS_LIBRARY}) +``` + +Upgrade to CMake 3.28 and link the specific targets you need instead. ### Building from source diff --git a/kernels/quantized/CMakeLists.txt b/kernels/quantized/CMakeLists.txt index 9778220722a..c36304d4fa0 100644 --- a/kernels/quantized/CMakeLists.txt +++ b/kernels/quantized/CMakeLists.txt @@ -157,14 +157,14 @@ if(NOT CMAKE_GENERATOR STREQUAL "Xcode" endif() add_library(quantized_kernels ${_quantized_kernels__srcs}) -# The thread pool carries the define that switches parallel_for from a serial -# fallback to the real threaded implementation, so without it quantize, -# dequantize and choose_qparams run on one core. Guarded because a bare metal -# target builds these kernels without a thread pool at all, where the serial -# fallback is the only correct choice. target_link_libraries( quantized_kernels PRIVATE executorch_core kernels_util_all_deps ) +# The thread pool carries the define that switches parallel_for from a serial +# fallback to the real threaded implementation. Without it choose_qparams runs +# on one core, as does the ARM path in quantize. Guarded because a bare metal +# target builds these kernels without a thread pool at all, where the serial +# fallback is the only correct choice. if(TARGET extension_threadpool) target_link_libraries(quantized_kernels PRIVATE extension_threadpool) endif() @@ -190,4 +190,15 @@ if(EXECUTORCH_BUILD_SHARED) executorch_quantized_ops quantized_ops_lib quantized_kernels executorch_shared ) + # Named after what the library provides rather than after the target that + # produces it, matching the optimized kernels next to it, so the shipped file + # reads as libexecutorch_kernels_quantized.so. The target name stays as it is + # because a source build already refers to it. + set_target_properties( + executorch_quantized_ops PROPERTIES OUTPUT_NAME + executorch_kernels_quantized + ) + # Ships beside libexecutorch.so in the wheel's lib/ directory, so it resolves + # the runtime from there rather than from wherever it was built. + executorch_target_shipped_runtime_path(executorch_quantized_ops) endif() diff --git a/setup.py b/setup.py index d7be291fd5e..acd156a4c41 100644 --- a/setup.py +++ b/setup.py @@ -1297,6 +1297,16 @@ def run(self): # noqa C901 if cmake_cache.is_enabled("EXECUTORCH_BUILD_MLX"): cmake_build_args += ["--target", "mlxdelegate"] + # Named explicitly because nothing else links it. The other shipped + # libraries are built as dependencies of the Python extension, but a C++ + # application is the only consumer of this one, so without naming it the + # target is generated and never built, and packaging then looks for a file + # that does not exist. + if cmake_cache.is_enabled("EXECUTORCH_BUILD_SHARED") and ( + cmake_cache.is_enabled("EXECUTORCH_BUILD_KERNELS_QUANTIZED") + ): + cmake_build_args += ["--target", "executorch_quantized_ops"] + if cmake_cache.is_enabled("EXECUTORCH_BUILD_KERNELS_LLM_AOT"): cmake_build_args += ["--target", "custom_ops_aot_lib"] cmake_build_args += ["--target", "quantized_ops_aot_lib"] @@ -1414,6 +1424,19 @@ def run(self): # noqa C901 "EXECUTORCH_BUILD_KERNELS_OPTIMIZED", ], ), + # The quantized kernels, as their own library rather than code + # fused into the AOT-only extension beside the Python bindings. + # A C++ application running a quantized model could not link + # them before. + BuiltFile( + src_dir="%CMAKE_CACHE_DIR%/kernels/quantized/", + src_name="libexecutorch_kernels_quantized.so", + dst="executorch/lib/libexecutorch_kernels_quantized.so", + dependent_cmake_flags=[ + "EXECUTORCH_BUILD_SHARED", + "EXECUTORCH_BUILD_KERNELS_QUANTIZED", + ], + ), # Install the XNNPACK delegate beside them, so a process has one # copy of it instead of one per component that uses it. BuiltFile( diff --git a/tools/cmake/executorch-wheel-config.cmake b/tools/cmake/executorch-wheel-config.cmake index eedda35a8d2..dade1a91c9d 100644 --- a/tools/cmake/executorch-wheel-config.cmake +++ b/tools/cmake/executorch-wheel-config.cmake @@ -67,10 +67,21 @@ # dependency and, for a registration-only library, the link options that keep it # from being dropped. The names, when present, are: # -# executorch::kernels_optimized -- The CPU operator kernels. Needed to run a -# model. executorch::backend_xnnpack -- The XNNPACK delegate. -# executorch::threadpool -- The shared thread pool. executorch::etdump -- -# The profiler. +# ~~~ +# executorch::kernels_optimized The CPU operator kernels. Needed to run a model. +# executorch::kernels_quantized The quantized operator kernels, for a quantized +# model. Not part of EXECUTORCH_LIBRARIES, see +# below. +# executorch::backend_xnnpack The XNNPACK delegate. +# executorch::threadpool The shared thread pool. +# executorch::etdump The profiler. +# ~~~ +# +# EXECUTORCH_LIBRARIES carries every component except the quantized kernels, +# which a consumer names explicitly instead. The export-time plugin that +# executorch.kernels.quantized loads carries its own copy of those kernels, so a +# process holding both stops on a repeated operator registration, and a consumer +# linking the aggregate would inherit that without asking for it. # # Check with if(TARGET executorch::) rather than assuming one exists. A # namespaced name that was never defined is a configure-time error that names @@ -274,6 +285,13 @@ if(_executorch_runtime_library AND NOT _executorch_targets_supported) # then a load failure saying the backend is not registered, which reads as a # model problem. Anything the wheel did not ship is simply not found and # skipped. + # + # The quantized kernels are deliberately absent, for the reason given at their + # component definition below: they collide with the export-time plugin that + # executorch.kernels.quantized loads, and a process holding both dies. This + # route has no per-component target to opt into, so they are offered through + # EXECUTORCH_QUANTIZED_KERNELS_LIBRARY instead and a consumer that wants them + # links that as well. foreach(_executorch_component IN ITEMS libexecutorch_kernels_optimized libexecutorch_backend_xnnpack libexecutorch_threadpool libexecutorch_etdump @@ -308,6 +326,21 @@ if(_executorch_runtime_library AND NOT _executorch_targets_supported) endif() endforeach() unset(_executorch_component_library) + # Held out of the aggregate above, so name it separately. A consumer that + # wants quantized operators and does not load the Python plugin in the same + # process links this too. Empty when the wheel shipped no such library. + _executorch_find_library( + EXECUTORCH_QUANTIZED_KERNELS_LIBRARY libexecutorch_kernels_quantized + ) + if(EXECUTORCH_QUANTIZED_KERNELS_LIBRARY AND CMAKE_SYSTEM_NAME STREQUAL + "Linux" + ) + # The same scoped retention the aggregate entries get, for the same reason: + # a registration-only library exports nothing the application references. + set(EXECUTORCH_QUANTIZED_KERNELS_LIBRARY + "-Wl,--push-state,--no-as-needed,${EXECUTORCH_QUANTIZED_KERNELS_LIBRARY},--pop-state" + ) + endif() message( STATUS "executorch: the prebuilt runtime is present but its imported targets need CMake 3.28 or " @@ -404,8 +437,13 @@ endif() # most: a registration-only library has no symbol the application references, so # a normal link drops it and its registration never runs. # -# Call as: _executorch_define_component( ) +# Call as: _executorch_define_component( +# [OPT_IN]) +# +# OPT_IN defines the target but keeps it out of EXECUTORCH_LIBRARIES, for a +# library a consumer has to choose deliberately rather than receive by default. function(_executorch_define_component _suffix _library_name) + cmake_parse_arguments(PARSE_ARGV 2 _component "OPT_IN" "" "") # Same reason the runtime target is skipped on older CMake: a component target # exports an $ORIGIN-relative search path, and a version that writes it wrong # produces a target that works in place and fails once deployed. @@ -428,10 +466,12 @@ function(_executorch_define_component _suffix _library_name) # returning here would hand the second caller a list with the runtime but # none of the components. A consumer linking that variable would then be # missing its kernels and fail at load with an unregistered operator. - set(EXECUTORCH_LIBRARIES - ${EXECUTORCH_LIBRARIES} ${_target} - PARENT_SCOPE - ) + if(NOT _component_OPT_IN) + set(EXECUTORCH_LIBRARIES + ${EXECUTORCH_LIBRARIES} ${_target} + PARENT_SCOPE + ) + endif() return() endif() add_library(${_target} SHARED IMPORTED) @@ -477,10 +517,12 @@ function(_executorch_define_component _suffix _library_name) "LINKER:--push-state,--no-as-needed,${_library},--pop-state" ) endif() - set(EXECUTORCH_LIBRARIES - ${EXECUTORCH_LIBRARIES} ${_target} - PARENT_SCOPE - ) + if(NOT _component_OPT_IN) + set(EXECUTORCH_LIBRARIES + ${EXECUTORCH_LIBRARIES} ${_target} + PARENT_SCOPE + ) + endif() endfunction() _executorch_define_component(threadpool executorch_threadpool) @@ -489,6 +531,25 @@ _executorch_define_component(threadpool executorch_threadpool) # checks, so it has to be defined here or a consumer following the documentation # gets a bare name that CMake hands to the linker as a literal flag. _executorch_define_component(kernels_optimized executorch_kernels_optimized) +# The quantized kernels, optional in the same way: a wheel built without them +# simply has no such library and the component is not defined. +# +# Opt in rather than part of the aggregate. The export-time plugin that +# executorch.kernels.quantized loads registers the same operator names, and the +# runtime stops on a repeat registration rather than choosing one, so a process +# holding both dies. Measured: linking this library and importing that module in +# either order aborts with "Re-registering quantized_decomposed::add.out". None +# of the other shipped components collide this way, so only this one is held +# back, and a consumer that wants it names it. +_executorch_define_component( + kernels_quantized executorch_kernels_quantized OPT_IN +) +# The same library exposed through a variable, so a consumer that follows the +# pre-3.28 recipe and later upgrades past 3.28 keeps working. Left empty when +# the wheel shipped no such library, matching the pre-3.28 branch above. +if(TARGET executorch::kernels_quantized) + set(EXECUTORCH_QUANTIZED_KERNELS_LIBRARY executorch::kernels_quantized) +endif() # The profiler. A C++ application could not record timing data from an installed # package before, because the implementation shipped only inside the Python # extension. @@ -715,13 +776,28 @@ foreach(_component ${executorch_FIND_COMPONENTS}) # One string rather than several arguments. Several make a list, and # message() joins a list with semicolons, which lands separators mid # sentence. - string( - CONCAT - executorch_NOT_FOUND_MESSAGE - "the required component '${_component}' needs CMake 3.28 or newer, because older " - "versions write the \$ORIGIN token in a runtime search path incorrectly; this " - "package is otherwise usable through EXECUTORCH_LIBRARIES" - ) + # + # The quantized kernels are held out of EXECUTORCH_LIBRARIES on purpose, + # so a consumer who wants them names + # EXECUTORCH_QUANTIZED_KERNELS_LIBRARY instead. See the OPT_IN comment + # at the component definition above. + if(_component STREQUAL "kernels_quantized") + string( + CONCAT + executorch_NOT_FOUND_MESSAGE + "the required component '${_component}' needs CMake 3.28 or newer, because " + "older versions write the \$ORIGIN token in a runtime search path incorrectly; " + "this package is otherwise usable through EXECUTORCH_QUANTIZED_KERNELS_LIBRARY" + ) + else() + string( + CONCAT + executorch_NOT_FOUND_MESSAGE + "the required component '${_component}' needs CMake 3.28 or newer, because older " + "versions write the \$ORIGIN token in a runtime search path incorrectly; this " + "package is otherwise usable through EXECUTORCH_LIBRARIES" + ) + endif() else() # One string rather than several arguments. Several make a list, and # message() joins a list with semicolons, which lands separators mid