-
Notifications
You must be signed in to change notification settings - Fork 969
Opaque PyObject ABI support #5807
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
eb53c2d
e41b509
91becaf
1020688
3afa9ae
f8d6cae
a590874
a96219f
e7ac9c0
d981be7
55b6acd
733aa82
c43061e
3812a64
25a65a6
4a83024
9d0e2ed
a78b5df
42a73e1
060c3ca
c1bd2c7
ba8b09a
f15a7fc
3fa17d0
1970421
57c2045
f80849e
c0805a9
719cef5
072ef0a
0b743b6
264307f
ed48e75
b4138c4
f3ee6af
ba47f50
e52cb59
3cf60b1
607e4b0
449eb8a
5371fd8
19d78d2
14bf4d1
91a0419
285e79c
9f23720
d38c274
e2d8f8c
15ff21c
930e1a9
a609f8c
dd5b3a9
7419b86
603aaf9
c9b9157
7560348
becc8af
b734e77
a6c8710
a827be9
0ceda48
d1ef669
26c7dc0
95ca7d2
b596885
22a5332
ec1269a
04eb8bb
101949c
828afdd
cd78153
5e5671e
266527f
294a4f0
b37445d
a2584f5
1b5ec7a
553ef54
2f755ae
f326c74
710e835
861edd8
c8e40b9
45d5cc1
e3dc133
966a82f
9d9a0e2
f52f1d8
77039d2
0765953
cde688c
b11b262
9cb420a
7fe98ab
41be837
58162a4
48a7d20
c899390
e0c8826
942ecc9
90bef3c
ea2eab7
8354ef7
6854756
3a21665
97087aa
80a591c
be1b77f
090d11d
81daf34
3ce9275
4763179
b6d6f99
4828759
1e2a2c3
ecc365a
14c80db
af92c51
17583a8
1caf685
85bc302
75b8962
cbedefc
ec46a20
63486a3
f4e86f7
ee68ff0
816fd56
8e04584
1e073e7
8fd0acb
7d5881c
5c73549
1e71ed4
b3ef667
df3bf04
fbacc26
0efa792
766ecbf
deb83a7
9174ca7
01a6b0a
b8b5bff
285eb19
95be8d0
9a33592
0c7b53a
505d9e1
20c8f57
afd0eec
b40805a
4dba451
107ab27
5334eaa
92ad56c
12fa458
9337202
f76a5c8
ab1333b
52124f8
d9133fe
87d6ae4
1aea9b0
4bb2202
3e1ec34
781648b
f091c9a
ef4fb92
7ee1ed0
0709d59
f509e72
f8aa9a5
037563a
23bf337
14c2d9f
9af08e1
13448e3
7ba3daf
a59f8dc
4bcd9fa
5a47645
9a32727
c536ebe
53014c0
208fe8f
3b32ea1
d9ade28
dab0337
0e4d6d8
c85c89c
6c514fb
5518b84
0123231
ed9807c
2b8d7a2
b772cf3
c345a0d
caf1af8
2531c96
1908b69
d232ac7
941aca4
4d38642
dd200da
fd0f14f
13e04c7
ebc99b9
6214c3d
7725030
303877a
cb5a464
607f8bc
335fa8a
be8b1a1
68ffca0
f9b50b7
6419422
f85c163
fd01447
0e945b3
c9c3b01
abaf0e0
a49e696
1cf8751
e6d31b9
5502313
3c25970
f49ee61
854062d
7b3da72
91c7db3
afd124c
5d9afb9
dac521d
4be2121
88d34b3
46b1d50
e573a16
fbefb3b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -28,28 +28,57 @@ An example output of doing this is shown below: | |
|
|
||
| ```console | ||
| $ PYO3_PRINT_CONFIG=1 cargo build | ||
| Compiling pyo3 v0.14.1 (/home/david/dev/pyo3) | ||
| error: failed to run custom build command for `pyo3 v0.14.1 (/home/david/dev/pyo3)` | ||
| Compiling pyo3-ffi v0.28.3 (/Users/goldbaum/Documents/pyo3/pyo3-ffi) | ||
| error: failed to run custom build command for `pyo3-ffi v0.28.3 (/Users/goldbaum/Documents/pyo3/pyo3-ffi)` | ||
|
|
||
| Caused by: | ||
| process didn't exit successfully: `/home/david/dev/pyo3/target/debug/build/pyo3-7a8cf4fe22e959b7/build-script-build` (exit status: 101) | ||
| process didn't exit successfully: `/Users/goldbaum/Documents/pyo3/target/debug/build/pyo3-ffi-120b26b17564bf98/build-script-build` (exit status: 101) | ||
| --- stdout | ||
| cargo:rustc-check-cfg=cfg(Py_LIMITED_API) | ||
| cargo:rustc-check-cfg=cfg(Py_GIL_DISABLED) | ||
| cargo:rustc-check-cfg=cfg(PyPy) | ||
| cargo:rustc-check-cfg=cfg(GraalPy) | ||
| cargo:rustc-check-cfg=cfg(RustPython) | ||
| cargo:rustc-check-cfg=cfg(py_sys_config, values("Py_DEBUG", "Py_REF_DEBUG", "Py_TRACE_REFS", "COUNT_ALLOCS")) | ||
| cargo:rustc-check-cfg=cfg(pyo3_disable_reference_pool) | ||
| cargo:rustc-check-cfg=cfg(pyo3_leak_on_drop_without_reference_pool) | ||
|
Comment on lines
+43
to
+44
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Interesting, this is a bit messy! I opened #6067 to move these. |
||
| cargo:rustc-check-cfg=cfg(Py_3_8) | ||
| cargo:rustc-check-cfg=cfg(Py_3_9) | ||
| cargo:rustc-check-cfg=cfg(Py_3_10) | ||
| cargo:rustc-check-cfg=cfg(Py_3_11) | ||
| cargo:rustc-check-cfg=cfg(Py_3_12) | ||
| cargo:rustc-check-cfg=cfg(Py_3_13) | ||
| cargo:rustc-check-cfg=cfg(Py_3_14) | ||
| cargo:rustc-check-cfg=cfg(Py_3_15) | ||
| cargo:rustc-check-cfg=cfg(Py_3_16) | ||
| cargo:rustc-check-cfg=cfg(pyo3_dll, values("python3", "python3_d", "python3t", "python3t_d", "python38", "python38_d", "python39", "python39_d", "python310", "python310_d", "python311", "python311_d", "python312", "python312_d", "python313", "python313_d", "python313t", "python313t_d", "python314", "python314_d", "python314t", "python314t_d", "python315", "python315_d", "python315t", "python315t_d", "python316", "python316_d", "python316t", "python316t_d", "libpypy3.11-c")) | ||
| cargo:rerun-if-env-changed=PYO3_CONFIG_FILE | ||
| cargo:rerun-if-env-changed=PYO3_CROSS | ||
| cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR | ||
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION | ||
| cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION | ||
| cargo:rerun-if-env-changed=PYO3_NO_PYTHON | ||
| cargo:rerun-if-env-changed=PYO3_ENVIRONMENT_SIGNATURE | ||
| cargo:rerun-if-env-changed=PYO3_PYTHON | ||
| cargo:rerun-if-env-changed=VIRTUAL_ENV | ||
| cargo:rerun-if-env-changed=CONDA_PREFIX | ||
| cargo:rerun-if-env-changed=PATH | ||
| cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG | ||
|
|
||
| -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile -- | ||
| implementation=CPython | ||
| version=3.8 | ||
| version=3.14 | ||
| shared=true | ||
| abi3=false | ||
| lib_name=python3.8 | ||
| lib_dir=/usr/lib | ||
| executable=/usr/bin/python | ||
| target_abi=CPython-free_threaded-3.14 | ||
| lib_name=python3.14t | ||
| lib_dir=/Users/goldbaum/.pyenv/versions/3.14.4t/lib | ||
| executable=/Users/goldbaum/.pyenv/versions/3.14.4t/bin/python | ||
| pointer_width=64 | ||
| build_flags= | ||
| build_flags=Py_GIL_DISABLED | ||
| python_framework_prefix= | ||
| suppress_build_script_link_lines=false | ||
|
|
||
| note: unset the PYO3_PRINT_CONFIG environment variable and retry to compile with the above config | ||
| ``` | ||
|
|
||
| The `PYO3_ENVIRONMENT_SIGNATURE` environment variable can be used to trigger rebuilds when its value changes, it has no other effect. | ||
|
|
@@ -74,7 +103,8 @@ The PyO3 ecosystem has two packaging tools, [`maturin`] and [`setuptools-rust`], | |
| PyO3 has some functionality for configuring projects when building Python extension modules: | ||
|
|
||
| - The `PYO3_BUILD_EXTENSION_MODULE` environment variable, which must be set when building Python extension modules. `maturin` and `setuptools-rust` set this automatically. | ||
| - The `abi3` Cargo feature and its version-specific `abi3-pyXY` companions, which are used to opt-in to the limited Python API in order to support multiple Python versions in a single wheel. | ||
| - The `abi3` and `abi3t` Cargo feature and their version-specific `abi3-pyXY` and `abi3t-pyXY` companions, which are used to opt-in to the limited Python API and a flavor of stable ABI in order to support multiple Python versions in a single wheel. | ||
| The free-threaded build of CPython cannot load `abi3` wheels but both builds can load `abi3t` wheels. | ||
|
|
||
| This section describes the packaging tools before describing how to build manually without them. | ||
| It then proceeds with an explanation of the `PYO3_BUILD_EXTENSION_MODULE` environment variable. | ||
|
|
@@ -212,60 +242,74 @@ This should only be set when building a library for distribution. | |
| > | ||
| > Projects are encouraged to migrate off the feature, as it caused [major development pain](faq.md#i-cant-run-cargo-test-or-i-cant-build-in-a-cargo-workspace-im-having-linker-issues-like-symbol-not-found-or-undefined-reference-to-_pyexc_systemerror) due to the lack of linking. | ||
|
|
||
| ### `Py_LIMITED_API`/`abi3` | ||
| ### `Py_LIMITED_API`/`abi3`/`abi3t` | ||
|
|
||
| By default, Python extension modules can only be used with the same Python version they were compiled against. | ||
| For example, an extension module built for Python 3.5 can't be imported in Python 3.8. | ||
| [PEP 384](https://www.python.org/dev/peps/pep-0384/) introduced the idea of the limited Python API, which would have a stable ABI enabling extension modules built with it to be used against multiple Python versions. | ||
| This is also known as `abi3`. | ||
| The ABI defined by PEP 384 is called `abi3`, since it's stable for all Python 3.X releases. | ||
| In 2026, the steering council approved [PEP 803](https://www.python.org/dev/peps/pep-0803/)) which defines a new stable ABI, `abi3t`, that can be used on the free-threaded build of CPython. | ||
| Extensions built using the `abit3` ABI are importable on both the GIL-enabled and free-threaded builds of any CPython version newer than the target version. | ||
| So, `abi3t` extensions built for the Python 3.15 limited API will be importable on Python 3.16, 3.17, and all future Python 3.X versions. | ||
|
|
||
| The advantage of building extension modules using the limited Python API is that package vendors only need to build and distribute a single copy (for each OS / architecture), and users can install it on all Python versions from the [minimum version](#minimum-python-version-for-abi3) and up. | ||
| The advantage of building extension modules using the limited Python API is that package vendors only need to build and distribute a single copy (for each OS / architecture), and users can install it on all Python versions from the [minimum version](#minimum-python-version-for-abi3-and-abi3t-builds) and up. | ||
| The downside of this is that PyO3 can't use optimizations which rely on being compiled against a known exact Python version. | ||
| It's up to you to decide whether this matters for your extension module. | ||
| It's also possible to design your extension module such that you can distribute `abi3` wheels but allow users compiling from source to benefit from additional optimizations - see the [support for multiple python versions](./building-and-distribution/multiple-python-versions.md) section of this guide, in particular the `#[cfg(Py_LIMITED_API)]` flag. | ||
| It's also possible to design your extension module such that you can distribute `abi3` or `abi3t` wheels but allow users compiling from source to benefit from additional optimizations - see the [support for multiple python versions](./building-and-distribution/multiple-python-versions.md) section of this guide, in particular the `#[cfg(Py_LIMITED_API)]` flag. | ||
|
|
||
| There are three steps involved in making use of `abi3` when building Python packages as wheels: | ||
| There are three steps involved in targeting `abi3` or `abi3t` when building Python packages as wheels: | ||
|
|
||
| 1. Enable the `abi3` feature in `pyo3`. | ||
| 1. Enable the `abi3` and/or `abi3t` feature in `pyo3`. | ||
| This ensures `pyo3` only calls Python C-API functions which are part of the stable API, and on Windows also ensures that the project links against the correct shared object (no special behavior is required on other platforms): | ||
|
|
||
| ```toml | ||
| [dependencies] | ||
| pyo3 = { {{#PYO3_CRATE_VERSION}}, features = ["abi3"] } | ||
| pyo3 = { {{#PYO3_CRATE_VERSION}}, features = ["abi3", "abi3t"] } | ||
| ``` | ||
|
|
||
| 2. Ensure that the built shared objects are correctly marked as `abi3`. | ||
| Enabling both the `"abi3"` and `"abi3t"` features will produce an `abi3` extension if targeting Python 3.14 or older and an `abi3t` extension if targeting 3.15 or newer. | ||
| If you always want to produce `abi3t` wheels, you can enable just the `"abi3t"` feature, which will produce extensions targeting `abi3t` if the host interpreter is Python 3.15 or newer, but will produce version-specific extensions otherwise. | ||
| If you only enable the `"abi3"` feature, you will produce `abi3` extensions if the host interpreter is a GIL-enabled build of CPython and version-specific extensions otherwise. | ||
|
|
||
| We suggest enabling both features and using multiple python interpreters to build several wheels. | ||
| You can build an `abi3` wheel for your minimum supported Python version, a `cp314t` version-specific wheel using a free-threaded Python 3.14 interpreter, and an `abi3t` wheel using a Python 3.15 interpreter. | ||
| This will produce wheels targeting all versions of CPython that PyO3 supports. | ||
|
|
||
| 2. Ensure that the built shared objects are correctly marked as `abi3` and `abi3t`. | ||
| This is accomplished by telling your build system that you're using the limited API. | ||
| [`maturin`] >= 0.9.0 and [`setuptools-rust`] >= 0.11.4 support `abi3` wheels. | ||
| [`maturin`] >= 0.9.0 and [`setuptools-rust`] >= 0.11.4 support `abi3` wheels and [`maturin`] >= 1.14 supports `abi3t` wheels. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ping @messense, it probably makes sense to coordinate the release of PyO3 0.29 and a maturin release that supports abi3t. |
||
|
|
||
| See the [corresponding](https://github.com/PyO3/maturin/pull/353) [PRs](https://github.com/PyO3/setuptools-rust/pull/82) for more. | ||
| See the [corresponding](https://github.com/PyO3/maturin/pull/353) [PRs](https://github.com/PyO3/setuptools-rust/pull/82) [for](https://github.com/PyO3/maturin/pull/3113) more. | ||
|
|
||
| 3. Ensure that the `.whl` is correctly marked as `abi3`. | ||
| 3. Ensure that the `.whl` is correctly marked as `abi3` or `abi3t`. | ||
| For projects using `setuptools`, this is accomplished by passing `--py-limited-api=cp3x` (where `x` is the minimum Python version supported by the wheel, e.g. `--py-limited-api=cp35` for Python 3.5) to `setup.py bdist_wheel`. | ||
|
|
||
| #### Minimum Python version for `abi3` | ||
| #### Minimum Python version for `abi3` and `abi3t` builds | ||
|
|
||
| Because a single `abi3` wheel can be used with many different Python versions, PyO3 has feature flags `abi3-py38`, `abi3-py39`, `abi3-py310` etc. to set the minimum required Python version for your `abi3` wheel. | ||
| For example, if you set the `abi3-py38` feature, your wheel can be used on all Python 3 versions from Python 3.8 and up. | ||
| `maturin` and `setuptools-rust` will give the wheel a name like `my-extension-1.0-cp38-abi3-manylinux2020_x86_64.whl`. | ||
|
|
||
| Similarly, there is an `abi3t-py315` feature and future PyO3 versions will offer `abi3t-py316` and so on. | ||
|
|
||
| As your extension module may be run with multiple different Python versions you may occasionally find you need to check the Python version at runtime to customize behavior. | ||
| See [the relevant section of this guide](./building-and-distribution/multiple-python-versions.md#checking-the-python-version-at-runtime) on supporting multiple Python versions at runtime. | ||
|
|
||
| PyO3 is only able to link your extension module to abi3 version up to and including your host Python version. | ||
| E.g., if you set `abi3-py39` and try to compile the crate with a host of Python 3.8, the build will fail. | ||
|
|
||
| > [!NOTE] | ||
| > If you set more that one of these `abi3` version feature flags the lowest version always wins. For example, with both `abi3-py38` and `abi3-py39` set, PyO3 would build a wheel which supports Python 3.8 and up. | ||
| > If you set more that one of these `abi3` version feature flags the lowest version always wins. | ||
| > For example, with both `abi3-py38` and `abi3-py39` set, PyO3 would build a wheel which supports Python 3.8 and up. | ||
|
|
||
| #### Building `abi3` extension modules without a Python interpreter | ||
| #### Building stable ABI extension modules without a Python interpreter | ||
|
|
||
| As an advanced feature, you can build a PyO3 wheel without calling Python interpreter with the environment variable `PYO3_NO_PYTHON` set. | ||
| Also, if the build host Python interpreter is not found or is too old or otherwise unusable, PyO3 will still attempt to compile `abi3` extension modules after displaying a warning message. | ||
| Also, if the build host Python interpreter is not found or is too old or otherwise unusable, PyO3 will still attempt to compile stable ABI extension modules after displaying a warning message. | ||
|
|
||
| #### Missing features | ||
|
|
||
| Due to limitations in the Python API, there are a few `pyo3` features that do not work when compiling for `abi3`. | ||
| Due to limitations in the Python API, there are a few `pyo3` features that do not work when compiling for the stable ABI. | ||
| These are: | ||
|
|
||
| - `#[pyo3(text_signature = "...")]` does not work on classes until Python 3.10 or greater. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for updating this 👍
I wonder if (once we move to
ui-testin #5863) we can somehow make a ui test with this file to keep it in sync.