ci: switch to `ui-test` instead of `trybuild` by davidhewitt · Pull Request #5863 · PyO3/pyo3

davidhewitt · 2026-03-06T14:13:29Z

I had some conversations at RustNation which inspired me to look into ui-test instead of trybuild for our UI testing. It seems pretty good on first glance:

Seems like there is more control on user normalizations
More choices on pass / fail granularity (e.g. in ci: install debug interpreter with uv #5447 we have a failure from trybuild attempting to run tests when we only need check-pass)
It seems to build a lot faster than trybuild in my testing this morning.

Pushing to get feedback in CI, I am optimistic this might be a nice improvement.

Icxolu · 2026-03-28T15:43:34Z

This looks interesting and does indeed seem to be quite a bit faster than trybuild. Having more control over the features, cfgs and compiler flags used is nice. I wonder whether coverage already works out of the box or if we need to configure something for that.

While playing around with it locally I ended up adjusting things a bit. I attached the diff below, in case that is useful.

Details

diff --git a/tests/test_compile_error.rs b/tests/test_compile_error.rs
index 22784c9df0..de83f6a7e1 100644
--- a/tests/test_compile_error.rs
+++ b/tests/test_compile_error.rs
@@ -82,6 +82,13 @@
         // generic pyclasses only supported on 3.9+, doesn't fail gracefully on older versions
         #[cfg(not(Py_3_9))]
         "invalid_pyclass_generic.rs".into(),
+        // an extra "note" is emitted on abi3
+        #[cfg(any(not(Py_LIMITED_API), not(Py_3_12)))]
+        "invalid_base_class.rs".into(),
+        #[cfg(all(Py_LIMITED_API, not(Py_3_10)))]
+        "invalid_pyfunction_argument.rs".into(),
+        #[cfg(all(Py_LIMITED_API, not(Py_3_10)))]
+        "invalid_pyclass_args.rs".into(),
     ]);
 
     match std::env::var("UI_TEST").as_deref() {
diff --git a/tests/ui/ambiguous_associated_items.rs b/tests/ui/ambiguous_associated_items.rs
index 3c9f4b0039..bd905fab99 100644
--- a/tests/ui/ambiguous_associated_items.rs
+++ b/tests/ui/ambiguous_associated_items.rs
@@ -1,4 +1,5 @@
 //@check-pass
+#![allow(dead_code)]
 use pyo3::prelude::*;
 
 #[pyclass(eq)]
diff --git a/tests/ui/ambiguous_associated_items.stderr b/tests/ui/ambiguous_associated_items.stderr
deleted file mode 100644
index a8b30ca7fb..0000000000
--- a/tests/ui/ambiguous_associated_items.stderr
+++ /dev/null
@@ -1,15 +0,0 @@
-warning: enum `DeriveItems` is never used
-  --> tests/ui/ambiguous_associated_items.rs:20:6
-   |
-20 | enum DeriveItems {
-   |      ^^^^^^^^^^^
-   |
-   = note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default
-
-warning: enum `DeriveItemsFromPyObject` is never used
-  --> tests/ui/ambiguous_associated_items.rs:43:6
-   |
-43 | enum DeriveItemsFromPyObject {
-   |      ^^^^^^^^^^^^^^^^^^^^^^^
-
-warning: 2 warnings emitted
diff --git a/tests/ui/invalid_base_class.stderr b/tests/ui/invalid_base_class.stderr
index 5ee05a405a..7f6a85b2d6 100644
--- a/tests/ui/invalid_base_class.stderr
+++ b/tests/ui/invalid_base_class.stderr
@@ -6,6 +6,7 @@
   |
   = help: the trait `PyClassBaseType` is not implemented for `PyBool`
   = note: `PyBool` must have `#[pyclass(subclass)]` to be eligible for subclassing
+  = note: with the `abi3` feature enabled, PyO3 does not support subclassing native types
   = help: the following other types implement trait `PyClassBaseType`:
             PyAny
             PyArithmeticError
@@ -15,7 +16,7 @@
             PyBaseExceptionGroup
             PyBlockingIOError
             PyBrokenPipeError
-          and 69 others
+          and 63 others
   = note: this error originates in the attribute macro `pyclass` (in Nightly builds, run with -Z macro-backtrace for more info)
 
 error[E0277]: pyclass `PyBool` cannot be subclassed
@@ -26,6 +27,7 @@
     |
     = help: the trait `PyClassBaseType` is not implemented for `PyBool`
     = note: `PyBool` must have `#[pyclass(subclass)]` to be eligible for subclassing
+    = note: with the `abi3` feature enabled, PyO3 does not support subclassing native types
     = help: the following other types implement trait `PyClassBaseType`:
               PyAny
               PyArithmeticError
@@ -35,7 +37,7 @@
               PyBaseExceptionGroup
               PyBlockingIOError
               PyBrokenPipeError
-            and 69 others
+            and 63 others
 note: required by a bound in `PyClassImpl::BaseType`
    --> src/impl_/pyclass.rs:184:33
     |
diff --git a/tests/ui/invalid_property_args.stderr b/tests/ui/invalid_property_args.stderr
index e2c719babd..f819a76271 100644
--- a/tests/ui/invalid_property_args.stderr
+++ b/tests/ui/invalid_property_args.stderr
@@ -66,12 +66,12 @@
              and 151 others
      = note: required for `PhantomData<i32>` to implement `for<'py> PyO3GetField<'py>`
 note: required by a bound in `PyClassGetterGenerator::<ClassT, FieldT, OFFSET, false, false, IMPLEMENTS_INTOPYOBJECT>::generate`
-    --> src/impl_/pyclass.rs:1328:26
+    --> src/impl_/pyclass.rs:1297:26
      |
-1324 |     pub const fn generate(&self, name: &'static CStr, doc: Option<&'static CStr>) -> PyMethodDefType
+1293 |     pub const fn generate(&self, name: &'static CStr, doc: Option<&'static CStr>) -> PyMethodDefType
      |                  -------- required by a bound in this associated function
 ...
-1328 |         for<'py> FieldT: PyO3GetField<'py>,
+1297 |         for<'py> FieldT: PyO3GetField<'py>,
      |                          ^^^^^^^^^^^^^^^^^ required by this bound in `PyClassGetterGenerator::<ClassT, FieldT, OFFSET, false, false, IMPLEMENTS_INTOPYOBJECT>::generate`
 
 error: aborting due to 9 previous errors
diff --git a/tests/ui/not_send.rs b/tests/ui/not_send.rs
index d93d605327..6ff785d463 100644
--- a/tests/ui/not_send.rs
+++ b/tests/ui/not_send.rs
@@ -1,8 +1,9 @@
+//@normalize-stderr-test: ".*/src/rust/(.*)" -> "../src/$1"
 use pyo3::prelude::*;
 
 fn test_not_send_detach(py: Python<'_>) {
-    py.detach(|| { drop(py); });
-//~^ ERROR: `*mut pyo3::Python<'static>` cannot be shared between threads safely
+    py.detach(|| drop(py));
+    //~^ ERROR: `*mut pyo3::Python<'static>` cannot be shared between threads safely
 }
 
 fn main() {
diff --git a/tests/ui/not_send.stderr b/tests/ui/not_send.stderr
index 837f0a6963..81ae40120a 100644
--- a/tests/ui/not_send.stderr
+++ b/tests/ui/not_send.stderr
@@ -1,14 +1,14 @@
 error[E0277]: `*mut pyo3::Python<'static>` cannot be shared between threads safely
-   --> tests/ui/not_send.rs:4:15
+   --> tests/ui/not_send.rs:5:15
     |
-  4 |     py.detach(|| { drop(py); });
-    |        ------ ^^^^^^^^^^^^^^^^ `*mut pyo3::Python<'static>` cannot be shared between threads safely
+  5 |     py.detach(|| drop(py));
+    |        ------ ^^^^^^^^^^^ `*mut pyo3::Python<'static>` cannot be shared between threads safely
     |        |
     |        required by a bound introduced by this call
     |
     = help: within `pyo3::Python<'_>`, the trait `Sync` is not implemented for `*mut pyo3::Python<'static>`
 note: required because it appears within the type `PhantomData<*mut pyo3::Python<'static>>`
-   --> /Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/marker.rs:819:12
+../src/library/core/src/marker.rs:819:12
     |
 819 | pub struct PhantomData<T: PointeeSized>;
     |            ^^^^^^^^^^^
@@ -18,7 +18,7 @@
 357 | struct NotSend(PhantomData<*mut Python<'static>>);
     |        ^^^^^^^
 note: required because it appears within the type `PhantomData<pyo3::marker::NotSend>`
-   --> /Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/marker.rs:819:12
+../src/library/core/src/marker.rs:819:12
     |
 819 | pub struct PhantomData<T: PointeeSized>;
     |            ^^^^^^^^^^^
@@ -29,11 +29,11 @@
     |            ^^^^^^
     = note: required for `&pyo3::Python<'_>` to implement `Send`
 note: required because it's used within this closure
-   --> tests/ui/not_send.rs:4:15
+   --> tests/ui/not_send.rs:5:15
     |
-  4 |     py.detach(|| { drop(py); });
+  5 |     py.detach(|| drop(py));
     |               ^^
-    = note: required for `{closure@tests/ui/not_send.rs:4:15: 4:17}` to implement `Ungil`
+    = note: required for `{closure@tests/ui/not_send.rs:5:15: 5:17}` to implement `Ungil`
 note: required by a bound in `pyo3::Python::<'py>::detach`
    --> src/marker.rs:560:12
     |
diff --git a/tests/ui/not_send2.rs b/tests/ui/not_send2.rs
index b1d3262ead..58382a7c20 100644
--- a/tests/ui/not_send2.rs
+++ b/tests/ui/not_send2.rs
@@ -1,3 +1,4 @@
+//@normalize-stderr-test: ".*/src/rust/(.*)" -> "../src/$1"
 use pyo3::prelude::*;
 use pyo3::types::PyString;
 
@@ -6,7 +7,7 @@
         let string = PyString::new(py, "foo");
 
         py.detach(|| {
-//~^ ERROR: `*mut pyo3::Python<'static>` cannot be shared between threads safely
+            //~^ ERROR: `*mut pyo3::Python<'static>` cannot be shared between threads safely
             println!("{:?}", string);
         });
     });
diff --git a/tests/ui/not_send2.stderr b/tests/ui/not_send2.stderr
index 6921905be4..c69132e46b 100644
--- a/tests/ui/not_send2.stderr
+++ b/tests/ui/not_send2.stderr
@@ -1,18 +1,18 @@
 error[E0277]: `*mut pyo3::Python<'static>` cannot be shared between threads safely
-   --> tests/ui/not_send2.rs:8:19
+   --> tests/ui/not_send2.rs:9:19
     |
-  8 |           py.detach(|| {
+  9 |           py.detach(|| {
     |  ____________------_^
     | |            |
     | |            required by a bound introduced by this call
-  9 | |
- 10 | |             println!("{:?}", string);
- 11 | |         });
+ 10 | |
+ 11 | |             println!("{:?}", string);
+ 12 | |         });
     | |_________^ `*mut pyo3::Python<'static>` cannot be shared between threads safely
     |
     = help: within `pyo3::Bound<'_, PyString>`, the trait `Sync` is not implemented for `*mut pyo3::Python<'static>`
 note: required because it appears within the type `PhantomData<*mut pyo3::Python<'static>>`
-   --> /Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/marker.rs:819:12
+../src/library/core/src/marker.rs:819:12
     |
 819 | pub struct PhantomData<T: PointeeSized>;
     |            ^^^^^^^^^^^
@@ -22,7 +22,7 @@
 357 | struct NotSend(PhantomData<*mut Python<'static>>);
     |        ^^^^^^^
 note: required because it appears within the type `PhantomData<pyo3::marker::NotSend>`
-   --> /Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/marker.rs:819:12
+../src/library/core/src/marker.rs:819:12
     |
 819 | pub struct PhantomData<T: PointeeSized>;
     |            ^^^^^^^^^^^
@@ -38,11 +38,11 @@
     |            ^^^^^
     = note: required for `&pyo3::Bound<'_, PyString>` to implement `Send`
 note: required because it's used within this closure
-   --> tests/ui/not_send2.rs:8:19
+   --> tests/ui/not_send2.rs:9:19
     |
-  8 |         py.detach(|| {
+  9 |         py.detach(|| {
     |                   ^^
-    = note: required for `{closure@tests/ui/not_send2.rs:8:19: 8:21}` to implement `Ungil`
+    = note: required for `{closure@tests/ui/not_send2.rs:9:19: 9:21}` to implement `Ungil`
 note: required by a bound in `pyo3::Python::<'py>::detach`
    --> src/marker.rs:560:12
     |

Co-authored-by: Icxolu <10486322+Icxolu@users.noreply.github.com>

davidhewitt · 2026-03-28T18:40:04Z

Thanks!

Yes agreed, trybuild has served us very well as a drop-in-and-go solution for a long time, though ultimately it's now blocking #5447, and while a workaround is probably possible, it's felt like we've been pushing against the limits of what trybuild can handle for a while.

RE coverage: I might be wrong, but because the ui tests get compiled as part of the workspace under ui-test, I have a feeling that coverage will "just work" and be simpler than trybuild (which would create its own temporary workspace and target directory so it could configure flags).

davidhewitt added 2 commits March 6, 2026 14:10

try out ui-test instead of trybuild

270d33b

clippy

985d2d3

davidhewitt mentioned this pull request Mar 7, 2026

ci: install debug interpreter with uv #5447

Draft

davidhewitt added 3 commits March 7, 2026 14:15

Merge remote-tracking branch 'origin/main' into ui-test

00c27cf

fixup several ui-test setups

2f90d42

attempt to fix CI errors

4ea3933

davidhewitt added the CI-skip-changelog Skip checking changelog entry label Mar 8, 2026

davidhewitt added 2 commits March 20, 2026 11:24

Merge branch 'main' into ui-test

d8c5c3d

forward cargo target to ui-test

ae05fdd

davidhewitt added the CI-build-full label Mar 27, 2026

davidhewitt changed the title ~~try out ui-test instead of trybuild~~ ci: switch to ui-test instead of trybuild Mar 27, 2026

davidhewitt added 3 commits March 27, 2026 21:28

fix ignore mechanism

d598cdb

Merge branch 'main' into ui-test

d5df570

remove abi3-py37 reference

7129160

davidhewitt and others added 2 commits March 28, 2026 18:22

apply suggestions from code review

50a4e10

Co-authored-by: Icxolu <10486322+Icxolu@users.noreply.github.com>

skip ui testing on careful / valgrind

0818da7

davidhewitt added 2 commits March 28, 2026 19:24

fixup MSRV handling

f26cb22

tidy import

b4d0cff

davidhewitt added the CI-no-fail-fast If one job fails, allow the rest to keep testing label Mar 28, 2026

davidhewitt added 6 commits March 29, 2026 13:54

empty.rs not a real UI test

d645b47

add .gitattributes

3b891bd

clean up ui test harness

d5df48d

fix invalid_cancel_handle test

6e92432

clean up a bunch of cases

eacdcc4

fix cfg gate on compile error test entry

ab88ab7

davidhewitt mentioned this pull request Mar 31, 2026

Run tests with macros and without experimental inspect #5913

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: switch to `ui-test` instead of `trybuild`#5863

ci: switch to `ui-test` instead of `trybuild`#5863
davidhewitt wants to merge 20 commits intoPyO3:mainfrom
davidhewitt:ui-test

davidhewitt commented Mar 6, 2026

Uh oh!

Icxolu commented Mar 28, 2026

Uh oh!

davidhewitt commented Mar 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

davidhewitt commented Mar 6, 2026

Uh oh!

Icxolu commented Mar 28, 2026

Uh oh!

davidhewitt commented Mar 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants