Experimental _pytest.scenario API: nested configs + in-memory collection (PoC)

[RonnyPfannschmidt]

Context

pytest's own testsuite is heavily integration-based: of ~3200 real test functions in testing/, ~61% use pytester, ~91% of those write files to disk, and roughly 1400 assertion sites glob-match rendered terminal output (vs ~270 structured ones). Tests need a full session because stdout is the only observable most of them have — there is no API to collect an item from an in-memory object, or to build a Config without a real rootdir/cwd. testing/conftest.py even reorders tests by decompiling __code__.co_names because "uses pytester" is the only available proxy for "slow".

This PR is a proof of concept for fixing that at the API level. Draft — the API shape and naming are up for discussion.

What's included

New experimental package _pytest/scenario/ (internal, not exported from pytest):

def test_static_method(tmp_path):
    class Test:
        @staticmethod
        def test_something(): ...
        @pytest.fixture
        def fix(self): return 1
        @staticmethod
        def test_fix(fix): assert fix == 1

    record = run_tests(Test, rootpath=tmp_path)
    record.assert_outcomes(passed=2)

• ConfigSpec / configured() — a parsed+configured nested Config from declarative data, built through the real parse phases but without rootdir discovery, config files, conftests, plugin autoload, or env consultation. Hermetic by default; teardown is paired via _ensure_unconfigure().
• fake_module(name, *members) — group loose functions/classes into an in-memory module with an explicit name; collect_tests() / run_tests() / stepwise scenario() collect through the standard pytest_collection flow (so -k/-m, parametrize, fixture scoping and pytest_collection_modifyitems all work) via a ScenarioModule whose _getobj serves the preset object. Nodeids derive from synthetic rootdir-relative paths that are never touched on disk.
• RunRecord / ItemRunRecord — typed results derived from real report objects via pytest_report_teststatus, with assert_outcomes() signature-compatible with pytester and per-test lookup (record["test_x"].setup.longreprtext).

Core changes in support:

• Config.parse() split into behavior-preserving phase methods (_preparse_addopts, _apply_rootdir, _register_core_ini_options, _load_plugins_phase, _load_initial_conftests_phase, _finalize_parse(decide_args=)), plus ArgsSource.SPEC. The programmatic path shares real code with command line parsing instead of bypassing it.
• Class.from_parent() no longer silently discards a passed obj (it was accepted and dropped).
• get_config() gained an explicit invocation dir= (was hardcoded Path.cwd()).
• debugging/unittest guard their cross-plugin option reads (trace/usepdb) so they work when the debugging plugin is not registered.

Validation: 33 self-tests in testing/test_scenario.py run in ~0.3s with hermeticity asserted (no sys.path/sys.modules/cwd/environ mutation, no files created); seven existing tests in testing/python/{collect,fixtures,metafunc}.py converted, including the hand-rolled make_function harness.

Deliberate scope cuts (documented follow-ups)

• The default scenario plugin set excludes capture, terminal, assertion, cacheprovider and the process-global-state plugins. capsys/capfd inside scenarios are unavailable until CaptureManager becomes stack-aware (suspend/resume is currently absolute, restoring values memoised at __init__ — nested managers work only by LIFO luck today). That is the next pillar.
• Conftest loading in scenarios (load_conftests=True) raises NotImplementedError; plugin objects via ConfigSpec.extra_plugins are the replacement.
• Host process warning filters (e.g. our own filterwarnings = error) are inherited by scenarios; a scenario's inicfg={"filterwarnings": [...]} takes precedence.
• Longer term: rebase pytester's inline_run onto this API, migrate stdout outcome-assertions to RunRecord, and consider pytest.scenario graduation.

The changelog fragment is 14809.feature.rst.

---

Per our AI contribution policy: this was researched, designed and implemented with Claude Code under my direction and review.

🤖 Generated with Claude Code

[bluetech]

Nice initiative. Anything that brings more isolation, hermeticity and control of side-effects seems like a good direction to me.

I haven't dug into the code or details yet, so just some small comments:

---

The name "scenario" sounds possibly too generic for this, it doesn't sound like sometime that is intended for testing pytest itself, but like an end-user feature, like pytest-bdd has @scenario for example.

---

ConfigSpec / configured() — a parsed+configured nested Config from declarative data, built through the real parse phases but without rootdir discovery, config files, conftests, plugin autoload, or env consultation. Hermetic by default; teardown is paired via _ensure_unconfigure().

This sounds possibly nice but didn't look at the details. In due time, it would be nice to have this in a separate commit or PR to be considered separately.

---

fake_module(name, *members) — group loose functions/classes into an in-memory module with an explicit name; collect_tests() / run_tests() / stepwise scenario() collect through the standard pytest_collection flow (so -k/-m, parametrize, fixture scoping and pytest_collection_modifyitems all work) via a ScenarioModule whose _getobj serves the preset object. Nodeids derive from synthetic rootdir-relative paths that are never touched on disk.

👍 on the general idea.

---

RunRecord / ItemRunRecord — typed results derived from real report objects via pytest_report_teststatus, with assert_outcomes() signature-compatible with pytester and per-test lookup (record["test_x"].setup.longreprtext).

What's the reason to use separate types than the TestReport etc. themselves? I do believe pytester has some helpers to grab those, although we don't use them as much as we should.

---

Regarding these:

• Class.from_parent() no longer silently discards a passed obj (it was accepted and dropped).
• get_config() gained an explicit invocation dir= (was hardcoded Path.cwd()).
• debugging/unittest guard their cross-plugin option reads (trace/usepdb) so they work when the debugging plugin is not registered.

These all sound (just based on the description) look nice independent improvements. Consider submitting as separate PRs, to reduce the changes here.

---

The default scenario plugin set excludes capture, terminal, assertion, cacheprovider and the process-global-state plugins. capsys/capfd inside scenarios are unavailable until CaptureManager becomes stack-aware (suspend/resume is currently absolute, restoring values memoised at init — nested managers work only by LIFO luck today). That is the next pillar.

These sound like they can share the benefits with the threading work.

---

Longer term: rebase pytester's inline_run onto this API, migrate stdout outcome-assertions to RunRecord, and consider pytest.scenario graduation.

Ideally we can have this nice stuff in pytester from the start, instead of a separate API, just to keep things simple for users (mostly plugin authors). But maybe there's good reason to have a separate API, I don't know yet.

[RonnyPfannschmidt]

This is a initial poc

It includes a number of precursor enhancements that need to be extracted and landed first

As to why new types/apis - pytester is very intertwined with the problematic way to run things and completely dependent on capture- i wanted to start without that as limit to explore the details first