[FEAT]: pytest-xdist support for distributed test execution#73
Open
nina-msft wants to merge 4 commits into
Open
[FEAT]: pytest-xdist support for distributed test execution#73nina-msft wants to merge 4 commits into
pytest-xdist support for distributed test execution#73nina-msft wants to merge 4 commits into
Conversation
The controller's _aggregate_trial_results iterated session.items looking for a _rampart_trial_base attribute set on cloned items at collection time. Under pytest-xdist that attribute is not reliably reachable on the controller at pytest_sessionfinish, so trial-group verdicts silently disappeared from the terminal summary and the JSON report -- per-clone results were present but no aggregate FAIL/PASS line was emitted. Decouple aggregation from pytest.Item state: - Store trial metadata as data on RampartSession._trial_specs (a dict[clone_nodeid, TrialSpec]) at collection time on every process. - Ship rial_specs through the existing ampart.xdist.v1 worker payload (back-compatible: missing/empty list is treated as no trials). - Controller merges specs from each worker and aggregates from the merged data instead of session.items. Also fixes a related JSON-sink gap: JsonFileReportSink now projects eport.metadata, so xdist run-mode info (worker_count, dist_mode, incomplete reasons) actually lands in the emitted file. Tests: - New TestTrialSpecs unit class (round-trip, malformed entries, non-finite thresholds, idempotent merge, first-writer-wins). - handle_testnodedown test covering trial-spec merge. - Strengthened ests/integration/test_xdist_aggregation.py to assert the trial-group line is present and correct under --dist=loadgroup and --dist=load (the prior tests only checked per-clone counts and would have missed this regression). - JSON-sink metadata projection unit tests. 565 unit tests + 13 xdist integration tests pass; ruff clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Summary
Adds first-class
pytest-xdistsupport to RAMPART so attack/probe sessions can shard across worker processes and still produce a single coherent report. Worker processes serialize their results through xdist'sworkeroutputchannel; the controller deserializes, merges, and writes the final report.What's included
New: _xdist.py
rampart.xdist.v1) for worker → controller payloads_sanitizeJSON-safe coercion at the trust boundary with depth limit, byte-size cap, and ANSI-escape stripping on deserialize to defend against terminal injection from worker outputSizeLimitErrorraised byfinalize_workerwhen serialized payload exceeds the configured cap; controller logs and continues--rampart-xdist-max-bytes/ ini optionrampart_xdist_max_bytes(default 64 MiB), validated positiveKeyboardInterrupt/SystemExit, catchesExceptionTrial-group aggregation across workers
RampartSessionat collection time and shipped through therampart.xdist.v1payload (trial_specs), so the controller aggregates trial groups from merged worker data instead ofsession.items— which is not reliably populated with trial clones on the controller at session finish. Without this, trial-group FAIL/PASS verdicts silently disappeared under xdist even though per-clone results were present.trial_specsis back-compatible: payloads without trials emit an empty list; malformed entries are skipped and non-finite thresholds are clamped on deserialize.Updated plugin hooks (
plugin.py,_session.py)pytest_addoptionregisters the new CLI/ini optionfinalize_workerto surface size-limit truncation as a warningReporting
JsonFileReportSinknow projectsreport.metadata, so xdist run-mode info (worker_count, dist_mode, incomplete reasons) lands in the emitted JSON fileTests
--dist=loadgroupand--dist=loadDocs
Incidental
Unknown*cascade caused bymsgraph-sdkshipping without type stubs. No behavior change.Security notes
The xdist boundary treats worker output as untrusted:
_sanitize→ JSON-safe primitives onlySizeLimitErrorMAX_METADATA_DEPTH = 6) prevents pathological nestingValidation
uv run pre-commit run --all-files→ all hooks pass (ruff, ruff-format, pyright)pytest tests/unit/pytest_plugin/test_xdist.py→ 73 passedpytest tests/integration/test_xdist_aggregation.py→ 13 passed (trial-group verdicts verified under--dist=loadgroupand--dist=load)pytest -n 4 --dist=loadgroupagainst HelpDesk Bot example inrampart-examples, verified 1 report generated. Running with xdist also took 99.66s versus serial execution at 230.73s (yay for speed!)Breaking changes
None
Checklist
pre-commit run --all-filespasses