Skip to content

[FEAT]: pytest-xdist support for distributed test execution#73

Open
nina-msft wants to merge 4 commits into
microsoft:mainfrom
nina-msft:dev/nina-msft/8259
Open

[FEAT]: pytest-xdist support for distributed test execution#73
nina-msft wants to merge 4 commits into
microsoft:mainfrom
nina-msft:dev/nina-msft/8259

Conversation

@nina-msft

@nina-msft nina-msft commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Description

Summary

Adds first-class pytest-xdist support to RAMPART so attack/probe sessions can shard across worker processes and still produce a single coherent report. Worker processes serialize their results through xdist's workeroutput channel; the controller deserializes, merges, and writes the final report.

What's included

New: _xdist.py

  • Versioned schema (rampart.xdist.v1) for worker → controller payloads
  • _sanitize JSON-safe coercion at the trust boundary with depth limit, byte-size cap, and ANSI-escape stripping on deserialize to defend against terminal injection from worker output
  • SizeLimitError raised by finalize_worker when serialized payload exceeds the configured cap; controller logs and continues
  • Configurable cap via new pytest CLI option --rampart-xdist-max-bytes / ini option rampart_xdist_max_bytes (default 64 MiB), validated positive
  • Sink resolution re-raises KeyboardInterrupt/SystemExit, catches Exception

Trial-group aggregation across workers

  • Trial metadata is captured as data on RampartSession at collection time and shipped through the rampart.xdist.v1 payload (trial_specs), so the controller aggregates trial groups from merged worker data instead of session.items — which is not reliably populated with trial clones on the controller at session finish. Without this, trial-group FAIL/PASS verdicts silently disappeared under xdist even though per-clone results were present.
  • trial_specs is back-compatible: payloads without trials emit an empty list; malformed entries are skipped and non-finite thresholds are clamped on deserialize.

Updated plugin hooks (plugin.py, _session.py)

  • pytest_addoption registers the new CLI/ini option
  • Worker, controller, and non-xdist branches dispatched cleanly; worker branch wraps finalize_worker to surface size-limit truncation as a warning

Reporting

  • JsonFileReportSink now projects report.metadata, so xdist run-mode info (worker_count, dist_mode, incomplete reasons) lands in the emitted JSON file

Tests

  • 73 unit tests in test_xdist.py covering serialization, sanitization, size limits, ANSI stripping, sink resolution, option parsing, and trial-spec round-trip/merge
  • Integration test test_xdist_aggregation.py exercising end-to-end aggregation across workers, including trial-group verdicts under both --dist=loadgroup and --dist=load

Docs

  • New xdist.md user guide
  • Configuration, CI integration, pytest-integration, and API reference pages updated to describe the new option (under a dedicated "Pytest Options" section, not under env vars)
  • Architecture page notes the controller/worker split

Incidental

  • onedrive.py: file-level pyright directive suppressing the Unknown* cascade caused by msgraph-sdk shipping without type stubs. No behavior change.

Security notes

The xdist boundary treats worker output as untrusted:

  • All values pass through _sanitize → JSON-safe primitives only
  • Strings are stripped of ANSI escape sequences on deserialize (including nested strings inside dicts/lists) to prevent terminal injection from a compromised/misbehaving worker
  • Total payload size is capped to prevent controller memory exhaustion; truncation is recorded with a marker and raised as SizeLimitError
  • Depth cap (MAX_METADATA_DEPTH = 6) prevents pathological nesting

Validation

  • uv run pre-commit run --all-files → all hooks pass (ruff, ruff-format, pyright)
  • pytest tests/unit/pytest_plugin/test_xdist.py → 73 passed
  • pytest tests/integration/test_xdist_aggregation.py → 13 passed (trial-group verdicts verified under --dist=loadgroup and --dist=load)
  • Ran pytest -n 4 --dist=loadgroup against HelpDesk Bot example in rampart-examples, verified 1 report generated. Running with xdist also took 99.66s versus serial execution at 230.73s (yay for speed!)

Breaking changes

None

Checklist

  • pre-commit run --all-files passes
  • Tests added or updated for changes
  • Documentation updated

@nina-msft nina-msft requested review from a team and bashirpartovi June 4, 2026 01:35
The controller's _aggregate_trial_results iterated session.items looking for a _rampart_trial_base attribute set on cloned items at collection time. Under pytest-xdist that attribute is not reliably reachable on the controller at pytest_sessionfinish, so trial-group verdicts silently disappeared from the terminal summary and the JSON report -- per-clone results were present but no aggregate FAIL/PASS line was emitted.

Decouple aggregation from pytest.Item state:

- Store trial metadata as data on RampartSession._trial_specs (a dict[clone_nodeid, TrialSpec]) at collection time on every process.

- Ship 	rial_specs through the existing 
ampart.xdist.v1 worker payload (back-compatible: missing/empty list is treated as no trials).

- Controller merges specs from each worker and aggregates from the merged data instead of session.items.

Also fixes a related JSON-sink gap: JsonFileReportSink now projects 
eport.metadata, so xdist run-mode info (worker_count, dist_mode, incomplete reasons) actually lands in the emitted file.

Tests:

- New TestTrialSpecs unit class (round-trip, malformed entries, non-finite thresholds, idempotent merge, first-writer-wins).

- handle_testnodedown test covering trial-spec merge.

- Strengthened 	ests/integration/test_xdist_aggregation.py to assert the trial-group line is present and correct under --dist=loadgroup and --dist=load (the prior tests only checked per-clone counts and would have missed this regression).

- JSON-sink metadata projection unit tests.

565 unit tests + 13 xdist integration tests pass; ruff clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant