coverage: capture what the Windows coverage crash actually is - #2426
Open
rluo8 wants to merge 4 commits into
Open
coverage: capture what the Windows coverage crash actually is#2426rluo8 wants to merge 4 commits into
rluo8 wants to merge 4 commits into
Conversation
The "Run cuda.core tests (with 8MB stack)" step on the Windows coverage runner exits 139 on every nightly, and its log holds nothing else: three lines, no pytest output at all. The step before it runs cuda.bindings through the same wrapper, the same 8 MB thread and the same coverage wheel and finishes normally with 479 passed, so the crash is specific to cuda.core -- and with no "collected N items" line to go on, we cannot even tell whether it reaches a test. PYTHONFAULTHANDLER is set job-wide because it costs nothing; the rest is scoped to the one step that crashes. No behaviour change. - PYTHONFAULTHANDLER names the exception and dumps every thread's Python stack. Unlike pytest's built-in faulthandler plugin, the env var also covers tests that spawn their own `python -c` children. PYTHONUNBUFFERED keeps the final lines out of the block buffer of a non-tty pipe. - run_pytest_with_stack.py --isolate runs pytest in a child process so a surviving parent can report its raw exit status. The shell only shows a translation of that status; the parent names the NTSTATUS outright. The child's status is passed through unchanged. - pytest_crashlog records session progress to a line-buffered file, so the last line survives a native fault and localises it to interpreter startup, conftest import, a named module's collection, a named test, or interpreter shutdown. The wrapper opens the log before calling pytest.main, because pytest imports the initial conftests -- and for cuda_core that pulls in cuda.core -- before pytest_configure runs. A no-op unless PYTEST_CRASHLOG is set. Verified on a Windows GPU runner against the linetrace coverage wheel: identical test results with and without the change. Signed-off-by: Rui Luo <ruluo@nvidia.com>
Contributor
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
Hi @mdboom , could you please help trigger a coverage run with this draft PR ? The Windows coverage run always failed when running cuda.core tests. This is to add some debug logs for triage. |
Contributor
|
/ok to test 8291f92 |
Contributor
|
/ok to test 9ed18a7 |
|
TEMPORARY -- not for merge. The exit-139 crash this branch instruments only reproduces on the Windows coverage runner, and coverage.yml is reachable only from the nightly schedule and from workflow_dispatch on the default branch. Every iteration of the diagnostics has therefore needed a maintainer to sync the mirror branch by hand. copy-pr-bot already pushes this PR to pull-request/<n> on the upstream repo, and a push event reads the workflow from the pushed commit, so the trigger takes effect from the PR itself without merging. ci.yml uses the same trigger. A PR run covers the Windows leg alone -- coverage-vars, build-wheel-windows, coverage-windows -- roughly 28 minutes, 18 of them on an A100. The two skipped jobs are skipped for different reasons: coverage-linux has no bearing on a Windows crash, and combine-and-deploy would both fail on the missing .coverage.linux and publish a half report to gh-pages. always() does not stop the latter, so its condition is gated as well. concurrency cancels superseded runs, which would otherwise queue another A100 job per push. The nightly is unchanged: on refs/heads/main both conditions hold and all five jobs run as before. Signed-off-by: Rui Luo <ruluo@nvidia.com>
rluo8
marked this pull request as ready for review
July 29, 2026 02:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is a draft pr for diagnostics. Please do not merge.
The
Run cuda.core tests (with 8MB stack)step ofCI: Coveragealways exited 139. Its log is three lines — the traced command,Segmentation fault, exit 139 — with no pytest header, nocollected N items, no test ids and no faulthandler traceback. I couldn't reproduce this failures on my local systems.Three diagnostics, no behaviour change:
PYTHONFAULTHANDLER/PYTHONUNBUFFERED— names the exception and dumps every thread's stack, and unlike pytest's built-in plugin it reaches tests that spawn their ownpython -cchildren.--isolate— runs pytest in a child process so a surviving parent reports the raw NTSTATUS instead of the shell's translation of it (Git Bash renders an access violation asSegmentation fault/ 139). The child's status is passed through unchanged.pytest_crashlog— line-buffered progress log whose last line survives a native fault, localising it to interpreter startup, conftest import, a module's collection, a test, or shutdown. Opened beforepytest.main, since pytest imports the conftests — which pull incuda.core— beforepytest_configure. A no-op unlessPYTEST_CRASHLOGis set.