fix: bound the CEC subprocess reap, quieten three Sentry noise sources, and report the version to GA4

[vpetersson-bot]

Issues Fixed

Fixes #3264, #3265, #3266, #3267, #3268 — found during the 2026-08-07 Sentry triage sweep, then reproduced and validated on six physical testbeds. Also fixes the GA4 "Version distribution" report, which has never worked.

What changed

diagnostics — bound the CEC subprocess reap (#3264)

subprocess.run(..., capture_output=True, timeout=N) burns its entire timeout when a descendant still holds the inherited stdout pipe, because communicate() keeps draining. New _run_bounded(): start_new_session=True + killpg(SIGKILL), output to temp files instead of pipes, a timeout on every wait, and deliberately not a context manager (Popen.__exit__ is one of the hazards).

Measured, fast-exiting child with a grandchild holding stdout: 0.07-0.12s versus 8.0s for subprocess.run, on all three architectures.

diagnostics — three distinct CEC states (#3267)

'CEC error' covered three different situations and reported the most common one as a fault. Now 'No CEC adapter' (libcec raised), 'No CEC display detected' (adapter works, no peer answered — the normal case for a plain monitor), 'CEC adapter unresponsive' (libcec hung and was killed), and 'CEC error' for genuinely unexpected.

upgrade_containers.sh — pass /dev/cec0 on pi2/pi3/pi3-64/pi4-64 (#3267, root cause)

The vchiq→cec0 rewrite only matched pi5 and x86|arm64, so every other Pi fell through the case and kept the vchiq node libcec cannot open on a vc4-kms kernel. A targeted Pi 5 fix (#2863) never extended. Guarded on [ -e /dev/cec0 ], leaving vchiq untouched otherwise, since compose refuses to start when a listed host node is missing.

viewer — three log/behaviour fixes (#3265, #3266, #3268)

• Wayland headless-boot recovery logs a successful self-heal at ERROR, filing a Sentry issue per recovery #3265 the Wayland recovery logged its own success at ERROR, and the Sentry logging integration promotes ERROR to events, so every self-heal filed an issue. Demoted to warning; give-up-after-cap stays ERROR.
• Viewer fails to launch on eglfs/linuxfb boards, exhausting its attempt budget (not covered by #3240) #3266 short-circuit the launch retry loop when the display device has vanished — retrying cannot re-snapshot the container's /dev, so it only delays the restart that actually fixes it.
• Viewer idle log spam evicts crash diagnostics from the volatile journal within ~8h on low-RAM boards #3268 log the empty-playlist notice on transition, drop the per-tick URL line to debug, and throttle the not-displayable arm to once per offending asset.

telemetry — send the release version

The GA4 "Version distribution" report reads a version_name dimension that nothing has ever sent — not this payload, and not the pre-#2798 one either, whose Pi_Version was the hardware model. Hence ~3,700 devices as "(not reported)". Sourced from get_anthias_release() (pyproject.toml, ships in the image) rather than an env var, because missing env vars are the known failure mode here; verified resolving to 2026.7.3 inside anthias-anthias-celery-1.

Hardware validation — six boards

Board	Stack	Result
Pi 2	armhf / Qt5 / linuxfb	#3266 before/after + false-positive hunt, #3268, #3264
Pi 3 A+	armv7l, 361 MB	#3264 under real memory pressure; found #3267's hang
Pi 3-64	arm64 / Qt6 / eglfs	#3264 in the real worker, #3267 A/B, guard inert
Pi 4	arm64 / Qt6 / eglfs	#3267 with a real /dev/cec0; genuine D-state child
Pi 5	arm64 / Qt6 / Wayland	#3265 induced for real, with a Sentry A/B
x86	x86_64 / Wayland	negative control; redis-py bound

Rock Pi 4 was off-network throughout, so the generic arm64 image is not covered.

#3266 — before: 3 attempts / 6.02s / generic message. After: 1 attempt / 2.94s / distinct message. The false-positive risk was attacked directly and does not exist: with /dev/fb0 replaced by an unopenable node, Qt emitted two of the three signatures and the guard correctly declined, spending its full budget. Inert on eglfs and Wayland, with positive controls proving the code is live rather than dead.

#3264 — worst-case overrun +0.0027s. A genuine uninterruptible D-state child was produced on arm64 (O_DIRECT to the SD card, 98.7% D, wchan mmc_blk_rw_wait) and the reap still held, killpg reaping in 31.5ms. 40 real dispatches: fd delta 0, zombie delta 0, temp-file delta 0.

#3265 — induced for real on the Pi 5 with a second live cage: recovery at WARNING(30), give-up still ERROR(40), and a capturing-transport A/B showing ERROR→1 event, WARNING→0. Bounded-restart counter and cap unchanged.

#3268 — 1437-1824 lines/hour → 0 across boards, with the latch resetting in both directions.

#3267 — the decisive experiment, on the Pi 4 with the host's real /dev/cec0 passed into a throwaway container: cec.init() OK at 1187ms, Device(TV) OK at 1430ms, is_on() raises OSError: Power status not found → 'No CEC display detected' at 2910ms. Control with vchiq only → 'No CEC adapter' at 82ms. The two strings discriminate exactly as designed.

Claims corrected during validation — the testbeds refuted me five times

Recording these because the earlier commit messages in this branch asserted them:

1. The mechanism story was wrong. "SoftTimeLimitExceeded re-enters an unbounded wait so the task sails past the hard limit" does not reproduce on CPython 3.13, which calls process.kill() before every wait() — 0.00s overshoot, agreed by three boards. The real win is the pipe-drain stall. The docstring now says so, and the root cause of ANTHIAS-A/9/B/31 remains open — this is a robustness and latency fix, not a proven cure.
2. Display power is permanently 'CEC error' on mainline-KMS Pis: containers get /dev/vchiq but libcec needs /dev/cec0 #3267 did not land on two boards. libcec hangs on the vchiq-only Pi 2 and Pi 3 A+ rather than raising, so the 'No CEC adapter' branch was unreachable and operators saw the same 'CEC error'. Hence the 'CEC adapter unresponsive' state, and the passthrough fix.
3. The first Viewer idle log spam evicts crash diagnostics from the volatile journal within ~8h on low-RAM boards #3268 fix left a 5× worse offender untouched — the not-displayable arm loops on a 0.5s wait, measured at 7155-7200 lines/hour on three boards.
4. My own latch went permanently silent in the "empty → unavailable → empty" sequence.
5. The _run_bounded docstring claimed init would reap orphans. Container PID 1 is the celery worker; a killed grandchild persists. Caught independently by three boards.

Also corrected: #3266 does not reduce Sentry event volume (2 events before and after); it improves grouping and cuts the black-screen delay. And it cannot fire on the handshake-timeout path, whose message carries no Qt output — so the "~21 min" saving does not apply there.

Known-unfixed, deliberately

• Root cause of get_display_power still SIGKILLs the celery worker despite the soft-limit guard #3264's Sentry events. Bounding the reap removes a real stall but the hard-limit SIGKILL was never reproduced. get_display_power still SIGKILLs the celery worker despite the soft-limit guard #3264 stays open.
• cec_available() is still wrong on vchiq-only boards — it returns True because vchiq exists, so those boards still spawn a doomed subprocess every tick until the passthrough fix reaches them.
• The generic arm64 image is unvalidated (Rock Pi 4 off-network).
• Testbeds report to the production Sentry project as environment=production, so testbed errors are indistinguishable from fleet errors except by the pinned release tag. Two boards flagged it; worth its own fix.

Testing

1570 passed, 3 skipped (-m "not integration"), mypy clean across 169 files, ruff check + format clean.

Checklist

• I have performed a self-review of my own code.
• New and existing unit tests pass locally and on CI with my changes.
• I have done an end-to-end test for Raspberry Pi devices.
• I have tested my changes for x86 devices.
• I added a documentation for the changes I have made (when necessary).

[codecov]

Codecov Report

❌ Patch coverage is 82.92683% with 7 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (master@82c5def). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/anthias_server/lib/diagnostics.py	81.08%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #3269   +/-   ##
=========================================
  Coverage          ?   90.65%           
=========================================
  Files             ?       76           
  Lines             ?     8467           
  Branches          ?      898           
=========================================
  Hits              ?     7676           
  Misses            ?      570           
  Partials          ?      221           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[vpetersson-bot]

Closing in favour of four focused PRs. This one had grown to 977 lines across 8 files and 5 issues, plus an unrelated telemetry fix — too much to review as a unit, and worse, it welded together changes with completely different risk profiles.

PR	Contents	Risk
#3272	diagnostics bounded CEC reap + distinct CEC states + the celery r.set move	Medium — server library
#3271	Viewer: recovery log level, framebuffer short-circuit, idle log throttling	Medium — changes viewer restart behaviour
#3270	telemetry version_name for the GA4 report	Trivial — one outbound field
#3273	upgrade_containers.sh /dev/cec0 passthrough	High — runs during real OTA upgrades

The split matters most for #3273. It is simultaneously the riskiest change here (a mistake means containers fail to start fleet-wide) and the least validated — I never ran an actual upgrade, only verified the sed and reasoned from the existing x86|arm64 precedent. Buried in this PR, that caveat was easy to miss behind pages of reap measurements. On its own it can be held, reviewed and reverted independently.

Verified lossless: every changed file is covered by exactly one of the four, and 7 of 8 are byte-identical to this branch. The eighth, celery_tasks.py, differs only in a comment I reworded so it no longer asserts a state that #3273 changes — the two can now merge in either order.

All the hardware validation carries over; it was per-issue anyway. Each PR body keeps the measurements relevant to it, plus the caveats — including that #3264's root cause remains open: bounding the reap fixes a real 8-second stall, but three boards refuted my original mechanism claim and the hard-limit SIGKILL was never reproduced.

No commits are lost — the branch fix/sentry-triage-findings remains if anything needs to be recovered from the combined history.