Skip to content

e2e/qa: stop counting devices that cannot accept users as failures - #4168

Open
ben-dz wants to merge 3 commits into
mainfrom
bdz/qa-skip-devices-not-accepting-users
Open

e2e/qa: stop counting devices that cannot accept users as failures#4168
ben-dz wants to merge 3 commits into
mainfrom
bdz/qa-skip-devices-not-accepting-users

Conversation

@ben-dz

@ben-dz ben-dz commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary of Changes

Mainnet Beta QA failed 7 times over Aug 8-10; three of those (infra runs 31248009915, 31266248546, 31286128831) had one cause. qa_alldevices_unicast_test.go checks device readiness at five sites and logs Ignoring <x> failure for device not ready for users, but each site increments FailedTests before that check, so the carve-out only suppressed the log line while the device still counted as failed. laconic-dfw-sw01, laconic-mia-sw01 and laconic-was-sw01 have been activated with max_users=0 since Aug 6, and the CLI refuses those connects outright; cmh-mn-qa01 draws from a 13-device pool, so three unusable devices produced a 21-29% per-host rate against the 20% gate.

Gating the counter is not enough: Success() also requires PacketsSent > 0 && PacketsReceived > 0, which a device that never connected can never satisfy. So not-ready devices are excluded from ComputeFailureStats entirely — including from the per-host denominator — behind a new Device.Ready() that replaces the condition previously spelled out at six call sites.

That exclusion opens a fail-open path worth reviewing closely: with every device excluded, totalDevices is 0, the overall rate is NaN, and NaN > threshold is false, so a run that tested nothing would pass. Devices are created with max_users=0 until activation unlocks them, so a new or metro-drained fleet lands exactly there. The rate math is now guarded by a t.Fatalf, and FailureStats.Skipped carries the excluded codes into test output so "3 devices skipped" is distinguishable from "3 devices tested and fine".

Separately, client_unicast.go wrapped an always-nil lastErr — the retry loop returns early on any error — so every connectivity failure this weekend read failed to ping after 3 retries: %!w(<nil>). It now reports packet counts.

Ready() is a deliberate subset of the program's is_device_eligible_for_provisioning: a device at users_count + reserved_seats >= max_users hits the same CLI rejection and still counts as a failure. That is the same class of false red one field over, but fixing it means re-adding the capacity check #3697 removed, so it is left as a follow-up. MaxUnicastUsers is not consulted — 0 there means "no per-type limit", the inverse of drained.

Testing Verification

go test ./e2e/internal/qa/... and golangci-lint run --build-tags=qa ./e2e/... (0 issues) pass. The four new TestComputeFailureStats subtests were confirmed to fail against the unfixed aggregation and pass after; without the fix the mainnet shape reports total=6 failed=3 where it should report total=3 failed=0. TestPingFailureError asserts the message carries packet counts and never renders %!w. Existing fixtures in that file had zero-valued Status/MaxUsers and so were not Ready(); they are now explicitly activated, which is why the diff touches subtests it does not otherwise change.

Not verified: the qa-tagged e2e tests need live fleet access and cannot run here, so the change is unexercised against the real fleet. TestQAAgentConnectivity/Ping fails in this container for lack of raw-socket privileges, on main as well as this branch.

A reviewer also flagged that a not-ready device still consumes a batch host slot, and on the two-host devnet run that can trip the existing fewer than 2 clients connected fatal regardless of this fix. That predates this change and did not fire in any weekend run; filing separately rather than widening this diff.

ben-dz added 3 commits August 10, 2026 14:24
The all-devices unicast test checked device readiness at four sites and
logged "Ignoring <x> failure for device not ready for users", but each
site incremented FailedTests before the check, so the carve-out only
suppressed the log line while the device still counted as failed.

Devices left activated with max_users=0 are drained on purpose and the
CLI refuses the connect, so exclude them from ComputeFailureStats
entirely rather than gating the counter: Success() also requires
PacketsSent > 0 && PacketsReceived > 0, which a device that never
connected can never satisfy.

Also fix the ping failure message, which wrapped an always-nil lastErr
and rendered as "failed to ping after 3 retries: %!w(<nil>)" for every
connectivity failure. Report the packet counts instead.
Excluding not-ready devices from the failure stats opened a fail-open
path: with every device excluded, totalDevices is 0, the overall rate is
NaN, and NaN > threshold is false, so a run that tested nothing passed.
Reachable when a fleet is drained metro-wide or newly created (devices
are created with max_users=0 and unlocked on activation).

Report which devices were skipped so an operator can tell "3 devices
skipped" from "3 devices tested and fine", and fail when nothing was
left to rate.
@ben-dz
ben-dz requested review from bgm-malbeclabs and nikw9944 and removed request for nikw9944 August 10, 2026 15:51
@ben-dz
ben-dz marked this pull request as ready for review August 10, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant