e2e/qa: stop counting devices that cannot accept users as failures - #4168
Open
ben-dz wants to merge 3 commits into
Open
e2e/qa: stop counting devices that cannot accept users as failures#4168ben-dz wants to merge 3 commits into
ben-dz wants to merge 3 commits into
Conversation
The all-devices unicast test checked device readiness at four sites and logged "Ignoring <x> failure for device not ready for users", but each site incremented FailedTests before the check, so the carve-out only suppressed the log line while the device still counted as failed. Devices left activated with max_users=0 are drained on purpose and the CLI refuses the connect, so exclude them from ComputeFailureStats entirely rather than gating the counter: Success() also requires PacketsSent > 0 && PacketsReceived > 0, which a device that never connected can never satisfy. Also fix the ping failure message, which wrapped an always-nil lastErr and rendered as "failed to ping after 3 retries: %!w(<nil>)" for every connectivity failure. Report the packet counts instead.
Excluding not-ready devices from the failure stats opened a fail-open path: with every device excluded, totalDevices is 0, the overall rate is NaN, and NaN > threshold is false, so a run that tested nothing passed. Reachable when a fleet is drained metro-wide or newly created (devices are created with max_users=0 and unlocked on activation). Report which devices were skipped so an operator can tell "3 devices skipped" from "3 devices tested and fine", and fail when nothing was left to rate.
ben-dz
requested review from
bgm-malbeclabs and
nikw9944
and removed request for
nikw9944
August 10, 2026 15:51
ben-dz
marked this pull request as ready for review
August 10, 2026 15:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
Mainnet Beta QA failed 7 times over Aug 8-10; three of those (infra runs 31248009915, 31266248546, 31286128831) had one cause.
qa_alldevices_unicast_test.gochecks device readiness at five sites and logsIgnoring <x> failure for device not ready for users, but each site incrementsFailedTestsbefore that check, so the carve-out only suppressed the log line while the device still counted as failed.laconic-dfw-sw01,laconic-mia-sw01andlaconic-was-sw01have been activated withmax_users=0since Aug 6, and the CLI refuses those connects outright;cmh-mn-qa01draws from a 13-device pool, so three unusable devices produced a 21-29% per-host rate against the 20% gate.Gating the counter is not enough:
Success()also requiresPacketsSent > 0 && PacketsReceived > 0, which a device that never connected can never satisfy. So not-ready devices are excluded fromComputeFailureStatsentirely — including from the per-host denominator — behind a newDevice.Ready()that replaces the condition previously spelled out at six call sites.That exclusion opens a fail-open path worth reviewing closely: with every device excluded,
totalDevicesis 0, the overall rate isNaN, andNaN > thresholdis false, so a run that tested nothing would pass. Devices are created withmax_users=0until activation unlocks them, so a new or metro-drained fleet lands exactly there. The rate math is now guarded by at.Fatalf, andFailureStats.Skippedcarries the excluded codes into test output so "3 devices skipped" is distinguishable from "3 devices tested and fine".Separately,
client_unicast.gowrapped an always-nillastErr— the retry loop returns early on any error — so every connectivity failure this weekend readfailed to ping after 3 retries: %!w(<nil>). It now reports packet counts.Ready()is a deliberate subset of the program'sis_device_eligible_for_provisioning: a device atusers_count + reserved_seats >= max_usershits the same CLI rejection and still counts as a failure. That is the same class of false red one field over, but fixing it means re-adding the capacity check #3697 removed, so it is left as a follow-up.MaxUnicastUsersis not consulted — 0 there means "no per-type limit", the inverse of drained.Testing Verification
go test ./e2e/internal/qa/...andgolangci-lint run --build-tags=qa ./e2e/...(0 issues) pass. The four newTestComputeFailureStatssubtests were confirmed to fail against the unfixed aggregation and pass after; without the fix the mainnet shape reportstotal=6 failed=3where it should reporttotal=3 failed=0.TestPingFailureErrorasserts the message carries packet counts and never renders%!w. Existing fixtures in that file had zero-valuedStatus/MaxUsersand so were notReady(); they are now explicitly activated, which is why the diff touches subtests it does not otherwise change.Not verified: the
qa-tagged e2e tests need live fleet access and cannot run here, so the change is unexercised against the real fleet.TestQAAgentConnectivity/Pingfails in this container for lack of raw-socket privileges, onmainas well as this branch.A reviewer also flagged that a not-ready device still consumes a batch host slot, and on the two-host devnet run that can trip the existing
fewer than 2 clients connectedfatal regardless of this fix. That predates this change and did not fire in any weekend run; filing separately rather than widening this diff.