fix(collect): stop false fleet-wide offline alerts - #162
Open
gsanchietti wants to merge 5 commits into
Open
Conversation
edospadoni
deployed
to
fix_link_failed - my-collect-qa PR #162
July 29, 2026 08:00 — with 
Render
Active
Render
Active
Contributor
|
🔗 Redirect URIs Added to Logto The following redirect URIs have been automatically added to the Logto application configuration: Redirect URIs:
Post-logout redirect URIs:
These will be automatically removed when the PR is closed or merged. |
gsanchietti
marked this pull request as ready for review
July 29, 2026 08:07
gsanchietti
force-pushed
the
fix_link_failed
branch
from
July 29, 2026 08:12
99739a3 to
1f7f658
Compare
edospadoni
deployed
to
fix_link_failed - my-collect-qa PR #162
July 29, 2026 08:12 — with
Render
Active
Render
Active
gsanchietti
marked this pull request as draft
July 29, 2026 08:12
Systems send heartbeats in synchronized waves, so when collect stops recording them every affected system looks offline at the same instant and each raises a critical LinkFailed alert to its customer. A single gap can therefore page the whole fleet for a fault on our side. Machines that are really dead stop beating at arbitrary moments, so a tight cluster of identical last_heartbeat values is our own fault, not theirs. Detect that cluster and skip those alerts, while systems that went quiet on their own still alert normally. Assisted-by: Claude Code:claude-opus-5
Systems beat roughly every 10 minutes, so the old 20 minute timeout tolerated only two missed beats. Two lost beats in a row was enough to mark the entire fleet offline at once. 30 minutes allows three, so one missed beat plus a late one no longer trips it. Docs still claimed 10 minutes, which has been wrong for a while. Assisted-by: Claude Code:claude-opus-5
Alertmanager merges an incoming alert over the one it clears, keeping only the original start time. Our resolve carried no text, so it blanked the description on every recovery — the "all clear" email and the alert history entry both ended up empty. Give the resolve its own text. Labels are untouched, so it still matches and clears the right alert. Assisted-by: Claude Code:claude-opus-5
Alert history read a "summary" field that nothing ever writes: every alert source uses the per-language summary_en and summary_it instead. The column was empty for every row ever saved. Fall back to those. Assisted-by: Claude Code:claude-opus-5
A heartbeat stall was invisible in production: the only trace sat at debug level, and the heartbeat queue was the one queue whose size was never measured. Diagnosing one meant working backwards from the alerts it caused. Report throughput and queue size every 5 minutes at info instead. Assisted-by: Claude Code:claude-opus-5
gsanchietti
force-pushed
the
fix_link_failed
branch
from
July 29, 2026 08:16
1f7f658 to
d9b0488
Compare
edospadoni
deployed
to
fix_link_failed - my-collect-qa PR #162
July 29, 2026 08:16 — with
Render
Active
Render
Active
gsanchietti
marked this pull request as ready for review
July 29, 2026 08:17
edospadoni
deployed
to
fix_link_failed - my-collect-qa PR #162
July 29, 2026 15:01 — with
Render
Active
Render
Active
edospadoni
force-pushed
the
fix_link_failed
branch
from
July 30, 2026 04:52
4aef4f6 to
d9b0488
Compare
edospadoni
deployed
to
fix_link_failed - my-collect-qa PR #162
July 30, 2026 04:52 — with
Render
Active
Render
Active
Member
|
down deploy |
Contributor
|
💤 Preview environment suspended Compute billing stopped for:
Skipped (not found):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A gap in our own heartbeat recording used to make the whole fleet look
offline at once, sending a critical alert to every customer for a fault on
our side. Collect now recognises that pattern and stays quiet, while
systems that genuinely went offline still alert as before.
Also fixes two long-standing blanks in alert data: the "all clear" email
and the saved alert history were both losing their description text.
How to test
Stop heartbeating from a test system and confirm a single LinkFailed alert
arrives after the timeout, that the recovery email is no longer empty, and
that the saved history row has a summary.