Skip to content

feat: add rate of change as alert trigger condition#1943

Draft
dhable wants to merge 4 commits intomainfrom
feature/rate-of-change-alerts
Draft

feat: add rate of change as alert trigger condition#1943
dhable wants to merge 4 commits intomainfrom
feature/rate-of-change-alerts

Conversation

@dhable
Copy link
Contributor

@dhable dhable commented Mar 18, 2026

Summary

  • Adds Rate of Change as a new alert condition type alongside the existing threshold rules, modeled after Datadog's Change Alerts, Grafana's diff/percent_diff reducers, and Splunk's Sudden Change detectors.
  • Supports both absolute and percentage change modes, comparing the current evaluation window to the immediately preceding window.
  • Works for both saved search alerts and dashboard tile alerts.
  • The /alerts page summary now shows the condition type (Threshold vs Rate of Change) and whether percentage change is in use.

Changes

Data model (common-utils, api/models, api/utils/zod):

  • New enums: AlertConditionType (threshold | rate_of_change), AlertChangeType (absolute | percentage)
  • Zod validation requiring changeType when conditionType is rate_of_change
  • Mongoose schema and IAlert interface updated with optional fields for backward compatibility

Evaluation engine (api/tasks/checkAlerts/index.ts):

  • Extended date range to fetch 2 windows for rate-of-change comparison
  • New computeRateOfChange() function (absolute and percentage modes)
  • processAlert loop tracks previous bucket values per group, skips baseline bucket, computes change

Frontend (app/):

  • Condition type and change type selectors in both saved search and dashboard tile alert forms
  • Alert card summary on /alerts page shows condition type label and % suffix
  • AlertPreviewChart accepts new props for rate-of-change context

API (api/routers, api/utils/externalApi, api/controllers, api/tasks/checkAlerts/template):

  • GET /alerts response includes conditionType and changeType
  • External API translation includes new fields
  • Notification titles reflect rate-of-change context

Test plan

  • Unit tests: schema validation (11 tests in alertTypes.test.ts), computeRateOfChange (12 tests), external API translation (2 tests)
  • Integration tests: API CRUD with rate-of-change alerts (5 tests in alerts.test.ts, 2 in external API), ClickHouse evaluation (4 new tests in singleInvocationAlert.test.ts covering absolute fires, percentage fires, below-threshold suppression, empty-baseline)
  • E2E tests: 2 new Playwright tests for saved search and dashboard tile rate-of-change alert creation
  • Manual: verify on dev stack that rate-of-change alerts display correctly and evaluate properly

Made with Cursor

Add a new "Rate of Change" condition type for alerts alongside the
existing threshold rules. This compares the current evaluation window's
value to the immediately preceding window and fires when the absolute
or percentage change exceeds the configured threshold -- similar to
Datadog's Change Alert, Grafana's diff/percent_diff reducers, and
Splunk's Sudden Change detector.

- New enums: AlertConditionType (threshold | rate_of_change),
  AlertChangeType (absolute | percentage)
- Zod validation requiring changeType when conditionType is rate_of_change
- Evaluation engine: extended date range for 2-window lookback,
  computeRateOfChange function, baseline bucket tracking in processAlert
- Frontend: condition type / change type selectors in saved search and
  dashboard tile alert forms, improved alert card summary on /alerts page
- Notification templates updated for rate-of-change context
- Unit tests (schema validation, computeRateOfChange, external API)
- Integration tests (API CRUD, ClickHouse evaluation with 4 scenarios)
- E2E Playwright tests for saved search and dashboard tile flows

Made-with: Cursor
@changeset-bot
Copy link

changeset-bot bot commented Mar 18, 2026

🦋 Changeset detected

Latest commit: 89d0617

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Minor
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel
Copy link

vercel bot commented Mar 18, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hyperdx-oss Ready Ready Preview, Comment Mar 19, 2026 4:47pm

Request Review

@github-actions
Copy link
Contributor

github-actions bot commented Mar 18, 2026

E2E Test Results

All tests passed • 93 passed • 3 skipped • 949s

Status Count
✅ Passed 93
❌ Failed 0
⚠️ Flaky 2
⏭️ Skipped 3

Tests ran across 4 shards in parallel.

View full report →

AlertSource,
AlertThresholdType,
} from '@/models/alert';
import { SavedSearch } from '@/models/savedSearch';
AlertThresholdType,
} from '@/models/alert';
import { SavedSearch } from '@/models/savedSearch';
import { Source } from '@/models/source';
Comment on lines +23 to +28
import Alert, {
AlertChangeType,
AlertConditionType,
AlertSource,
AlertThresholdType,
} from '@/models/alert';
- Remove unused imports (SavedSearch, Source, AlertConditionType)
- Fix backward-compat test: Mongoose default makes conditionType
  'threshold' on new alerts, not undefined
- Add conditionType to external API snapshot test expectation
- Fix E2E locator: use 'Rate of Change' text to avoid strict mode
  violation from ambiguous 'change' match

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant