Skip to content

obs(api): record team sandbox index SET size as a histogram - #3604

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:obs/redis-team-index-size-gauge
Open

obs(api): record team sandbox index SET size as a histogram#3604
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:obs/redis-team-index-size-gauge

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Summary

Adds api.redis_storage.team_index.size (histogram, unit {entry}).

Once per heal pass (every 5 min) a new recordTeamIndexSizes helper pipelines SCARD over every team's sandbox index SET and records each cardinality as a histogram observation. Overhead is one ZRange(globalTeamsSet) + N SCARD (pipelined) per interval — negligible next to the forEachSandboxBatch SSCAN scan that already runs at the same cadence.

Why

The team index SET can accumulate stale entries when sandbox keys disappear outside the normal Remove() path (Redis key eviction, crash, etc.) — see #3566. Before this change there was no way to observe whether accumulation was happening short of running SCARD manually or waiting for the Admin API.

With the histogram:

Changes

File Change
shared/pkg/telemetry/meters.go Add ApiRedisStorageTeamIndexSize histogram constant, desc, unit
redis/main.go Add teamIndexSize metric.Int64Histogram to expirationIndexMetrics; wire up in newExpirationIndexMetrics
redis/heal.go Call recordTeamIndexSizes at end of each heal pass; add recordTeamIndexSizes method

Test plan

  • Unit: recordTeamIndexSizes records correct SCARD values per team (testcontainers Redis)
  • Unit: teams with SCARD == 0 are not recorded (avoids polluting histogram with empty-set noise)
  • Integration: histogram emits observations after a heal pass completes

/cc @jakubno @dobrac @ValentaTomas

Add api.redis_storage.team_index.size (histogram, unit: {entry}).

Once per heal pass (every 5 min) recordTeamIndexSizes pipelines SCARD
over every team index SET and records each cardinality as a histogram
observation. Overhead is one ZRange + N SCARD (pipelined) per interval,
negligible next to the forEachSandboxBatch scan that precedes it.

Use the p99/max to detect unbounded accumulation of stale entries:
healthy steady state tracks active sandbox counts; a widening gap
between p99 and the running sandbox count indicates stale entries are
not being cleaned up by Remove().

Related: e2b-dev#3566 (stale-entry root cause), e2b-dev#3567 (orphan SREM fix)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants