Skip to content

fix(ai-gateway): shed usage writes when the pool is full - #5115

Open
jrf0110 wants to merge 1 commit into
mainfrom
fix/usage-record-pool-backpressure
Open

fix(ai-gateway): shed usage writes when the pool is full#5115
jrf0110 wants to merge 1 commit into
mainfrom
fix/usage-record-pool-backpressure

Conversation

@jrf0110

@jrf0110 jrf0110 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #5034, #5098, and #5105. The shared Sentry event for insertUsageRecord failed (code=unknown constraint=none) shows the failure originates at pg-pool acquisition inside Drizzle, before BEGIN or any usage SQL. The paired usage record handoff to primary region failed event is the caller reporting that failed Frankfurt delivery.

Production diagnostics already show the local pool at idle=0 on 96-98% of sampled requests, with as many as 245 waiters against max=10. Letting every handoff join that queue causes connection-acquisition timeouts and keeps unrelated work on the same Fluid instance under pressure.

This change adds admission control to POST /api/internal/usage/record: after authentication, it returns a retryable 503 before reading the body or starting a write when the primary pool has no idle connection and is already at its configured maximum. The existing client retries 503 because this response proves no write began, preserving idempotency. Requests are still admitted when an idle connection exists or the pool can grow. If the pool maximum is unavailable, the check fails open.

The third linked event, TRPCError: Usage data temporarily unavailable, is a separate Snowflake query exceeding its 5-second timeout. Its stack does not touch this PostgreSQL path, so this PR does not claim to fix it.

Verification

  • pnpm --filter web test -- --runInBand src/lib/ai-gateway/usage-record-diagnostics.test.ts src/lib/ai-gateway/usage-record-client.test.ts (53 tests passed)
  • pnpm --filter web typecheck
  • pnpm --filter web lint (0 warnings, 0 errors)
  • pnpm format
  • git diff --check

Visual Changes

N/A

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant