diag(ai-gateway): measure whether pool slots are leaked

[RSO]

Summary

Item 2 of the follow-ups from #5098. This is a measurement, not a fix — it exists to decide between two explanations that call for opposite remedies.

#5098's instrumentation established that pool acquisition dominates /api/internal/usage/record: idle == 0 on 96–98% of sampled requests with up to 245 queued against max: 10, while PostgreSQL reports zero active queries and Supavisor reports client_waiting: 0 with pool_size 320 against server_active 67. Connections are checked out without doing work. Either:

1. Fluid compute packs high concurrency onto few instances and ten connections is simply too few. Fix: more connections.
2. Slots are leaked and never return. drizzle-orm 0.45.2 issues BEGIN outside its try/finally, so a failed BEGIN never reaches the release() in the finally and burns that slot for the life of the process. Ten of those kill an instance permanently. Fix: release on a failed BEGIN.

Getting this wrong is expensive in both directions. If it's a leak, raising max postpones and conceals it — and max at drizzle.ts:93 is a single module-level constant shared by every route in both Vercel projects, whose inline comment records that raising it previously exhausted Supabase's connection limit across ~2,200 instances.

The arithmetic is what makes this worth measuring rather than assuming: at ~47 req/s with Frankfurt-local statements of 2–10 ms, you need roughly 5 connections, not 245 queued.

The discriminator

The low-water mark of checked-out connections. A healthy pool returns to zero checked out whenever the instance goes quiet, and idleTimeoutMillis of 5 s then closes the idle clients. A leaked client is never returned to _idle, so it counts as checked out forever and the low-water mark cannot fall below the number of leaks.

A low-water mark rather than an instantaneous reading, because this pool is shared with every other route on the instance — /api/profile, /api/defaults and the rest. One quiet moment resets it, so unrelated concurrent traffic cannot inflate it. An instantaneous checked_out > 0 proves nothing; a floor that climbs does.

What it adds

All read-only counters updated on pg-pool events. No queries, no timers.

• acquires / releases / connects / removes, plus outstanding as a cross-check that should track pool.totalCount - pool.idleCount. A persistent divergence means the accounting is wrong and nothing else should be trusted.
• min_checked_out and max_checked_out.
• begin_failures — counted where the retry loop already inspects the error. Under the drizzle bug each one is exactly one leaked slot, which makes it directly comparable to min_checked_out. Matching on the message is safe only for BEGIN: it takes no parameters, so unlike every other statement in this path the message carries no prompt text or client IP.
• ms_since_last_request and checked_out_at_entry, read before the request acquires anything.

Emission now also triggers on the first request after 5 s of instance quiet. Those requests are fast, so the duration threshold would never surface them — yet they are the only moment a leaked slot is distinguishable from a busy one.

Counters do not attach under NODE_ENV=test, matching how drizzle.ts already special-cases the test pool.

Verification

Automated: tsgo --noEmit -p apps/web/tsconfig.json clean; ./scripts/lint-all.sh 0 warnings 0 errors; 17 new probe tests and 6 new emission/clock tests; 923 passing across 70 suites in src/lib/ai-gateway + src/lib/drizzle.test.ts + src/lib/db-pool-leak-probe.test.ts + src/app/api/internal; pnpm format and git diff --check clean. Targeted verification, not pnpm validate.

Manual: none, and it cannot be meaningful here. The behaviour under investigation only appears under production concurrency; locally the pool never exceeds a couple of connections, so a local run would show min_checked_out: 0 whether or not the bug exists. The unit tests cover the classifier and the arithmetic; only production can answer the question.

• After deploy, read the query in Reviewer Notes and reach a verdict.

Visual Changes

N/A

Reviewer Notes

How to read the result. The verdict is a comparison, not a threshold:

['vercel'] | where message startswith '{"type":"usage_record_timing"'
| extend quiet=toint(extract('"ms_since_last_request":([0-9]+)',1,message)),
         entry=toint(extract('"checked_out_at_entry":([0-9]+)',1,message)),
         minco=toint(extract('"min_checked_out":([0-9]+)',1,message)),
         beginf=toint(extract('"begin_failures":([0-9]+)',1,message)),
         uptime=toint(extract('"instance_uptime_ms":([0-9]+)',1,message))
| where quiet >= 5000
| summarize samples=count(), p50_entry=percentile(entry,50), max_entry=max(entry),
            p50_min=percentile(minco,50), max_min=max(minco),
            max_begin=max(beginf), p95_uptime=percentile(uptime,95)
  by bin(_time, 5m)

• min_checked_out at 0 and checked_out_at_entry near 0 on quiet instances → no leak. The pool is genuinely busy, and sizing (ideally a dedicated pool, to bound the blast radius) is the right next step.
• min_checked_out above 0 and climbing with uptime, tracking begin_failures → leak confirmed. The fix is wrapping db.transaction so a failed BEGIN releases its client, not more connections.
• min_checked_out above 0 but begin_failures at 0 → a leak from some other path, and I would want to know that before touching anything.

Things worth challenging:

• min_checked_out is process-lifetime and never reset, so on a long-lived instance a single early quiet moment pins it at 0 forever and could mask a leak that started later. Comparing it against checked_out_at_entry on quiet requests is the mitigation, but a windowed low-water mark would be strictly better if this proves inconclusive.
• The probe attaches four listeners to the shared primary pool at module load. They are counter increments, but it is a process-level side effect in a hot shared module.
• begin_failures is only incremented on the usage-write path. A BEGIN failure anywhere else in the process leaks a slot without being counted, which is exactly the third case above.
• This touches route.ts in the same import block as perf(ai-gateway): drop the usage-record dedupe pre-check #5105, so expect a trivial conflict depending on merge order. Both changes are independent; either can land first.

[kilo-code-bot]

SUGGESTION: A 400 response consumes the quiet-instance observation point without emitting it

noteRequestStart() runs for every authorized request, but a request that fails schema validation returns 400 before reportTiming is ever called. If the first request after a >=5s quiet period is a 400, it resets lastRequestAtMs, so the next (fast) request no longer meets QUIET_INSTANCE_MS and the one moment a leaked slot is distinguishable from a busy one is lost. Rare on this internal endpoint, but if these observation points matter, consider updating the clock only on requests that can actually emit (e.g. moving the call after validation, or resetting it on the 400 path).

---

Reply with @kilocode-bot fix it to have Kilo Code address this issue.

[kilo-code-bot]

Code Review Summary

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	0
SUGGESTION	1

Issue Details (click to expand)

SUGGESTION

File	Line	Issue
apps/web/src/app/api/internal/usage/record/route.ts	54	A 400 validation failure consumes the quiet-instance observation point via noteRequestStart() without ever emitting it, losing the leak-observation window

Files Reviewed (6 files)

• apps/web/src/app/api/internal/usage/record/route.ts - 1 issue
• apps/web/src/lib/ai-gateway/processUsage.ts - 0 issues (begin-failure counting placed correctly before the retry/conflict decision; no raw error logging)
• apps/web/src/lib/ai-gateway/usage-record-diagnostics.ts - 0 issues (new signature backwards compatible; no stale callers)
• apps/web/src/lib/ai-gateway/usage-record-diagnostics.test.ts - 0 issues
• apps/web/src/lib/db-pool-leak-probe.ts - 0 issues (4 pool listeners attach once at module load, bounded fixed-shape counters, no timers — no memory leak introduced)
• apps/web/src/lib/db-pool-leak-probe.test.ts - 0 issues

Fix these issues in Kilo Cloud

---

_{Reviewed by kimi-k3 · Input: 86.1K · Output: 22.6K · Cached: 951.3K}

_{Review guidance: REVIEW.md from base branch main}

[jrf0110]

Probably not, but do we need something to teardown these listeners? Perhaps if drizzle ever recreates the pool?