fix(ai-gateway): apply provider policy before Vercel routing

[chrarnoldus]

Summary

• resolve organization and group provider policies before managed gateway selection
• check Vercel inference-provider availability against the effective allow-list
• keep direct BYOK, custom LLM, experiment, and Vercel BYOK routes exempt from the extra policy lookup
• cache access decisions per effective model and cover the ordering with a route regression test

Testing

• CI only, as requested

[chrarnoldus]

@kilocode-bot please review this PR. The automated review check failed with Agent wrapper failed while processing the message and produced no review comments.

[kilo-code-bot]

Review: ordering fix looks right, a few things to tighten

I traced the ordering claim and it holds: getProvider early-returns for direct BYOK, Vercel BYOK, experiments and kilo-internal/... custom LLMs before the Vercel branch, so getRoutingProviderConfig is structurally unreachable on those routes — the exemption doesn't depend on the caller remembering anything. And gating on effectiveProviderConfig is consistent with what actually ends up on the wire, because the access-check block later does requestBodyParsed.body.provider = effectiveProviderConfig (a full replacement, so the caller's ignore/order/zdr were already being dropped when an org policy applies) and convertProviderOptions reads that same replaced value. Also confirmed eligibleProviderRoutes and provider_allow_list are both OpenRouter provider slugs, so the intersection and the later openRouterToVercelInferenceProviderId mapping line up.

No blocking correctness bug found. Main concerns, in order:

1. The policy lookup is eager, not lazy — await getRoutingProviderConfig?.() sits in the argument list, so it resolves before shouldRouteToVercel runs any of its cheap gates. See the inline comment on get-provider.ts; this is the one I'd actually change.
2. Silent fallback via default parameter in shouldRouteToVercel — an explicitly-passed undefined means "use the unpoliced request body". Fine today, fragile as a contract.
3. Two undocumented behavior changes in the rewritten denial guard (empty org provider_allow_list, and efficientPoolBlockedResponse now covering the empty-intersection case).
4. Failure-mode shift: because provider selection now awaits both balanceAndSettingsPromise and organizationGroupPolicyPromise, a transient failure in either now rejects getProvider (→ 500) on paths that previously returned a specific response before ever touching the policy — unavailable model, rules-engine block, unsupported api kind. Failing closed on a policy read is defensible; just worth being deliberate about it, and note the void organizationGroupPolicyPromise.catch(() => {}) guard no longer describes all consumers.

Test coverage gaps

The one added route test asserts only that the callback exists on the first getProvider call and resolves to { only: [...] }. Not covered:

• shouldRouteToVercel's new parameter has no unit test at all — vercel/index.test.ts only exercises the helpers. The highest-value test for this PR is: policy only contains a provider with no compatible Vercel inference provider ⇒ false, even though request.body.provider is empty. That is precisely the bug being fixed, and right now nothing pins it.
• The intersection path (org provider_allow_list ∩ group eligibleProviderRoutes) reaching the callback — the route test's getEffectiveModelDecision mock returns { allowed: true } with no eligibleProviderRoutes, so the interesting branch in resolveAccessCheck is untested.
• Anonymous requests ⇒ getRoutingProviderConfig is undefined.
• The quarantine-3 call site resolving the policy for the overridden model id — that's the only reason the cache is keyed by model, and it's the case most likely to regress.
• The stated caching guarantee itself (e.g. getEffectiveModelDecision called once per distinct model, twice across a quarantine downgrade).

Performance-wise the memoization is sound: at most two entries per request, no cross-request state, no staleness window, and the map holds the promise rather than the resolved value so concurrent callers share one lookup.

[kilo-code-bot]

The follow-up review completed successfully this time — no need to retry. Summary of what happened: the earlier automated review run hit an "Agent wrapper failed" error and posted nothing, but this run succeeded.

Posted: review with summary + 6 inline comments

Key takeaways:

• Core fix verified correct: exemptions are structural (early returns before Vercel branch), and gating against effectiveProviderConfig matches what's actually sent downstream.
• Main concern: getRoutingProviderConfig is now awaited eagerly for every request, before cheaper checks (shouldRouteToVercel, routing %, abuse classification) run — contradicts the PR's own JSDoc/comment about overlapping the DB read with later work. Suggested passing the thunk down instead.
• Other flags: default-param fallback (= request.body.provider) fails open on future undefined regressions; two undocumented behavior changes (empty provider_allow_list → blanket 403; empty intersection now returns a different error response); closure reads a mutable let instead of its own argument; DB failures now surface as 500s where 403/404 used to short-circuit first; and test gaps around shouldRouteToVercel's new param, group-intersection, anonymous case, and a relaxed toMatchObject assertion.

No further action needed from me unless you want me to kick off a follow-up Cloud Agent session to address any of these.

[kilo-code-bot]

Pulling the one item I'd actually block on out of the review thread, since it's easy to lose among the nits.

Make the policy lookup lazy instead of eager

getRoutingProviderConfig is typed as a thunk, and the JSDoc says it "resolves organization/group provider policy only when selecting a managed gateway" — but it's awaited in the argument position:

(await shouldRouteToVercel(
  requestedModel,
  request,
  taskId || user.id,
  await getRoutingProviderConfig?.()   // <- resolved before shouldRouteToVercel runs at all
))

So the laziness the type suggests doesn't exist. For every non-anonymous request that gets past the BYOK / custom-LLM / experiment early returns, this now resolves balanceAndSettingsPromise and organizationGroupPolicyPromise + getEffectiveModelDecision before shouldRouteToVercel has checked any of its three cheap gates:

• isVercelRoutingOptOut(...) — model is on the opt-out list
• passesVercelRoutingPercentage(...) — most traffic fails this today
• vercelModels.has(vercelModelId) — model isn't on Vercel at all

The provider config is only consulted after all three pass, and even then only inside if (provider && (provider.only || provider.ignore?.length)). The large majority of requests reaching this line never need it, and they now pay for it on the critical path — ahead of classifyAbuse, which starts only after getProvider resolves.

It also removes a property the previous code called out explicitly. The comment this PR deleted:

// Started right after auth so the DB read overlapped the work above; the
// decision itself is in-memory against the cached provider index.

That overlap was the reason organizationGroupPolicyPromise is kicked off right after auth. Awaiting it during provider selection collapses it back onto the serial path.

Suggested shape — keep the thunk intact through shouldRouteToVercel and call it at the point of use:

export async function shouldRouteToVercel(
  requestedModel: string,
  request: GatewayRequest,
  randomSeed: string,
  getRoutingProviderConfig?: () => Promise<OpenRouterProviderConfig | undefined>
) {
  // ... opt-out, percentage, and vercelModels.has checks unchanged ...

  const provider = (await getRoutingProviderConfig?.()) ?? request.body.provider;
  if (provider && (provider.only || provider.ignore?.length)) {
    // ... unchanged ...
  }
}

and at the call site simply getRoutingProviderConfig instead of await getRoutingProviderConfig?.(). The memoization in getAccessCheck already makes the later await getAccessCheck(...) in the access-check block free, so nothing is lost by deferring — the only change is that requests which never reach the provider-preference branch stop waiting on two DB reads they don't use.

Two side benefits: the JSDoc becomes accurate, and the ?? request.body.provider fallback moves from a default parameter into an explicit expression at the decision site (which addresses the "silently uses the unpoliced body" concern I raised separately).

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (1 files)

• apps/web/src/app/api/openrouter/[...path]/route.test.ts

Previous Review Summaries (4 snapshots, latest commit 04a5301)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 04a5301)

Status: 2 Issues Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
apps/web/src/app/api/openrouter/[...path]/route.test.ts	319	Abuse-block test now expects 404; rulesEngineBlockResponse() still returns 403
apps/web/src/app/api/openrouter/[...path]/route.test.ts	433	Test expects 403 but model denial returns 404; skip assertion is vacuous without organizationId

Files Reviewed (1 files)

• apps/web/src/app/api/openrouter/[...path]/route.test.ts - 2 issues

Fix these issues in Kilo Cloud

Previous review (commit e9383ba)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	1
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
apps/web/src/app/api/openrouter/[...path]/route.test.ts	433	Test expects 403 but model denial returns 404; skip assertion is vacuous without organizationId

Files Reviewed (2 files)

• apps/web/src/app/api/openrouter/[...path]/route.test.ts - 1 issue
• apps/web/src/app/api/openrouter/[...path]/route.ts

Fix these issues in Kilo Cloud

Previous review (commit 5fa207f)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	1
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
apps/web/src/app/api/openrouter/[...path]/route.test.ts	433	Test expects 403 but model denial returns 404; skip assertion is vacuous without organizationId

Files Reviewed (2 files)

• apps/web/src/app/api/openrouter/[...path]/route.test.ts - 1 issue
• apps/web/src/app/api/openrouter/[...path]/route.ts

Fix these issues in Kilo Cloud

Previous review (commit 1b79795)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (5 files)

• apps/web/src/app/api/openrouter/[...path]/route.ts
• apps/web/src/app/api/openrouter/[...path]/route.test.ts
• apps/web/src/lib/ai-gateway/providers/get-provider.ts
• apps/web/src/lib/ai-gateway/providers/vercel/index.ts
• apps/web/src/lib/ai-gateway/providers/vercel/index.test.ts

---

_{Reviewed by grok-4.6 · Input: 161.7K · Output: 4.7K · Cached: 122.2K}

_{Review guidance: REVIEW.md from base branch main}

[evanjacobson]

Please fix tests