fix(ai-gateway): remove overbroad model-slug rule (closes #60)#72
Open
ramseydallal-lab wants to merge 1 commit intovercel:mainfrom
Open
Conversation
The `\d+-\d+[)'"]` validation rule on the ai-gateway skill was
designed to catch typo'd hyphenated model slugs in `gateway('…')`
calls. In practice the rule:
1. Contradicts Anthropic's canonical model IDs, which are
hyphenated. From https://docs.anthropic.com/en/docs/about-claude/models
(verified 2026-04-30):
| Claude API ID | claude-sonnet-4-6 |
| Claude API alias | claude-sonnet-4-6 |
Direct provider SDK calls — `anthropic('claude-sonnet-4-6')` —
are correct AS WRITTEN with hyphens. The rule fires on every
such call as a false positive.
2. False-positives outside model-slug context entirely. The pattern
matches any digit-hyphen-digit followed by `)`, `'`, or `"`,
so date strings ("01-15-2024"), package versions, and similar
text trip the rule. Issue vercel#60 documents this across multiple
real repos.
3. Trains operators to dismiss validator output. Issue vercel#38 shows
the compounding noise problem; vercel#59 documents the validator's
own SKILL.md tripping its own rules. Each individual noise
source compounds.
The remaining provider/-prefix rule
(`gateway\(['"][^'"/]+['"]\)`) still enforces the actual hard
constraint — that gateway calls use `provider/model` format.
Slug typos within that format will surface from the gateway
itself with a clearer error than this lint can produce.
Changes
-------
- skills/ai-gateway/SKILL.md
- Remove the validate-rule entry (kept as a comment with vercel#60
pointer for future archeology).
- Rewrite the "Model Slug Rules" prose to document the real
asymmetry: gateway-catalog form uses dots
(anthropic/claude-opus-4.6 per Vercel's models page), while
direct provider SDK calls use the provider's native IDs
(Anthropic publishes hyphens). Both are correct in their
contexts; the validator should not assert one onto the other.
- generated/skill-manifest.json
- Drop the corresponding rule entry (regenerated artifact, but
committed in this repo per existing pattern).
- tests/validate-rules.test.ts
- Invert the direct assertion (test at line 426): hyphenated
AND dotted forms now both pass without violation.
- Update three multi-skill overlap tests to use
`gateway('opaque')` (missing-provider rule, severity:error)
instead of the deleted slug rule as the ai-gateway trigger.
- Update the line-number test's message-includes string to
match the still-present "missing provider/ prefix" rule.
- Update the negative-test's comment + extend coverage to both
forms.
- tests/posttooluse-validate.test.ts
- Convert "detects gateway from 'ai' with hyphenated model slug"
test from "expect 'dots not hyphens' violation" to "expect
NO 'dots not hyphens' violation, for both hyphen and dot
forms." ai-gateway skill is still expected to be matched on
the file via the `'ai'` import pattern.
- tests/posttooluse-chain.test.ts
- Cosmetic: fix the test fixture string `anthropic('claude-sonnet-4.6')`
to the canonical hyphen form `anthropic('claude-sonnet-4-6')`.
Test logic unchanged.
Verification
------------
The rule's removal was already validated against a real-world
failure: a production app shipped `anthropic('claude-sonnet-4.6')`
(dotted, matching what the validator told them was correct), the
Anthropic API rejected the model ID, every critique call surfaced
as `ai_error` for end users. Switching to `claude-sonnet-4-6`
(hyphens) — what this rule had been flagging as wrong — fixed it.
Local tests not run (bun not installed in the patch environment);
relying on CI.
Closes vercel#60.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the
\d+-\d+[)'"]model-slug validation rule on theai-gatewayskill. The rule was supposed to catch typo'd hyphenated slugs ingateway('…')calls but in practice produces false positives that compound into the broader noise problem documented in #38 / #59.Closes #60.
The bug, by example
Authoritative Anthropic docs (platform.claude.com/docs/en/docs/about-claude/models, verified 2026-04-30):
| Claude API ID |
claude-sonnet-4-6|| Claude API alias |
claude-sonnet-4-6|| Pricing | $3 / $15 per MTok |
Anthropic's canonical model identifiers use hyphens. They have used hyphens consistently across
claude-3-5-sonnet,claude-opus-4-1,claude-sonnet-4-5,claude-opus-4-7— every model.A real production app shipped this code:
Anthropic API responded
model_not_found, every coach/critique call surfaced asai_errorto end users for several days. Fixing the slug toclaude-sonnet-4-6(the form the validator was flagging as wrong) restored service.Three failure modes of the deleted rule
Inverts Anthropic's actual convention. Direct provider SDK calls (
anthropic('…'),openai('…')) take the provider's native model IDs, which Anthropic publishes with hyphens. The rule fires on every such call as a false positive.Matches outside model-slug context. The pattern
\d+-\d+[)'"]greedily matches any digit-hyphen-digit followed by),', or". Real-world hits include date strings ("01-15-2024"), package versions, file paths, semver ranges in JSDoc — anywhere the substring appears in code.Trains operators to dismiss validator output. Combined with the over-matching documented in posttooluse-validate: regex matches fire inside JSDoc and line comments — false positives on documentation prose #59 (validator scans full file content and re-matches its own SKILL.md rule strings), users learn to ignore all plugin warnings, losing the value of the rules that are correct.
Why the gateway-vs-direct asymmetry exists, and why a rule can't fix it
The Vercel AI Gateway's catalog uses dots in version numbers. Per vercel/docs/ai-gateway/models-and-providers, the Gateway example is:
So both these calls are correct in their respective contexts:
A regex on the file content can't tell which surface a string is destined for, so any version of this rule will be wrong half the time. The remaining provider/-prefix rule (
gateway\(['"][^'"/]+['"]\)) still enforces the actual hard constraint: gateway calls must useprovider/modelformat. Slug typos within that format will surface from the gateway itself with a clearer error than a regex lint can produce.Files changed
skills/ai-gateway/SKILL.mdvalidate:rule (kept as comment with #60 pointer); rewrite Model Slug Rules prose to document gateway-vs-direct asymmetry honestlygenerated/skill-manifest.jsontests/validate-rules.test.tsgateway('opaque')(existing missing-provider rule) as the ai-gateway trigger; extend the negative test to cover both formstests/posttooluse-validate.test.tstests/posttooluse-chain.test.tsanthropic('claude-sonnet-4.6')→anthropic('claude-sonnet-4-6')in test fixture (test logic unchanged)Test plan
git diff --stat— 5 files, +73/-54bun test— local environment doesn't have bun; relying on PR CIOut of scope
@ai-sdk/anthropicdirect calls to AI Gateway — that's a downstream consumer concern, not the plugin's job to enforce.🤖 Generated with Claude Code