Skip to content

fix(ai-gateway): remove overbroad model-slug rule (closes #60)#72

Open
ramseydallal-lab wants to merge 1 commit intovercel:mainfrom
ramseydallal-lab:fix/issue-60-remove-overbroad-model-slug-rule
Open

fix(ai-gateway): remove overbroad model-slug rule (closes #60)#72
ramseydallal-lab wants to merge 1 commit intovercel:mainfrom
ramseydallal-lab:fix/issue-60-remove-overbroad-model-slug-rule

Conversation

@ramseydallal-lab
Copy link
Copy Markdown

Summary

Removes the \d+-\d+[)'"] model-slug validation rule on the ai-gateway skill. The rule was supposed to catch typo'd hyphenated slugs in gateway('…') calls but in practice produces false positives that compound into the broader noise problem documented in #38 / #59.

Closes #60.

The bug, by example

Authoritative Anthropic docs (platform.claude.com/docs/en/docs/about-claude/models, verified 2026-04-30):

| Claude API ID | claude-sonnet-4-6 |
| Claude API alias | claude-sonnet-4-6 |
| Pricing | $3 / $15 per MTok |

Anthropic's canonical model identifiers use hyphens. They have used hyphens consistently across claude-3-5-sonnet, claude-opus-4-1, claude-sonnet-4-5, claude-opus-4-7 — every model.

A real production app shipped this code:

import { anthropic } from '@ai-sdk/anthropic'
const result = await generateText({
  model: anthropic('claude-sonnet-4.6'),  // typo: dot, written this way because the validator said so
  ...
})

Anthropic API responded model_not_found, every coach/critique call surfaced as ai_error to end users for several days. Fixing the slug to claude-sonnet-4-6 (the form the validator was flagging as wrong) restored service.

Three failure modes of the deleted rule

  1. Inverts Anthropic's actual convention. Direct provider SDK calls (anthropic('…'), openai('…')) take the provider's native model IDs, which Anthropic publishes with hyphens. The rule fires on every such call as a false positive.

  2. Matches outside model-slug context. The pattern \d+-\d+[)'"] greedily matches any digit-hyphen-digit followed by ), ', or ". Real-world hits include date strings ("01-15-2024"), package versions, file paths, semver ranges in JSDoc — anywhere the substring appears in code.

  3. Trains operators to dismiss validator output. Combined with the over-matching documented in posttooluse-validate: regex matches fire inside JSDoc and line comments — false positives on documentation prose #59 (validator scans full file content and re-matches its own SKILL.md rule strings), users learn to ignore all plugin warnings, losing the value of the rules that are correct.

Why the gateway-vs-direct asymmetry exists, and why a rule can't fix it

The Vercel AI Gateway's catalog uses dots in version numbers. Per vercel/docs/ai-gateway/models-and-providers, the Gateway example is:

model: gateway('anthropic/claude-opus-4.6')

So both these calls are correct in their respective contexts:

gateway('anthropic/claude-opus-4.6')   // Gateway catalog form, dots
anthropic('claude-opus-4-7')           // Direct provider SDK, hyphens

A regex on the file content can't tell which surface a string is destined for, so any version of this rule will be wrong half the time. The remaining provider/-prefix rule (gateway\(['"][^'"/]+['"]\)) still enforces the actual hard constraint: gateway calls must use provider/model format. Slug typos within that format will surface from the gateway itself with a clearer error than a regex lint can produce.

Files changed

File Change
skills/ai-gateway/SKILL.md Remove the validate: rule (kept as comment with #60 pointer); rewrite Model Slug Rules prose to document gateway-vs-direct asymmetry honestly
generated/skill-manifest.json Drop the corresponding rule entry
tests/validate-rules.test.ts Invert the direct assertion (line 426) — both forms now must NOT raise the deleted message; update 3 multi-skill overlap tests to use gateway('opaque') (existing missing-provider rule) as the ai-gateway trigger; extend the negative test to cover both forms
tests/posttooluse-validate.test.ts Convert "detects gateway from 'ai' with hyphenated model slug" test to assert NO violation, looped over both hyphen and dot forms; ai-gateway skill match still expected
tests/posttooluse-chain.test.ts Cosmetic: fix anthropic('claude-sonnet-4.6')anthropic('claude-sonnet-4-6') in test fixture (test logic unchanged)

Test plan

  • git diff --stat — 5 files, +73/-54
  • Manually validated assertion polarity — for each touched test, confirmed expected violations match remaining ai-gateway rule set
  • CI: bun test — local environment doesn't have bun; relying on PR CI

Out of scope

  • The broader #59 (validator over-matches inside SKILL.md content) — separate refactor; this PR's scope is just the slug rule.
  • Migration from @ai-sdk/anthropic direct calls to AI Gateway — that's a downstream consumer concern, not the plugin's job to enforce.

🤖 Generated with Claude Code

The `\d+-\d+[)'"]` validation rule on the ai-gateway skill was
designed to catch typo'd hyphenated model slugs in `gateway('…')`
calls. In practice the rule:

1. Contradicts Anthropic's canonical model IDs, which are
   hyphenated. From https://docs.anthropic.com/en/docs/about-claude/models
   (verified 2026-04-30):

     | Claude API ID | claude-sonnet-4-6 |
     | Claude API alias | claude-sonnet-4-6 |

   Direct provider SDK calls — `anthropic('claude-sonnet-4-6')` —
   are correct AS WRITTEN with hyphens. The rule fires on every
   such call as a false positive.

2. False-positives outside model-slug context entirely. The pattern
   matches any digit-hyphen-digit followed by `)`, `'`, or `"`,
   so date strings ("01-15-2024"), package versions, and similar
   text trip the rule. Issue vercel#60 documents this across multiple
   real repos.

3. Trains operators to dismiss validator output. Issue vercel#38 shows
   the compounding noise problem; vercel#59 documents the validator's
   own SKILL.md tripping its own rules. Each individual noise
   source compounds.

The remaining provider/-prefix rule
(`gateway\(['"][^'"/]+['"]\)`) still enforces the actual hard
constraint — that gateway calls use `provider/model` format.
Slug typos within that format will surface from the gateway
itself with a clearer error than this lint can produce.

Changes
-------
- skills/ai-gateway/SKILL.md
  - Remove the validate-rule entry (kept as a comment with vercel#60
    pointer for future archeology).
  - Rewrite the "Model Slug Rules" prose to document the real
    asymmetry: gateway-catalog form uses dots
    (anthropic/claude-opus-4.6 per Vercel's models page), while
    direct provider SDK calls use the provider's native IDs
    (Anthropic publishes hyphens). Both are correct in their
    contexts; the validator should not assert one onto the other.
- generated/skill-manifest.json
  - Drop the corresponding rule entry (regenerated artifact, but
    committed in this repo per existing pattern).
- tests/validate-rules.test.ts
  - Invert the direct assertion (test at line 426): hyphenated
    AND dotted forms now both pass without violation.
  - Update three multi-skill overlap tests to use
    `gateway('opaque')` (missing-provider rule, severity:error)
    instead of the deleted slug rule as the ai-gateway trigger.
  - Update the line-number test's message-includes string to
    match the still-present "missing provider/ prefix" rule.
  - Update the negative-test's comment + extend coverage to both
    forms.
- tests/posttooluse-validate.test.ts
  - Convert "detects gateway from 'ai' with hyphenated model slug"
    test from "expect 'dots not hyphens' violation" to "expect
    NO 'dots not hyphens' violation, for both hyphen and dot
    forms." ai-gateway skill is still expected to be matched on
    the file via the `'ai'` import pattern.
- tests/posttooluse-chain.test.ts
  - Cosmetic: fix the test fixture string `anthropic('claude-sonnet-4.6')`
    to the canonical hyphen form `anthropic('claude-sonnet-4-6')`.
    Test logic unchanged.

Verification
------------
The rule's removal was already validated against a real-world
failure: a production app shipped `anthropic('claude-sonnet-4.6')`
(dotted, matching what the validator told them was correct), the
Anthropic API rejected the model ID, every critique call surfaced
as `ai_error` for end users. Switching to `claude-sonnet-4-6`
(hyphens) — what this rule had been flagging as wrong — fixed it.

Local tests not run (bun not installed in the patch environment);
relying on CI.

Closes vercel#60.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model-slug validator flags Anthropic canonical hyphen slugs (claude-opus-4-7, claude-sonnet-4-6) as typos expecting dots

1 participant