Skip to content

fix(den): widen config_object.search_text and bound the derived projection - #3386

Open
benjaminshafii wants to merge 1 commit into
devfrom
fix/config-object-search-text-overflow
Open

fix(den): widen config_object.search_text and bound the derived projection#3386
benjaminshafii wants to merge 1 commit into
devfrom
fix/config-object-search-text-overflow

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

The crash

Production Sentry, den-api, POST /v1/plugins/import-mcps-from-github-url:

Failed query: insert into `config_object` (`id`, `organization_id`, `object_type`, `source_mode`,
  `title`, `description`, `search_text`, ...) values (?, ?, ?, ...)
errno: 1406  ER_DATA_TOO_LONG  "Data too long for column 'search_text' at row 1"

config_object.search_text was TEXT (65,535 bytes), but for skills the projection is the entire
SKILL.md body (store.ts deriveSkillProjection), and nothing on the import path capped it. The 1 MiB
request guard only protects POST /v1/config-objects; importGithubPluginMcps calls createConfigObject
in process and bypasses it. MySQL/PlanetScale run strict mode, so the row is rejected rather than truncated.

For a skill row this was the only column that could overflow — title is validated to 64 chars and
description to 1024 — which is why the Sentry params contained the whole SKILL.md.

This is the sibling that d2b6199 / migration 0048 missed. That commit widened
config_object_version.raw_source_text and normalized_payload_json to mediumtext for exactly this
failure, so the full SKILL.md saved fine in the version row (MEDIUMTEXT, ~12.58 MB of plaintext) and blew
up on the projection row (TEXT, 64 KiB).

Changes

File Change
den-db/src/schema/sharables/plugin-arch.ts text("search_text") -> mediumtext("search_text")
den-db/drizzle/0050_lean_lockjaw.sql ALTER TABLE config_object MODIFY COLUMN search_text mediumtext;
plugin-system/store.ts clampConfigObjectTitletitle is varchar(255) and the non-skill branch derived it from an unclamped "<plugin> / <server>" name or an arbitrary connector file's first line, so it could 1406 on the same insert
plugin-system/store.ts 1 MiB byte guard in deriveSkillProjection — an oversized repository now returns 400 skill_source_too_large instead of a driver 500
plugin-system/store.ts clampSearchProjection — non-skill projections held to the same budget; search_text ships inside list responses, so mediumtext alone would allow 16 MB payloads
plugin-system/schemas.ts response max 65535 -> configObjectInputMaxPayloadBytes (now exported)

Tests run

New: ee/apps/den-api/test/plugin-system-config-object-projection-size-db.test.ts (5 cases, real MySQL 8.4).

cd ee/apps/den-api && bun test test/plugin-system-config-object-projection-size-db.test.ts
  -> 5 pass, 0 fail

The red run before the fix reproduced the Sentry event verbatim — same statement, same column list,
errno: 1406, code: ER_DATA_TOO_LONG, column 'search_text'. Attribution was verified by toggling the
column type back on the test database:

case search_text = text search_text = mediumtext
>64 KiB SKILL.md keeps full projection fail — 1406 ... 'search_text' pass
multibyte: <65,535 chars but >65,535 bytes fail — same 1406 pass
over-long derived title pass (code fix) pass
non-skill >1 MiB clamped, no U+FFFD pass (code fix) pass
>1 MiB skill -> 400 not 500 pass (code fix) pass

Also run:

  • npx tsc -p tsconfig.json --noEmit in ee/apps/den-api and ee/packages/den-db — both clean
  • 0050_lean_lockjaw.sql applied against a pre-0050 database (search_text -> mediumtext, 16,777,215)
  • fresh db:migrate/db:push on a scratch database agrees with the schema (no pending diff on re-generate)
  • 12 neighbouring suites individually green: plugin-system-* (access, create-bundle, cross-org-idor,
    marketplace-defaults, marketplace-seeding-db, config-object-ownership), github-plugin-import-schema,
    github-plugin-mcp-auth, github-discovery, connector-cleanup, deprecated-skill-hubs

Honest gaps

  • No fraimz. This is a DB-constraint fix with no UI surface; the real-MySQL red/green above is the
    proof. Driving the import through the desktop/cloud UI would need a Den server plus a public repo
    shipping a >64 KiB skill — happy to do it if a reviewer wants it.
  • test/mcp-agent-config-policy.test.ts has 1 pre-existing failure (an import-route capability-search
    assertion). Confirmed identical on unmodified origin/dev — unrelated to this change, not fixed here.

Deploy note

The ALTER widens a column on config_object (rewrite; no index on search_text). Backward compatible —
widening only. On PlanetScale it goes through a deploy request.

Review asks

Please don't rubber-stamp this. Specific things worth pushing back on:

  1. Widen vs truncate. I widened the column to keep large skills fully searchable, matching 0048.
    The alternative was truncating search_text to 64 KiB with no migration. Is a rewrite on
    config_object acceptable at current table size?
  2. clampSearchProjection (1 MiB, lossy) for non-skill objects vs a hard 400 like skills get. I chose
    lossy because a connector-synced file has no authored contract to reject, but that is a judgement call.
  3. New 400 skill_source_too_large is a behaviour change for connector sync: a >1 MiB skill file now
    fails its sync instead of erroring later at the driver. Confirm that is the failure mode we want.
  4. Whether the response-schema max should state the column ceiling (16 MB) rather than the enforced 1 MiB.

…ction

A GitHub skill import (POST /v1/plugins/import-mcps-from-github-url) crashed
with MySQL 1406 "Data too long for column 'search_text'". The skill projection
carries the whole SKILL.md body, but search_text was TEXT (65,535 bytes) and
nothing on the import path capped it: the 1 MiB request guard only covers
POST /v1/config-objects, while importGithubPluginMcps calls createConfigObject
in process.

Migration 0048 widened config_object_version.raw_source_text and
normalized_payload_json for this exact failure and missed this sibling column,
so the full SKILL.md persisted in the version row and blew up on the projection.

- widen config_object.search_text to mediumtext (migration 0050)
- clamp the derived title to varchar(255); the non-skill branch took an
  unclamped "<plugin> / <server>" name or an arbitrary file's first line
- enforce the 1 MiB payload cap on imported and connector-synced skills so an
  oversized repository returns 400 skill_source_too_large instead of a 500
- clamp non-skill projections to the same budget; search_text ships inside list
  responses, so mediumtext alone would allow 16 MB payloads
@vercel

vercel Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview Jul 31, 2026 1:05pm
openwork-den Ready Ready Preview Jul 31, 2026 1:05pm
openwork-den-worker-proxy Ready Ready Preview Jul 31, 2026 1:05pm
openwork-landing Ready Ready Preview, v0 Jul 31, 2026 1:05pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
openwork-diagnostics Skipped Skipped Jul 31, 2026 1:05pm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant