Skip to content

docs(runner): make the guide tell the truth, and make drift fail a test (DEV-2203) - #188

Open
danielzytohoc wants to merge 2 commits into
masterfrom
fix/DEV-2203-guide-parity
Open

docs(runner): make the guide tell the truth, and make drift fail a test (DEV-2203)#188
danielzytohoc wants to merge 2 commits into
masterfrom
fix/DEV-2203-guide-parity

Conversation

@danielzytohoc

@danielzytohoc danielzytohoc commented Aug 14, 2026

Copy link
Copy Markdown

A parity audit (DEV-2203) read the five guide tracks in runner/docs/guide/ against the product they describe. Fourteen claims lost the argument. This PR fixes the text and adds four parity tests to runner/pipeline/guide-tracks.test.mjs so the next drift fails CI instead of a reader. No product code changes.

The fourteen fixes, grouped

Stale status

  1. everyone.md — deleted the "Status: landing shortly" block and its two interim workarounds: PATCH /api/mcp/demos/:id is merged (workers/api/src/index.ts), so changing a demo by asking works now.
  2. everyone.md — added the real boundary: only MCP-created demos can be updated this way; browser-built demos answer 403 with "this demo was not created through the MCP; edit it at /edit/ in the browser".

Dead URLs
3. All printed ?docs= URLs now name guides/rows/rows-sorting/react/exampleSortingDemo.tsx, which exists in the 18.0 bucket — column-adding is not in 18.0 and rows/row-sorting exists in no bucket (the guide is rows-sorting). The versioned example pins &v=18.0.0; there is no 17.x docs bucket.
4. support.md/all-demos?owner=marek-martuszewskiowner=marek.martuszewski; the slug is the email local part (demoOwners.ts), dots included.

Over-promises
5. overview.md + developers.md — the budget prose claimed "saving or forking still works" while paused. budgetGate also gates POST /api/demos and the rebuild, so a Save/Fork is refused at the top tiers. The guide now promises only the static reads.
6. overview.md — "Imports and file drops with no Handsontable in them are refused": only imports are guarded (import-url.ts); dropFiles.ts has no such check. Now imports only.
7. support.md — Download sat in the "Signed in, Files also lets you" list; it has never been auth-gated (TopBar.tsx says so explicitly). Moved out of the signed-in list.
14. developers.md — "node_modules, build output and lockfiles are refused, not dropped" was claimed for every path; it is true for the MCP only. The file drop skips them quietly and stops at 50 files (with a "Stopped at 50 files" report). Split per path. (Note: the audit called the 50-file cut silent; dropFiles.ts reports it — truncated feeds rejectionMessage() — so the guide says "stops … saying where it stopped".)

Naming
8. support.md + overview.md + devrel.md — the guide handed out /d/<id>/ as "the client link", but both product surfaces (Share's Public client link field, My demos' Copy link) hand out /share/<id>. The guide now documents actual behavior: /share/<id> is the public client link (read-only playground); /d/<id>/ is the bare static page it displays.
10. everyone.md — the quoted error "This demo is not available" exists nowhere; replaced with the real string "This demo is unavailable." (App.tsx; the revoked page says "This demo has been revoked", share.ts).

Precision
9. everyone.md — the MCP path requires a description (400 without one, index.ts), unlike the editor. Said so.
11. devrel.md/d/<id>/ "cannot be framed at all, on any site": CSP is frame-ancestors 'self', which is exactly how /share and full mode display it. Now "on any other site".
12. developers.md — "Integers below 1000 are read as a version": 15–19 are majors, 20–999 are refused outright (MIN_BARE_NUMERIC_PKG_PR_NEW_REF), ≥1000 is a build ref. Made precise.
13. developers.md — "Container (tier 2) — … MUI, Ant Design, Fluent UI, Base Web": those four are tier: 1 with engine: "container" in catalog.json. The section speaks in engines now (as overview.md does) and admits React (Vite, JS) also runs in a container.

The four parity tests

  • every ?docs= URL the guide prints exists in the release bucket — extracts each ?docs= path from the five docs, asserts it is a docsPath in apps/authoring/public/docs-examples/18.0/manifest.json.
  • the guide's numbers are the product's numbers — imports MAX_MCP_FILES/MAX_MCP_BYTES (mcp-create.ts), MAX_DESCRIPTION (demo-info.ts), DEFAULT_MIN_MAJOR/DEFAULT_MAX_MAJOR (built runtime), and source-greps the unexported MIN_BARE_NUMERIC_PKG_PR_NEW_REF and PAYLOAD_TTL_SECONDS; asserts the markdown states each (50, 256 KB, 4,000, 15–19, 1000, 24 hours).
  • the guide's starter keys match the catalog — every ?example=<key> (and the URL table's backticked alternates) is a framework key in runner/catalog.json.
  • the guide's container claims match the catalog engines — maps the display names in developers.md's container/in-browser lists to catalog keys and asserts engine === "container" / "sandpack"; catalog-tier vocabulary is banned from that section (the assertion that would have caught fix 13).

Spot-checked: re-breaking one ?docs= URL fails test 1; restoring the "(tier 2)" wording fails test 4.

Needs a product decision (flagging for Dan)

The client-link naming tension is documented, not resolved: the product hands out /share/<id> under the label "Public client link" (ShareLinks.tsx) and from My demos' Copy link (MyDemos.tsx), while the docs vocabulary used to reserve "client link" for /d/<id>/. The guide now documents current behavior. If the intent was for clients to receive /d/<id>/ links, that is a product change to App.tsx/MyDemos.tsx — not made here.

Test results

  • pnpm test — 364 pass (incl. the 4 new parity tests)
  • pnpm e2e e2e/guide.spec.ts — 10 pass

Note

Low Risk
Documentation and test-only changes; no runtime or API behavior is modified in this PR.

Overview
Runner guide tracks (developers, devrel, everyone, overview, support) are rewritten so URLs, limits, starter engines, budget behavior, and sharing vocabulary match the live product—without changing application code.

Docs fixes include swapping printed ?docs= examples to guides/rows/rows-sorting/react/exampleSortingDemo.tsx, treating /share/<id> as the public client link vs /d/<id>/ as the bare static page, correcting budget-pause text (Save/Fork blocked at top tiers; static reads still work), splitting MCP vs file-drop handling of node_modules/lockfiles, noting MCP-required descriptions, fixing owner filter slug (marek.martuszewski), clarifying bare version integers (1519 majors, 20999 refused, PR refs from 1000), and describing container vs in-browser engines (including React Vite JS in a container).

CI adds four parity tests in guide-tracks.test.mjs: every guide ?docs= path exists in the 18.0 manifest, numeric limits match mcp-create, demo-info, dropFiles, and version constants, ?example= keys match catalog.json, and developers.md container/in-browser lists match catalog engine values.

Reviewed by Cursor Bugbot for commit 7c1712e. Bugbot is set up for automated code reviews on this repo. Configure here.

…st (DEV-2203)

A parity audit read the five guide tracks against the product they describe
and found fourteen places where they disagree. The guide loses every one of
those arguments, so the text moves; and four new parity tests in
guide-tracks.test.mjs read the product's own files so the next drift fails
CI instead of a reader.

Stale status:
- everyone: the 'landing shortly' block for updating by asking is gone —
  PATCH /api/mcp/demos/:id is merged, so asking works now, with the real
  boundary stated (MCP-created demos only; browser demos answer 403 and
  point at /edit).

Dead URLs:
- ?docs= examples pointed at guides/columns/column-adding and
  guides/rows/row-sorting, which exist in no bucket; every printed URL now
  names rows-sorting/react/exampleSortingDemo.tsx, which is in 18.0.
- the versioned ?docs= example used v=17.1.0, a version with no docs
  bucket; it pins 18.0.0.
- /all-demos?owner= used a hyphenated slug; the slug is the email local
  part, dots included.

Over-promises:
- the budget prose claimed saving and forking survive the ceiling; a Save
  or Fork boots a build container and the top tiers refuse it, so the
  guide now promises only what R2 serves — existing links and embeds.
- 'imports and file drops with no Handsontable are refused' — only imports
  are guarded; the drop path has no such check.
- 'node_modules, build output and lockfiles are refused, not dropped' —
  true on the MCP path only; the file drop skips them quietly and stops at
  50 files, saying where it stopped.
- Download sat in the signed-in Files list; it has never been auth-gated.

Naming:
- the guide handed /d/<id>/ out as 'the client link', but both product
  surfaces (Share's 'Public client link', My demos' Copy link) hand out
  /share/<id>; the guide now describes what the product does, with
  /d/<id>/ as the bare static page the playground displays.
- 'This demo is not available' is a string nowhere in the product; the
  real one is 'This demo is unavailable.'

Precision:
- the MCP path requires a description (400 without one); said so.
- bare integers: 15-19 are majors, 20-999 are refused, 1000 and up are
  pkg.pr.new refs — 'below 1000 is a version' was two-thirds wrong.
- the 'Container (tier 2)' list named four tier-1 starters; the section
  speaks in engines now, and admits React (Vite, JS) is a container too.
- /d/<id>/ 'cannot be framed at all, on any site' — frame-ancestors is
  'self', which is how /share displays it; now 'on any other site'.

The four new tests: every printed ?docs= path must exist in the 18.0
manifest; every stated number must equal the product constant it came from
(MCP caps, description ceiling, major range, PR-ref floor, payload TTL);
every ?example= key must be in the catalog; and every starter the container
list names must have engine 'container' in catalog.json, with catalog-tier
vocabulary banned from that section outright.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b07cbc6. Configure here.

Comment thread runner/docs/guide/developers.md
…Bugbot #188)

Master rewrote 'Changing a demo, also by asking' for the shipped
update_demo tool — its version wins, with the one fact it lacked grafted
in: updating only works on demos the MCP created (a browser-built demo
answers with the /edit/<id> refusal, index.ts ~777), which is a separate
boundary from ownership.

The numbers parity test now also pins MAX_DROP_FILES from dropFiles.ts:
the guide's 'stops at 50 files' named a constant the test never read,
so the drop claim could drift while the MCP caps stayed green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant