Optimize docs for LLM navigation (AI SEO / llms.txt / markdown mirrors)#426
Merged
Conversation
…rects - generate-full-docs.js: include .md pages (SDK API refs were missing), sweep files not listed in _meta.json (recovers CLI subtrees), convert JSX (Callout/Cards/Tabs/Image) to plain markdown, prepend canonical Source: URLs, guarantee H1 per page, drop "page has moved" stubs - generate-sitemap-xml.js: exclude hidden _-prefixed paths, include .md routes, fix nested index URLs - next.config.js: fix 4 redirects pointing at the removed advanced-features/ tree (they 404ed), redirect deleted duplicate page - delete duplicate features/features.mdx, add LibertAI to partners nav - add robots.txt explicitly allowing AI crawlers, linking sitemap Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tadata - generate-full-docs.js now emits: per-page raw-markdown mirrors served at <page-url>.md (with title/description/source/last_updated frontmatter from git history), an auto-generated llms.txt covering all pages with .md links + agent instructions, section-scoped llms-full.txt bundles (protocol/developers/validators/api-references), and llms-full.txt as a real file (replaces symlink); all gitignored - theme.config.tsx: per-page meta descriptions from frontmatter, rel=alternate text/markdown link, canonical URL, TechArticle + BreadcrumbList JSON-LD on every page - copy-page.tsx: "Copy page" / "View as Markdown" / "Open in ChatGPT/Claude" now use the real .md mirrors instead of DOM innerText - add answer-first frontmatter descriptions to 11 key pages - fix missing H1s on 4 hand-written pages (api-references, genlayerlabs, transaction-encoding, transaction-execution) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
✅ Deploy Preview for genlayer-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
|
Warning Review limit reached
More reviews will be available in 51 minutes and 48 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more credits in the billing tab to continue. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (19)
📝 WalkthroughWalkthroughThis PR adds a markdown-mirror doc export system: it enriches MDX sources with frontmatter, refactors generation scripts to emit per-page .md mirrors and aggregated bundles, adds validation checks, introduces Accept-header middleware to serve .md to agents, updates routing/redirects, and wires UI/SEO to expose mirrors. ChangesLLM Export and Markdown Mirror System
Sequence Diagram(s)sequenceDiagram
participant Client as AI Agent / Browser
participant Middleware
participant NextRouter as Next.js Router
participant MarkdownAPI as /api/markdown-404
participant Mirror as Generated .md file
Client->>Middleware: GET /page (Accept: text/markdown)
Middleware->>NextRouter: rewrite -> /page.md
NextRouter->>Mirror: lookup /page.md
alt mirror exists
Mirror-->>Client: 200 text/markdown
else mirror missing
NextRouter->>MarkdownAPI: rewrite with requested=/page.md
MarkdownAPI-->>Client: 404 text/markdown + closest matches
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09837a7038
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }, | ||
| { | ||
| old: "/developers/intelligent-contracts/features/features", | ||
| new: "/developers/intelligent-contracts/features", |
There was a problem hiding this comment.
When users or crawlers hit the legacy feature overview URL, this redirect now sends them to /developers/intelligent-contracts/features, but this commit deleted pages/developers/intelligent-contracts/features/features.mdx without adding pages/developers/intelligent-contracts/features/index.mdx or index.md, so the target is not a routable Next/Nextra page. The empty :page* case for /developers/intelligent-contracts/advanced-features above has the same missing target; point these overview redirects to an existing feature page or add an index page.
Useful? React with 👍 / 👎.
… guard - theme.config.tsx: per-page og:title; visible "Last updated" dates via Nextra gitTimestamp (+ @napi-rs/simple-git for build-time git lookups) - middleware.ts: Accept: text/markdown content negotiation serves the .md mirror of any docs page, with Vary: Accept for CDN correctness - api/markdown-404: missed .md URLs return a text/markdown 404 with closest-match links from llms.txt (afterFiles rewrite in next.config) - scripts/check-llm-exports.js: build fails if export page counts drop below floors, JSX leaks in, or known content anchors disappear - sitemap lastmod now uses git commit dates (shared scripts/lib/git-dates) instead of file mtime, which equals checkout time on CI - root llms-full.txt omits the 66 per-command CLI pages (~180K tokens total now); full CLI reference stays in api-references/llms-full.txt - disambiguate near-duplicate titles for retrieval: Debugging in Studio vs Debugging Contract Code, Contract Storage vs Storage Quick Reference, Testing vs Testing Contracts in GenLayer Studio - fix stale genlayer-cli/_meta.json left from the command regrouping: keys now match the real category directories so the sidebar works Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Added a third commit with the remaining hardening items:
Deploy notes: Nextra warns that git timestamps need a deep clone on Vercel ( 🤖 Generated with Claude Code |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (6)
pages/api/markdown-404.ts (1)
10-36: ⚡ Quick winConsider caching the llms.txt link list for better performance.
The current implementation reads and parses
public/llms.txtsynchronously on every 404 request. For a production site with frequent 404s, consider caching the parsed link list in a module-level variable and refreshing it periodically or on file change.♻️ Suggested caching approach
+let cachedLinks: { title: string; url: string }[] | null = null; +let cacheTime = 0; +const CACHE_TTL = 60000; // 1 minute + function suggestions(requested: string): { title: string; url: string }[] { try { - const llms = fs.readFileSync(path.join(process.cwd(), "public", "llms.txt"), "utf8"); - const links: { title: string; url: string }[] = []; - const linkRegex = /^- \[([^\]]+)\]\((\S+?\.md)\)/gm; - let match: RegExpExecArray | null; - while ((match = linkRegex.exec(llms)) !== null) { - links.push({ title: match[1], url: match[2] }); + const now = Date.now(); + if (!cachedLinks || now - cacheTime > CACHE_TTL) { + const llms = fs.readFileSync(path.join(process.cwd(), "public", "llms.txt"), "utf8"); + const links: { title: string; url: string }[] = []; + const linkRegex = /^- \[([^\]]+)\]\((\S+?\.md)\)/gm; + let match: RegExpExecArray | null; + while ((match = linkRegex.exec(llms)) !== null) { + links.push({ title: match[1], url: match[2] }); + } + cachedLinks = links; + cacheTime = now; } const tokens = requested .replace(/\.md$/, "") .toLowerCase() .split(/[/-]/) .filter((t) => t.length > 2); - return links + return cachedLinks .map((link) => {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pages/api/markdown-404.ts` around lines 10 - 36, The suggestions function currently reads and parses public/llms.txt on every call causing unnecessary sync I/O; move the file read/parse into a module-level cached variable (e.g., parse once into an array like cachedLinks using the same linkRegex parsing logic) and have suggestions(requested) operate on that cachedLinks; add a simple refresh strategy (fs.watch or time-based reload) to update cachedLinks on change or interval and handle read/parse errors by falling back to an empty array so suggestions remains safe.scripts/check-llm-exports.js (2)
25-37: ⚡ Quick winDocument the rationale for coverage floor thresholds.
The hardcoded coverage floors (200, 130, 70) effectively prevent regressions, but future maintainers might not understand why these specific values were chosen. Consider adding comments explaining:
- Why 200 links in llms.txt (current count is 208 per PR description)
- Why 130 pages in full-documentation.txt
- Why 70 pages in api-references/llms-full.txt (CLI subtree)
This will help when the counts legitimately change in the future.
📝 Add explanatory comments
// 1. Coverage floors (update deliberately if pages are intentionally removed) +// As of PR `#426`, actual counts: llms.txt 208 links, root bundle 140+ pages, api bundle 80+ pages +// Floors set at ~95% of actual to catch major regressions while allowing minor fluctuations const llmsTxtLinks = (llmsTxt.match(/^- \[/gm) || []).length; if (llmsTxtLinks < 200) { failures.push(`llms.txt lists only ${llmsTxtLinks} pages (floor: 200) — pages are being dropped`); } const rootPageCount = (fullDocs.match(/^Source: /gm) || []).length; if (rootPageCount < 130) { failures.push(`full-documentation.txt has only ${rootPageCount} pages (floor: 130)`); } const apiPageCount = (apiBundle.match(/^Source: /gm) || []).length; if (apiPageCount < 70) { failures.push(`api-references/llms-full.txt has only ${apiPageCount} pages (floor: 70) — CLI subtree may be missing`); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/check-llm-exports.js` around lines 25 - 37, Add brief explanatory comments above the three threshold checks (the llmsTxtLinks check, the rootPageCount check, and the apiPageCount check) in scripts/check-llm-exports.js that document why the numeric floors were chosen (e.g., current observed counts like ~208 links, expected docs coverage, and the CLI subtree reason for the API count) and note that these are conservative guards against accidentally dropping pages and should be updated when pages are intentionally removed; include a reference to the source used to pick the floor (current counts/PR description) and a short note on when it is acceptable to change each value.
40-48: 💤 Low valueConsider expanding the JSX leak detection to cover more components.
The current check only looks for
Callout,Cards,CustomCard, andTabs. Other presentational components likeImage,Bleed, or custom components might also leak through if the transformation logic has gaps.While the current set covers the main block-level components, you might want to add a broader pattern or document which components are expected to be converted.
🔍 Expand detection pattern
// 2. No JSX leaks — components must be converted to markdown for (const [name, content] of [ ['full-documentation.txt', fullDocs], ['api-references/llms-full.txt', apiBundle], ]) { - const leak = content.match(/<(Callout|Cards|CustomCard|Tabs)[\s>]/); + // Check for common Nextra/custom components that should be converted + const leak = content.match(/<(Callout|Cards|CustomCard|Card|Tabs|Image|Bleed|AddToWallet)[\s>]/); if (leak) { failures.push(`JSX leaked into ${name}: found "${leak[0].trim()}" — check cleanMdxContent()`); } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/check-llm-exports.js` around lines 40 - 48, The JSX leak detector currently only matches a small set of component names (used in the leak variable check that inspects fullDocs and apiBundle) so add a broader detection regex or expand the list to catch other presentational components; specifically, replace the leak match with a pattern that catches any capitalized React-style component (e.g. /<([A-Z][A-Za-z0-9]*)\b/) or include additional names like Image, Bleed, etc., and keep pushing the same failure message into failures while referencing cleanMdxContent() as the place to fix conversions.scripts/generate-full-docs.js (3)
349-410: ⚡ Quick winAdd error handling for file operations in the main generation function.
The
generateFullDocsfunction performs multiple file system operations (readFileSync, writeFileSync, mkdirSync) that could fail due to permissions, disk space, or I/O errors. While failures will crash the build (which is appropriate), adding try-catch with context would make debugging easier.♻️ Add contextual error handling
function generateFullDocs() { + try { const pagesDir = path.join(process.cwd(), 'pages'); const outputDir = path.join(process.cwd(), 'public'); // Create output directory if it doesn't exist if (!fs.existsSync(outputDir)) { fs.mkdirSync(outputDir); } // ... rest of the function console.log( `generate-full-docs: ${pages.length} pages -> full-documentation.txt, llms-full.txt ` + `(${rootPages.length} pages, CLI command pages only in api-references bundle), ` + `${sections.length} section bundles, per-page .md mirrors, llms.txt` ); + } catch (error) { + console.error('generate-full-docs: FAILED'); + console.error(` Error: ${error.message}`); + if (error.code) console.error(` Code: ${error.code}`); + if (error.path) console.error(` Path: ${error.path}`); + throw error; // Re-throw to fail the build + } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/generate-full-docs.js` around lines 349 - 410, The generateFullDocs function performs many FS operations that lack contextual error handling; wrap the core work inside a try-catch (around the sequence that calls buildGitDates, collectPages, preparePage, writeFileSync/mkdirSync/unlinkSync, writePageMirrors and fs.readFileSync), and on error log a clear, contextual message including the operation and the page/path or symbol involved (e.g., when reading page file in collectPages/preparePage, writing full-documentation.txt or llms-full.txt, creating section dirs, or writing mirrors), then rethrow or call process.exit(1) to preserve the failing behavior; ensure the catch references the thrown error object so the logged message includes the actual error details to aid debugging.
106-114: ⚡ Quick winConsider adding error handling for malformed JSX attributes.
The
parseAttributesregex pattern is complex and could fail silently on malformed input. While the current pattern handles quoted strings and template literals, edge cases like unclosed quotes or nested braces could produce unexpected results.🛡️ Add defensive error handling
function parseAttributes(attrString) { const attrs = {}; + if (!attrString || typeof attrString !== 'string') return attrs; const attrRegex = /(\w+)=(?:"([^"]*)"|'([^']*)'|\{`([^`]*)`\}|\{"([^"]*)"\}|\{'([^']*)'\})/g; let match; while ((match = attrRegex.exec(attrRegex)) !== null) { attrs[match[1]] = match[2] ?? match[3] ?? match[4] ?? match[5] ?? match[6] ?? ''; } return attrs; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/generate-full-docs.js` around lines 106 - 114, The parseAttributes function’s complex attrRegex can fail silently or hang on malformed JSX (unclosed quotes/nested braces); wrap the parsing loop in a try/catch, add defensive checks inside the while loop to detect no-regex-progress (if attrRegex.lastIndex does not advance) and break to avoid infinite loops, validate that each matched attribute has a non-empty key and at least one captured value (otherwise log a warning or skip the attribute), and on catch emit a clear error/warning including the original attrString; keep references to parseAttributes and attrRegex when implementing these checks.
199-209: 💤 Low valueVerify the code fence pattern handles edge cases.
The regex
/```[\s\S]*?```|~~~[\s\S]*?~~~/gfor splitting code fences uses non-greedy matching, which should handle most cases. However, it won't handle:
- Fenced code blocks with custom language info that includes backticks (e.g.,
```js title="example")- Escaped backticks inside code blocks
- Malformed/unclosed fences
These are rare edge cases, but if they occur, JSX transformation could leak into code blocks.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/generate-full-docs.js` around lines 199 - 209, The split regex in cleanMdxContent can misidentify fenced code blocks with info strings or backticks inside them; update the fence-matching logic in cleanMdxContent (after transformBlockJsx and before transformJsxSegment mapping) to use a more robust pattern that matches an opening fence (``` or ~~~) plus its optional info string on the same line and then consumes until the matching closing fence (ensuring the same fence characters are used), and use flags to operate multiline so escaped/backtick characters inside the block are treated as content; keep transformBlockJsx and transformJsxSegment unchanged but replace the current /```[\s\S]*?```|~~~[\s\S]*?~~~/g split with this stronger fence-aware matcher so JSX transformations are only applied to non-code segments.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@package.json`:
- Around line 6-7: The package-lock.json is out of sync with package.json
causing CI to fail; regenerate and commit an updated lockfile: run npm install
locally to update package-lock.json so it reflects the current "dev" and "build"
script entries (and any other dependency changes), verify the working tree, then
add and commit the updated package-lock.json before pushing the branch.
- Line 40: The package.json entry "`@napi-rs/simple-git`": "^0.1.22" is unused in
the codebase; either remove this dependency from package.json (and regenerate
package-lock.json) or add a short comment in package.json near the dependencies
explaining why it's intentionally present (e.g., planned feature, experimental
local dev tool) and link to the PR/issue that tracks its future usage; make sure
to also update package-lock.json and run npm install to keep lockfile consistent
if you remove or justify the dependency.
In
`@pages/understand-genlayer-protocol/core-concepts/transactions/transaction-encoding-serialization-and-signing.mdx`:
- Line 3: Replace the misspelled word "relieable" with "reliable" in the
sentence that reads "packaged into a relieable cross-platform efficient format"
inside the transaction-encoding-serialization-and-signing.mdx content so the
sentence reads "packaged into a reliable cross-platform efficient format".
---
Nitpick comments:
In `@pages/api/markdown-404.ts`:
- Around line 10-36: The suggestions function currently reads and parses
public/llms.txt on every call causing unnecessary sync I/O; move the file
read/parse into a module-level cached variable (e.g., parse once into an array
like cachedLinks using the same linkRegex parsing logic) and have
suggestions(requested) operate on that cachedLinks; add a simple refresh
strategy (fs.watch or time-based reload) to update cachedLinks on change or
interval and handle read/parse errors by falling back to an empty array so
suggestions remains safe.
In `@scripts/check-llm-exports.js`:
- Around line 25-37: Add brief explanatory comments above the three threshold
checks (the llmsTxtLinks check, the rootPageCount check, and the apiPageCount
check) in scripts/check-llm-exports.js that document why the numeric floors were
chosen (e.g., current observed counts like ~208 links, expected docs coverage,
and the CLI subtree reason for the API count) and note that these are
conservative guards against accidentally dropping pages and should be updated
when pages are intentionally removed; include a reference to the source used to
pick the floor (current counts/PR description) and a short note on when it is
acceptable to change each value.
- Around line 40-48: The JSX leak detector currently only matches a small set of
component names (used in the leak variable check that inspects fullDocs and
apiBundle) so add a broader detection regex or expand the list to catch other
presentational components; specifically, replace the leak match with a pattern
that catches any capitalized React-style component (e.g.
/<([A-Z][A-Za-z0-9]*)\b/) or include additional names like Image, Bleed, etc.,
and keep pushing the same failure message into failures while referencing
cleanMdxContent() as the place to fix conversions.
In `@scripts/generate-full-docs.js`:
- Around line 349-410: The generateFullDocs function performs many FS operations
that lack contextual error handling; wrap the core work inside a try-catch
(around the sequence that calls buildGitDates, collectPages, preparePage,
writeFileSync/mkdirSync/unlinkSync, writePageMirrors and fs.readFileSync), and
on error log a clear, contextual message including the operation and the
page/path or symbol involved (e.g., when reading page file in
collectPages/preparePage, writing full-documentation.txt or llms-full.txt,
creating section dirs, or writing mirrors), then rethrow or call process.exit(1)
to preserve the failing behavior; ensure the catch references the thrown error
object so the logged message includes the actual error details to aid debugging.
- Around line 106-114: The parseAttributes function’s complex attrRegex can fail
silently or hang on malformed JSX (unclosed quotes/nested braces); wrap the
parsing loop in a try/catch, add defensive checks inside the while loop to
detect no-regex-progress (if attrRegex.lastIndex does not advance) and break to
avoid infinite loops, validate that each matched attribute has a non-empty key
and at least one captured value (otherwise log a warning or skip the attribute),
and on catch emit a clear error/warning including the original attrString; keep
references to parseAttributes and attrRegex when implementing these checks.
- Around line 199-209: The split regex in cleanMdxContent can misidentify fenced
code blocks with info strings or backticks inside them; update the
fence-matching logic in cleanMdxContent (after transformBlockJsx and before
transformJsxSegment mapping) to use a more robust pattern that matches an
opening fence (``` or ~~~) plus its optional info string on the same line and
then consumes until the matching closing fence (ensuring the same fence
characters are used), and use flags to operate multiline so escaped/backtick
characters inside the block are treated as content; keep transformBlockJsx and
transformJsxSegment unchanged but replace the current
/```[\s\S]*?```|~~~[\s\S]*?~~~/g split with this stronger fence-aware matcher so
JSX transformations are only applied to non-code segments.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f1bf6000-3ef0-4b86-91a3-b8f307d93c23
📒 Files selected for processing (41)
.gitignorecomponents/copy-page.tsxmiddleware.tsnext.config.jspackage.jsonpages/FAQ.mdxpages/api-references.mdxpages/api-references/genlayer-cli/_meta.jsonpages/api/markdown-404.tspages/developers.mdxpages/developers/decentralized-applications/_meta.jsonpages/developers/decentralized-applications/testing.mdxpages/developers/intelligent-contracts/_meta.jsonpages/developers/intelligent-contracts/debugging.mdxpages/developers/intelligent-contracts/features/_meta.jsonpages/developers/intelligent-contracts/features/debugging.mdxpages/developers/intelligent-contracts/features/features.mdxpages/developers/intelligent-contracts/features/storage.mdxpages/developers/intelligent-contracts/first-contract.mdxpages/developers/intelligent-contracts/introduction.mdxpages/developers/intelligent-contracts/storage.mdxpages/developers/intelligent-contracts/testing.mdxpages/developers/intelligent-contracts/tooling-setup.mdxpages/developers/networks.mdxpages/index.mdxpages/partners/_meta.jsonpages/partners/genlayerlabs.mdxpages/understand-genlayer-protocol/core-concepts/optimistic-democracy/equivalence-principle.mdxpages/understand-genlayer-protocol/core-concepts/transactions/transaction-encoding-serialization-and-signing.mdxpages/understand-genlayer-protocol/core-concepts/transactions/transaction-execution.mdxpages/understand-genlayer-protocol/what-is-genlayer.mdxpages/validators/setup-guide.mdxpublic/llms-full.txtpublic/llms.txtpublic/robots.txtpublic/sitemap.xmlscripts/check-llm-exports.jsscripts/generate-full-docs.jsscripts/generate-sitemap-xml.jsscripts/lib/git-dates.jstheme.config.tsx
💤 Files with no reviewable changes (3)
- public/llms.txt
- pages/developers/intelligent-contracts/features/features.mdx
- public/llms-full.txt
| "react-dom": "^18.3.1" | ||
| }, | ||
| "devDependencies": { | ||
| "@napi-rs/simple-git": "^0.1.22", |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if `@napi-rs/simple-git` is actually imported anywhere
echo "=== Searching for imports of `@napi-rs/simple-git` ==="
rg -n --type js --type ts "require.*`@napi-rs/simple-git`|from ['\"]`@napi-rs/simple-git`"
echo ""
echo "=== Searching for 'simpleGit' or 'SimpleGit' usage ==="
rg -n --type js --type ts "simpleGit|SimpleGit"Repository: genlayerlabs/genlayer-docs
Length of output: 179
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== Search for `@napi-rs/simple-git` usage (any file type) ==="
rg -n "`@napi-rs/simple-git`" .
echo ""
echo "=== Search for common import identifiers (any file type) ==="
rg -n "simpleGit|SimpleGit" .Repository: genlayerlabs/genlayer-docs
Length of output: 4830
Remove or justify unused @napi-rs/simple-git dependency
@napi-rs/simple-git only appears in package.json (line 40) and package-lock.json; there are no imports/usages of @napi-rs/simple-git (or simpleGit/SimpleGit) elsewhere in the repo. Remove it or add a comment explaining the rationale/planned use.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@package.json` at line 40, The package.json entry "`@napi-rs/simple-git`":
"^0.1.22" is unused in the codebase; either remove this dependency from
package.json (and regenerate package-lock.json) or add a short comment in
package.json near the dependencies explaining why it's intentionally present
(e.g., planned feature, experimental local dev tool) and link to the PR/issue
that tracks its future usage; make sure to also update package-lock.json and run
npm install to keep lockfile consistent if you remove or justify the dependency.
…rofit) Adds frontmatter descriptions and answer-first intro paragraphs across the hand-written docs (concepts, features, examples, Studio guides, partners, validators), so each page leads with a self-contained definition — the unit RAG systems and answer engines retrieve. Generated with scripts/retrofit/retrofit.js, a Codex CLI harness with two verification gates per page: - mechanical: only frontmatter + the intro region (H1 to first H2) may change; code fences, links, H1, and everything below the first H2 must be byte-identical - LLM verifier: an independent codex exec call rejects revisions with unsupported factual claims or non-canonical terminology 87/96 pages passed first try, 7 more on retry. 2 left unchanged where the verifier kept flagging pre-existing source inconsistencies: - examples/llm-hello-world.mdx: prose says "comparative equivalence principle" but the code uses gl.eq_principle.strict_eq - tools/genlayer-studio.mdx: intro mixes "AI smart contracts" wording Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Content retrofit complete — the previously out-of-scope item is now in (commits 94 hand-written pages (concepts, features, examples, Studio guides, validators, partners) now have answer-first frontmatter descriptions and answer-first intro paragraphs — each page leads with a self-contained definition, which is the unit RAG systems and answer engines actually retrieve. How it was done:
87/96 passed on the first attempt, 7 more on retry. The verifier earned its keep — examples of real catches: a claimed behavior contradicted by an early-return in the example contract's code, "verifies GitHub profiles" when the contract only fetches them, and "smart contracts" where canon says "Intelligent Contracts". 2 pages intentionally left unchanged — the verifier kept flagging pre-existing inconsistencies in the source that need a human content decision:
Verified post-retrofit: structural scan of all 96 files (frontmatter valid, single description key, heading spacing), export regression guard passing (208 pages), full 🤖 Generated with Claude Code |
CI installs with npm ci, which requires the npm lockfile to match package.json; the dependency was added via pnpm and the lockfile was not regenerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces "AI smart contracts" / generic "smart contracts" with the canonical "Intelligent Contracts", leads the intro with the entity name, and adds the meta description the retrofit verifier had held back pending this terminology decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
pages/understand-genlayer-protocol.mdx (1)
49-53: ⚡ Quick winUse real headings instead of bold pseudo-headings.
**Validator Selection Mechanism**,**Validator Operational Framework**, and**Putting It All Together**should be Markdown headings so they generate anchors and improve doc navigation.Suggested edit
-**Validator Selection Mechanism** +## Validator Selection Mechanism Token holders bolster network security by delegating tokens to validator candidates. A deterministic function f(x) then randomly designates Leader-Validator and Validators for each transaction. This process promotes fairness, helps decentralize validation power, and strengthens GenLayer's security and trustlessness. -**Validator Operational Framework** +## Validator Operational Framework Each GenLayer validator node integrates: ... -**Putting It All Together** +## Putting It All Together With Optimistic Democracy guiding consensus and validators empowered by AI, GenLayer enables a new class of blockchain applications.Also applies to: 68-68
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pages/understand-genlayer-protocol.mdx` around lines 49 - 53, Replace the bold pseudo-headings with real Markdown headings so they generate anchors and improve navigation: change the bold lines "**Validator Selection Mechanism**", "**Validator Operational Framework**", and "**Putting It All Together**" to proper Markdown headings (e.g., "## Validator Selection Mechanism", "## Validator Operational Framework", "## Putting It All Together") wherever they appear in the file, ensuring the heading level matches surrounding structure and keeping the existing paragraph content intact.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pages/developers/intelligent-contracts/tools/genlayer-studio/validators.mdx`:
- Line 8: The Markdown links currently pointing to
"/core-concepts/optimistic-democracy" are likely broken; update those link
targets (both occurrences referenced in this file) to the correct docs tree path
"/understand-genlayer-protocol/core-concepts/optimistic-democracy" (or to a
working relative path within this docs tree) so the anchors resolve; locate the
literal link strings in this MDX (the occurrences of
"/core-concepts/optimistic-democracy") and replace them with the corrected path.
In `@scripts/retrofit/schema-revision.json`:
- Around line 4-7: The "description" property in the schema currently only
specifies "type: string" but must enforce the advertised constraints; update the
"description" property schema to include "minLength": 50 and "maxLength": 160
and add a pattern like "^[^\n\r]*$" (or equivalent no-newline regex) to
guarantee a single-line string so invalid Codex outputs are rejected at
validation time.
---
Nitpick comments:
In `@pages/understand-genlayer-protocol.mdx`:
- Around line 49-53: Replace the bold pseudo-headings with real Markdown
headings so they generate anchors and improve navigation: change the bold lines
"**Validator Selection Mechanism**", "**Validator Operational Framework**", and
"**Putting It All Together**" to proper Markdown headings (e.g., "## Validator
Selection Mechanism", "## Validator Operational Framework", "## Putting It All
Together") wherever they appear in the file, ensuring the heading level matches
surrounding structure and keeping the existing paragraph content intact.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 40fd40e5-1670-4807-a128-371998c9a192
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (100)
.gitignorepages/developers/decentralized-applications/architecture-overview.mdxpages/developers/decentralized-applications/dapp-development-workflow.mdxpages/developers/decentralized-applications/genlayer-js.mdxpages/developers/decentralized-applications/project-boilerplate.mdxpages/developers/decentralized-applications/querying-a-transaction.mdxpages/developers/decentralized-applications/reading-data.mdxpages/developers/decentralized-applications/writing-data.mdxpages/developers/intelligent-contracts/crafting-prompts.mdxpages/developers/intelligent-contracts/deploying.mdxpages/developers/intelligent-contracts/deploying/cli-deployment.mdxpages/developers/intelligent-contracts/deploying/deploy-scripts.mdxpages/developers/intelligent-contracts/deploying/deployment-methods.mdxpages/developers/intelligent-contracts/deploying/network-configuration.mdxpages/developers/intelligent-contracts/equivalence-principle.mdxpages/developers/intelligent-contracts/examples/fetch-github-profile.mdxpages/developers/intelligent-contracts/examples/fetch-web-content.mdxpages/developers/intelligent-contracts/examples/github-profile-projects.mdxpages/developers/intelligent-contracts/examples/github-profile-summary.mdxpages/developers/intelligent-contracts/examples/llm-hello-world-non-comparative.mdxpages/developers/intelligent-contracts/examples/prediction.mdxpages/developers/intelligent-contracts/examples/storage.mdxpages/developers/intelligent-contracts/examples/user-storage.mdxpages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdxpages/developers/intelligent-contracts/examples/wizard-of-coin.mdxpages/developers/intelligent-contracts/features.mdxpages/developers/intelligent-contracts/features/calling-llms.mdxpages/developers/intelligent-contracts/features/error-handling.mdxpages/developers/intelligent-contracts/features/image-processing.mdxpages/developers/intelligent-contracts/features/interacting-with-evm-contracts.mdxpages/developers/intelligent-contracts/features/interacting-with-intelligent-contracts.mdxpages/developers/intelligent-contracts/features/messages.mdxpages/developers/intelligent-contracts/features/non-determinism.mdxpages/developers/intelligent-contracts/features/random.mdxpages/developers/intelligent-contracts/features/special-methods.mdxpages/developers/intelligent-contracts/features/transaction-context.mdxpages/developers/intelligent-contracts/features/upgradability.mdxpages/developers/intelligent-contracts/features/value-transfers.mdxpages/developers/intelligent-contracts/features/vector-storage.mdxpages/developers/intelligent-contracts/features/web-access.mdxpages/developers/intelligent-contracts/first-intelligent-contract.mdxpages/developers/intelligent-contracts/ideas.mdxpages/developers/intelligent-contracts/security-and-best-practices/prompt-injection.mdxpages/developers/intelligent-contracts/tools/genlayer-cli.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/advanced-features/custom-plugins.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/contract-state.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/deploying-contract.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/development-tips.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/execute-transaction.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/limitations.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/loading-contract.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/monitoring-node-logs.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/providers.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/reset-the-studio.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/troubleshooting.mdxpages/developers/intelligent-contracts/tools/genlayer-studio/validators.mdxpages/developers/intelligent-contracts/types/address.mdxpages/developers/intelligent-contracts/types/collections.mdxpages/developers/intelligent-contracts/types/dataclasses.mdxpages/developers/intelligent-contracts/types/primitive.mdxpages/developers/intelligent-contracts/when-to-use-genlayer.mdxpages/developers/staking-guide.mdxpages/partners/chutes.mdxpages/partners/comput3.mdxpages/partners/genlayerlabs.mdxpages/partners/heurist.mdxpages/partners/ionet.mdxpages/partners/libertai.mdxpages/partners/morpheus.mdxpages/understand-genlayer-protocol.mdxpages/understand-genlayer-protocol/core-concepts.mdxpages/understand-genlayer-protocol/core-concepts/accounts-and-addresses.mdxpages/understand-genlayer-protocol/core-concepts/economic-model.mdxpages/understand-genlayer-protocol/core-concepts/genvm.mdxpages/understand-genlayer-protocol/core-concepts/large-language-model-llm-integration.mdxpages/understand-genlayer-protocol/core-concepts/non-deterministic-operations-handling.mdxpages/understand-genlayer-protocol/core-concepts/optimistic-democracy.mdxpages/understand-genlayer-protocol/core-concepts/optimistic-democracy/appeal-process.mdxpages/understand-genlayer-protocol/core-concepts/optimistic-democracy/finality.mdxpages/understand-genlayer-protocol/core-concepts/optimistic-democracy/slashing.mdxpages/understand-genlayer-protocol/core-concepts/optimistic-democracy/staking.mdxpages/understand-genlayer-protocol/core-concepts/optimistic-democracy/unstaking.mdxpages/understand-genlayer-protocol/core-concepts/rollup-integration.mdxpages/understand-genlayer-protocol/core-concepts/transactions.mdxpages/understand-genlayer-protocol/core-concepts/transactions/transaction-encoding-serialization-and-signing.mdxpages/understand-genlayer-protocol/core-concepts/transactions/transaction-execution.mdxpages/understand-genlayer-protocol/core-concepts/transactions/transaction-statuses.mdxpages/understand-genlayer-protocol/core-concepts/transactions/types-of-transactions.mdxpages/understand-genlayer-protocol/core-concepts/validators-and-validator-roles.mdxpages/understand-genlayer-protocol/core-concepts/web-data-access.mdxpages/understand-genlayer-protocol/optimistic-democracy-how-genlayer-works.mdxpages/understand-genlayer-protocol/typical-use-cases.mdxpages/validators/genvm-configuration.mdxpages/validators/monitoring.mdxpages/validators/system-requirements.mdxpages/validators/upgrade.mdxpublic/sitemap.xmlscripts/retrofit/retrofit.jsscripts/retrofit/schema-revision.jsonscripts/retrofit/schema-verdict.json
✅ Files skipped from review due to trivial changes (83)
- scripts/retrofit/schema-verdict.json
- pages/developers/intelligent-contracts/tools/genlayer-studio/development-tips.mdx
- pages/understand-genlayer-protocol/core-concepts/rollup-integration.mdx
- pages/validators/system-requirements.mdx
- pages/developers/decentralized-applications/reading-data.mdx
- pages/developers/intelligent-contracts/deploying/deployment-methods.mdx
- pages/validators/genvm-configuration.mdx
- pages/developers/intelligent-contracts/tools/genlayer-studio/reset-the-studio.mdx
- pages/developers/intelligent-contracts/tools/genlayer-studio/loading-contract.mdx
- pages/developers/intelligent-contracts/features/web-access.mdx
- pages/developers/intelligent-contracts/tools/genlayer-studio/contract-state.mdx
- pages/developers/intelligent-contracts/examples/fetch-github-profile.mdx
- pages/understand-genlayer-protocol/core-concepts/optimistic-democracy/finality.mdx
- pages/developers/intelligent-contracts/examples/storage.mdx
- pages/developers/intelligent-contracts/tools/genlayer-studio/troubleshooting.mdx
- pages/developers/decentralized-applications/project-boilerplate.mdx
- pages/developers/intelligent-contracts/tools/genlayer-studio/providers.mdx
- pages/developers/intelligent-contracts/crafting-prompts.mdx
- pages/partners/chutes.mdx
- pages/developers/intelligent-contracts/examples/wizard-of-coin.mdx
- pages/understand-genlayer-protocol/core-concepts/optimistic-democracy.mdx
- pages/developers/intelligent-contracts/tools/genlayer-cli.mdx
- pages/understand-genlayer-protocol/core-concepts/optimistic-democracy/staking.mdx
- pages/developers/intelligent-contracts/features/transaction-context.mdx
- pages/developers/intelligent-contracts/features/error-handling.mdx
- pages/developers/intelligent-contracts/types/dataclasses.mdx
- pages/developers/intelligent-contracts/types/primitive.mdx
- pages/understand-genlayer-protocol/core-concepts/transactions.mdx
- pages/developers/intelligent-contracts/features/vector-storage.mdx
- pages/partners/morpheus.mdx
- pages/developers/intelligent-contracts/examples/github-profile-summary.mdx
- pages/developers/intelligent-contracts/features/random.mdx
- pages/developers/intelligent-contracts/features/upgradability.mdx
- pages/developers/intelligent-contracts/deploying.mdx
- pages/understand-genlayer-protocol/core-concepts/transactions/transaction-statuses.mdx
- pages/developers/decentralized-applications/genlayer-js.mdx
- pages/developers/intelligent-contracts/examples/github-profile-projects.mdx
- pages/developers/intelligent-contracts/features.mdx
- pages/developers/intelligent-contracts/features/interacting-with-intelligent-contracts.mdx
- pages/developers/intelligent-contracts/features/value-transfers.mdx
- scripts/retrofit/retrofit.js
- pages/developers/decentralized-applications/writing-data.mdx
- pages/understand-genlayer-protocol/core-concepts/transactions/types-of-transactions.mdx
- pages/developers/staking-guide.mdx
- pages/developers/intelligent-contracts/types/collections.mdx
- pages/developers/intelligent-contracts/types/address.mdx
- pages/understand-genlayer-protocol/core-concepts/web-data-access.mdx
- pages/understand-genlayer-protocol/core-concepts/optimistic-democracy/appeal-process.mdx
- pages/validators/monitoring.mdx
- pages/understand-genlayer-protocol/core-concepts/validators-and-validator-roles.mdx
- pages/developers/intelligent-contracts/examples/vector-store-log-indexer.mdx
- pages/developers/intelligent-contracts/tools/genlayer-studio/execute-transaction.mdx
- pages/developers/intelligent-contracts/tools/genlayer-studio/advanced-features/custom-plugins.mdx
- pages/developers/intelligent-contracts/features/image-processing.mdx
- pages/developers/decentralized-applications/architecture-overview.mdx
- pages/developers/intelligent-contracts/ideas.mdx
- pages/developers/intelligent-contracts/examples/llm-hello-world-non-comparative.mdx
- pages/developers/decentralized-applications/dapp-development-workflow.mdx
- pages/understand-genlayer-protocol/typical-use-cases.mdx
- pages/understand-genlayer-protocol/core-concepts/large-language-model-llm-integration.mdx
- pages/developers/intelligent-contracts/examples/prediction.mdx
- pages/developers/intelligent-contracts/features/messages.mdx
- .gitignore
- pages/developers/intelligent-contracts/examples/fetch-web-content.mdx
- pages/developers/intelligent-contracts/features/special-methods.mdx
- pages/understand-genlayer-protocol/core-concepts/transactions/transaction-encoding-serialization-and-signing.mdx
- pages/developers/intelligent-contracts/features/non-determinism.mdx
- pages/developers/intelligent-contracts/tools/genlayer-studio/deploying-contract.mdx
- pages/developers/intelligent-contracts/equivalence-principle.mdx
- pages/developers/intelligent-contracts/deploying/cli-deployment.mdx
- pages/understand-genlayer-protocol/core-concepts/non-deterministic-operations-handling.mdx
- pages/understand-genlayer-protocol/core-concepts/accounts-and-addresses.mdx
- pages/developers/intelligent-contracts/deploying/network-configuration.mdx
- pages/developers/intelligent-contracts/features/interacting-with-evm-contracts.mdx
- pages/understand-genlayer-protocol/core-concepts.mdx
- pages/developers/intelligent-contracts/deploying/deploy-scripts.mdx
- pages/understand-genlayer-protocol/optimistic-democracy-how-genlayer-works.mdx
- pages/developers/intelligent-contracts/security-and-best-practices/prompt-injection.mdx
- pages/understand-genlayer-protocol/core-concepts/economic-model.mdx
- pages/developers/intelligent-contracts/examples/user-storage.mdx
- pages/developers/intelligent-contracts/tools/genlayer-studio/monitoring-node-logs.mdx
- pages/developers/intelligent-contracts/first-intelligent-contract.mdx
- pages/developers/decentralized-applications/querying-a-transaction.mdx
🚧 Files skipped from review as they are similar to previous changes (2)
- pages/understand-genlayer-protocol/core-concepts/transactions/transaction-execution.mdx
- public/sitemap.xml
The page claimed to demonstrate the comparative Equivalence Principle (linking a dead anchor) while its code used strict_eq on an LLM call — an anti-pattern per GenLayer guidance, since LLM output is non-deterministic and exact-match consensus fails. New example: the leader's LLM generates a short salute and validators judge whether the response is a salute via gl.eq_principle.prompt_non_comparative, which is the canonical pattern for open-ended LLM output. Page prose rewritten to match. Both this and the sibling non-comparative example now pass genvm-lint: the inline-lambda input tripped the linter's reachability analysis (storage write flagged as reachable from a non-deterministic block), so both use a named get_input function instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
AI systems are becoming a primary way developers discover and consume documentation — both answer engines (ChatGPT, Claude, Perplexity, Google AI Overviews) that cite docs in responses, and coding agents (Claude Code, Cursor) that fetch docs at runtime while writing GenLayer contracts. This PR makes docs.genlayer.com properly navigable and consumable for both, following the patterns established by Anthropic, Stripe, Cloudflare, and Vercel docs (a curated
llms.txtindex → per-page raw-markdown payloads → bundled exports).Two research findings shaped the approach:
llms.txt— but coding agents and user-triggered fetchers use it heavily. For developer docs, that's exactly the audience that matters.first-contract).What was broken (and is now fixed)
These were silent failures in the existing pipeline, found during the audit:
llms-full.txt— the generator only collected.mdxfiles, and these are.md. Several CLI command subtrees (accounts/,contracts/,environment/,configuration/,transactions/) were also dropped because the traversal followed_meta.jsonkeys that don't match the directory layout. An agent reading our "complete documentation" got no SDK or CLI content. The export went from ~150 to 208 pages.<Callout>,<Cards>,<Tabs>,importstatements appeared verbatim in the file we point agents at./developers/intelligent-contracts/advanced-features/*, a directory that no longer exists (renamed tofeatures/), e.g./developers/intelligent-contracts/error-handlingwas a dead end.genlayer-cli/_meta.jsonstill listed flat command keys (init,deploy,account…) from before the category regrouping (8a65f00), so most entries pointed at nothing. Now matches the realenvironment//contracts//etc. directories._temp/,_providers/,_advanced/) and omitted the.md-based API reference routes; nestedindexpages produced wrong URLs.partners/libertai.mdxexisted but wasn't in navigation (added);features/features.mdxwas a byte-equivalent duplicate card index (deleted + redirect).New LLM-facing surface
<page-url>.mdwith frontmatter (title,description,sourcecanonical URL,last_updatedfrom git). This is the single highest-impact pattern: agents fetch individual pages constantly, and this makes each fetch ~99% cheaper. Generated at build time intopublic/, so no runtime cost.llms.txt— previously 14 hand-written links for 214 pages; now generated from the nav tree covering all 208 pages with descriptions, grouped by section, every link pointing at the.mdversion (the Anthropic pattern), plus an instructions block for agents (the Stripe pattern) and anOptionalsection (partners) per the llms.txt spec.understand-genlayer-protocol|developers|validators|api-references/llms-full.txt. The rootllms-full.txtomits the 66 per-command CLI pages to stay at ~180K tokens (inside a single context window); the full CLI reference lives in the api-references bundle. Both stated in llms.txt so nothing is silently hidden.robots.txt— explicitly allows all major AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, etc.) and points to the LLM resources.Accept: text/markdowncontent negotiation (middleware) — agents can request any normal docs URL with that header and receive the markdown mirror. Browsers never send this header, so human traffic is untouched.Vary: Acceptis set so CDNs cache the two representations separately..mdURL returns atext/markdown404 listing the 5 closest-match pages (scored against llms.txt) plus the index links, so agents self-correct instead of parsing an HTML error page..mdmirrors.SEO / retrieval quality
<meta name=description>andog:titlewere one global string on every page; both are now per-page (frontmatter-driven, with answer-first descriptions hand-written for 11 key pages and first-paragraph extraction as fallback elsewhere).TechArticle+BreadcrumbListJSON-LD and a canonical URL on every page;rel=alternate type=text/markdownadvertises the mirror.gitTimestamp; required adding@napi-rs/simple-git); sitemaplastmodnow uses git commit dates instead of file mtime (mtime equals checkout time on CI, which falsely marks everything as modified today — a signal crawlers learn to distrust).Source:line per page so content chunks cleanly at heading boundaries.Guardrails
scripts/check-llm-exports.jsruns indev/buildand fails if export page counts drop below floors, JSX leaks into the exports, or known content anchors (SDK reference, CLI commands, validator guide) disappear. The original SDK-reference omission survived unnoticed for months — this makes that class of regression impossible to ship silently.Verification
next buildpasses (225 static pages + middleware).mdfetch → section bundle → Source citation, all on a local prod serverdocs.genlayer.com/llms.txt, a few<page>.mdURLs,robots.txt, andAccept: text/markdownDeploy notes
VERCEL_DEEP_CLONE=trueon Vercel or fetch full history in CI; if unavailable, dates degrade gracefully (hidden on pages, mtime fallback in sitemap).pnpm-lock.yamlis gitignored in this repo, so the new dev dependency (@napi-rs/simple-git) resolves at install time.llms.txt,llms-full.txt, section bundles,.mdmirrors,full-documentation.txt) are all gitignored and rebuilt byscripts/generate-full-docs.json everydev/build.Content retrofit (added in later commits)
94 hand-written pages now have answer-first descriptions and intros, produced by
scripts/retrofit/retrofit.js— a Codex CLI harness with a mechanical gate (only frontmatter + intro may change; everything from the first H2 onward must be byte-identical) and an independent LLM verifier per page (rejects unsupported claims / non-canonical terminology). 2 pages were left unchanged because the verifier surfaced pre-existing source inconsistencies needing a human decision (see PR comments).Deliberately out of scope
_advanced/examples (kept hidden by team decision)🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Acceptheaders.Bug Fixes
Documentation
Chores