feat(agent): OpenUI bindings — library/fragment builders, toUIOutput, getUiStream() (DEV-773) by LukasParke · Pull Request #92 · OpenRouterTeam/typescript-agent

LukasParke · 2026-07-31T21:05:35Z

Summary

The Agent-SDK half of the OpenUI backport from Noetic (DEV-773, part of the DEV-765 umbrella; spec: DEV-764 RFC).

Deliberately thin per the spec: the API owns parsing/prompting/validation; the SDK ships builders, the tool render surface, and stream access.

New `src/lib/openui/` module

defineComponent / createLibrary — component vocabulary from Zod prop schemas; prop declaration order is normative (positional args in OpenUI Lang map by order)
fragment(library) + uiRef/uiState/uiBuiltin — typed constructors for tool-authored UI; literal props validated at construction time
openui(library) — produces the wire-shaped {id:'openui', library, dialect} plugin preference (Zod → JSON Schema); plugins already type-flows through CallModelInput, so no callModel signature change
translateUiEvent + UI stream event model

`tool()` render surface

Optional toUIOutput sibling of toModelOutput on regular/generator/HITL tools
Successful executions broadcast a tool.ui_fragment stream event (render-only — never sent to the model; throwing toUIOutput degrades to no-fragment)

`ModelResult.getUiStream()`

Streams UI events across all turns: tool-authored fragments + the API's response.openui.statement/fragment/document wire events
Wire events not yet in the SDK's stream-event union arrive via its forward-compat Unknown catch-all; translation reads the raw payload, so the stream works before and after the SDK regen (DEV-772)
Implements both the no-tools fast path (with hooks-session finalization) and the multi-turn broadcaster path

Deferred (documented in the ticket)

uiSubmitted()/uiInteracted()/uiToAssistant() stop predicates — need interaction events that only exist once Phase-3 surface state (DEV-774) lands
Items-stream surfacing of UI events; fragments on the auto-approve/pending-state paths

Test plan

28 new tests (openui.test.ts, openui-stream.test.ts): serialization, library ordering/validation, fragment builder, plugin wire shape, event translation (incl. Unknown encoding), getUiStream fast path, broadcastUiFragment success/skip/throw paths
Full suite: 774/774 passing (62 files), typecheck + biome clean

🤖 Generated with Claude Code

API example

import {
  callModel,
  createLibrary,
  defineComponent,
  fragment,
  openui,
  tool,
} from '@openrouter/agent';
import { z } from 'zod/v4';

const library = createLibrary([
  defineComponent({
    name: 'Text',
    props: z.object({ value: z.string() }),
  }),
]);

const ui = fragment(library);
const greeting = tool({
  name: 'greeting',
  inputSchema: z.object({ name: z.string() }),
  execute: ({ name }) => ({ message: `Hello, ${name}!` }),
  toUiOutput: ({ output }) => ui.Text(output.message),
});

const result = callModel(client, {
  model: 'anthropic/claude-sonnet-4.5',
  input: 'Greet Luke',
  tools: [greeting],
  plugins: [openui(library)],
});

for await (const event of result.getUiStream()) {
  if (event.type === 'fragment') {
    console.log(event.source);
  }
}

…r (DEV-773) The SDK half of the OpenUI backport from Noetic (DEV-765). Adds packages/agent/src/lib/openui/: defineComponent/createLibrary (Zod props with normative declaration order), the typed fragment() builder with uiRef/uiState/uiBuiltin, OpenUI Lang expression serialization, and the openui(library) helper producing the wire-shaped plugin preference (Zod -> JSON Schema). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…Stream() (DEV-773) Tools can now author OpenUI render fragments: an optional toUIOutput sibling of toModelOutput on every executable tool shape. Fragments are broadcast as tool.ui_fragment stream events after successful execution (render-only, never sent to the model; a throwing toUIOutput degrades to no-fragment). getUiStream() on ModelResult surfaces UI events across all turns: tool-authored fragments plus the API's response.openui.* wire events (statement/fragment/document) from the openui plugin. Wire events not yet in the SDK's stream-event union arrive via its forward-compat Unknown catch-all, so translation reads the raw payload — the stream works both before and after the SDK regen (DEV-772). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

New private package @openrouter/openui-playground: a local webapp for testing, benching, and evaluating OpenUI generative-UI support. - Progressive renderer over the demo component library (Stack/Card/ Heading/Text/Stat/Badge/Table/Input/Select/Button/Progress) — UI materializes statement-by-statement mid-stream - Two modes with identical event shapes: emulate (local library prompt + reference streaming parser over the text stream — works today) and native (openui() plugin + getUiStream() — flips on when DEV-771/772 land), so the paths can be A/B'd from the history table - Bench stats per run: TTFB, first-statement latency, total time, statement/diagnostic counts, token usage, cost; session history for comparing models and prompts - Reference incremental OpenUI Lang parser (the same logic DEV-770 ports into openrouter-web) with 11 conformance tests - Plain node:http + static client; no build step Verified end-to-end against live models: single-card and 12-statement dashboard prompts parse clean (0 diagnostics) and render progressively. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

serializeExpr emitted object keys raw, so a key with spaces, quotes, punctuation, or a leading digit produced source the parser rejects. Keys come from arbitrary tool-authored objects via toExpr, so they cannot be assumed to be identifiers. The grammar already accepts a quoted key (parseObject branches on '"'), so quoting the rest round-trips. String(NaN)/String(Infinity) also emitted bare identifiers, which parse back as refs to undefined names. JSON resolves the same hole as null; do that rather than emit source that cannot round-trip. fix(playground): describe enum props by their values, not "string" describeSchema returned on json.type before checking json.enum, but an enum serializes as {type: 'string', enum: [...]} — so every enum prop was described to the model as a plain string and it never saw which values are legal for Badge.tone, Stack.direction, Button.variant. docs(changeset): add the required minor changeset for the OpenUI exports ~30 new exports plus ModelResult.getUiStream and the toUIOutput tool option had no changeset, which the public-api-examples skill requires. The example is compile-checked against the real signatures.

The gate's no_god_files rule is fan-out > 15, not file size: model-result.ts sat at exactly 15 outbound edges and this PR's ./openui/ui-stream.js import made it 16. Verified by removing that one import — the violation disappears. Re-exported UiStreamEvent and translateUiEvent from stream-transformers.js, which model-result.ts already depends on and which owns every other wire-event translation the loop performs, so no new edge is added. Complex functions were 9 -> 13, all four new here. Each is split along a seam it already had: - translateUiEvent: one function per wire event type (cc=18 -> under) - scanStatements: string-literal and bracket-depth state machines extracted - generate: the native path's per-variant event mapping extracted - renderCall: form controls and Table extracted to renderControl/renderTable Verified with sentrux 0.5.7, the version CI pins: God files 0 -> 0, complex functions back to 9 (the 9 remaining are all pre-existing on main and untouched), gate reports 'No degradation detected'. Behavior unchanged — full suite green, typecheck and lint clean.

The table of what each stream emits is the reference consumers use to pick one; getUiStream was absent.

…escript-agent-openui-module-libraryfragment-builders

synapse-github-agent · 2026-08-03T23:29:11Z

cortex review — `6898806`

✅ Security · ⚠️ Experience (DX · UX · A11y) — incomplete · ⚠️ Performance — incomplete

Experience (DX · UX · A11y)

⚠️ Review incomplete for this category (failed — Invalid final response: empty or invalid output) — findings may be missing.

Performance

⚠️ Review incomplete for this category (failed — Invalid final response: empty or invalid output) — findings may be missing.

✔ 4 resolved since last push

uiRef / uiState / uiBuiltin names are serialized into OpenUI Lang source with no identifier validation. The serializer deliberately hardens every other value channel — strings go through JSON.stringify, non-identifier object keys are quoted, non-finite numbers become null (packages/agent/src/lib/openui/document.ts:64-113) — but ref/state/builtin names are emitted verbatim (case 'ref': return expr.name). A toUIOutput implementation that derives a ref from its input (which is model-controlled tool arguments) lets a malicious/injected model break out of the literal quoting and inject arbitrary expressions into the fragment, e.g. uiRef('x), @Run(dangerous_mutation'), which downstream renderers parse as extra action steps in what the client treats as trusted tool-authored UI. Validate names against /^[A-Za-z_][A-Za-z0-9_]*$/ at construction time. (packages/agent/src/lib/openui/fragment.ts:69)
Model-controlled parser diagnostics are interpolated into innerHTML unescaped → DOM XSS. d.source is passed through escapeHtml, but d.message beside it is not, and diagnostic messages embed raw model text: ExprParser.parseComplete throws trailing content after expression: '${this.src.slice(this.pos)}' (packages/openui-playground/src/lang/parser.ts) with the remainder of the model's line, and unexpected character '${ch}' / expected identifier at '...' do the same. A model output line such as a = Text("x") <img src=x onerror=...> becomes a diagnostic whose message contains the tag, which is then written into the document via innerHTML. Any prompt-injected or malicious model response therefore executes script in the playground origin — the same origin that can POST /api/generate with the server's API key. Same class at packages/openui-playground/public/app.js:447, where the user-supplied h.model string is interpolated into the history table's innerHTML. Fix: run every interpolated field through escapeHtml (or build the nodes with textContent). (packages/openui-playground/public/app.js:572)
API-key-backed generation endpoint listens on all interfaces with no auth or origin check. server.listen(PORT, …) defaults to 0.0.0.0, so anyone on the same network (café/office LAN, container host) can POST /api/generate with an arbitrary prompt/system/model and spend the developer's OPENROUTER_API_KEY (packages/openui-playground/src/server.ts:29-36); there is no rate limit or body-size cap on the path either (readBody, server.ts:58-65). Bind explicitly to 127.0.0.1 for a local-only tool. (packages/openui-playground/src/server.ts:190)
Static-file guard uses a bare string prefix check, so sibling directories escape the public root. normalize(join(PUBLIC_DIR, rel)) followed by file.startsWith(PUBLIC_DIR) accepts .../openui-playground/public-anything/...: a request for /../public.bak/secret resolves outside public/ yet still passes the prefix test and is served. Compare against PUBLIC_DIR + path.sep (or use path.relative and reject results starting with ..). (packages/openui-playground/src/server.ts:94)

_{Automatic first-pass review · updated in place on every push}

Two findings from cortex's review pass. XSS: the diagnostics panel escaped `source` but interpolated `message` and `line` raw into innerHTML. Every field there is model-controlled — `ParseFailure.message` is built from the offending source line, and in native mode diagnostics arrive verbatim off the wire — so a model could inject markup by emitting a crafted statement. All three fields are now escaped. A11y: rendered Input/Select carried no accessible name, so a screen reader announced an unlabelled field. Both signatures already have a `name` prop that was going unused for labelling; it now sets aria-label (and the real `name` attribute), falling back to the placeholder for Input.

synapse-github-agent

cortex panel verdict: comment — details in the consolidated review comment.

- escape diagnostic messages and history model names before innerHTML (XSS) - bind playground server to 127.0.0.1 (API-key-backed endpoint) - require trailing separator in static-file public-root prefix check - validate uiRef/uiState/uiBuiltin names as identifiers at construction - aria-label form controls from their name prop; progressbar ARIA + text % - aria-live status/diagnostics regions; error frames no longer overwritten by the green 'done' status - rename toUIOutput -> toUiOutput (match Ui casing convention pre-release) - warn (tool name + call id) when toUiOutput throws instead of catch {} - collect toUiOutput broadcasts and await as one batch off the follow-up critical path - sticky regexes + charCode skipWs in the playground parser (was O(n^2)) - memoize openui(library) wire shape per library (WeakMap) - drop the playground's no-op build script / outDir

…yfragment-builders' of https://github.com/OpenRouterTeam/typescript-agent into lukeparke/dev-773-typescript-agent-openui-module-libraryfragment-builders-2 # Conflicts: # packages/openui-playground/public/app.js

synapse-github-agent

cortex panel verdict: approve — details in the consolidated review comment.

…dule-libraryfragment-builders

synapse-github-agent · 2026-08-04T02:31:02Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/index.ts (resolver produced a line not present on any side: --- context after this hunk ---). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T02:32:45Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/lib/tool.ts (resolver produced a line not present on any side: Assistant). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T02:33:16Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/index.ts (resolver produced a line not present on any side: --- context after this hunk ---). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T02:45:35Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/lib/model-result.ts (resolver produced a line not present on any side: C: Analyze the semantics of both sides and combine them.). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T02:48:27Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/index.ts (resolver produced a line not present on any side: --- context after this hunk ---). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T03:00:32Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/lib/model-result.ts (resolver produced a line not present on any side:
). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T03:07:52Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/lib/model-result.ts (resolver produced a line not present on any side:
). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T03:10:16Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/lib/model-result.ts (resolver produced a line not present on any side:
). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T03:11:05Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/lib/model-result.ts (resolver produced a line not present on any side:
). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T03:28:07Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/lib/tool.ts (resolver produced a line not present on any side: --- context after this hunk ---). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T16:59:40Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/index.ts (resolver produced a line not present on any side: Ours and theirs are independent additions; interleave alphabetically: ToolTaskHa). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T17:03:13Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/index.ts (resolver produced a line not present on any side: ToolTaskHandle,). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T17:07:43Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/index.ts (resolver produced a line not present on any side: ToolTaskHandle,). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T17:08:28Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/index.ts (resolver produced a line not present on any side: ToolTaskHandle,). Manual update needed; label cortex-merge-conflict added.

synapse-github-agent · 2026-08-04T17:36:54Z

⚠️ cortex keep-fresh: could not auto-merge packages/agent/src/index.ts (resolver produced a line not present on any side: ToolTaskHandle,). Manual update needed; label cortex-merge-conflict added.

# Conflicts: # packages/agent/README.md # packages/agent/src/index.ts # packages/agent/src/lib/model-result.ts # packages/agent/src/lib/tool-types.ts # packages/agent/src/lib/tool.ts # pnpm-lock.yaml

… conflicts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration

Devin Review found 1 new potential issue.

7 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-08-31T21:16:12Z

+  const cached = wireCache.get(library);
+  if (cached) {
+    return cached;
+  }
+  const plugin = buildPlugin(library);
+  wireCache.set(library, plugin);


🟡 Library mutations return stale plugins

openui caches the first wire payload, although UiLibrary and its component definitions remain mutable. Later changes affect fragment building but never subsequent plugin requests.

Prompt for agents

The WeakMap cache in packages/agent/src/lib/openui/plugin.ts assumes UiLibrary is immutable, but packages/agent/src/lib/openui/library.ts exposes writable fields and retains mutable component definitions and a mutable Map. Either make createLibrary produce a genuinely immutable snapshot, including cloned/frozen definitions and schema-related state, or remove/version the cache so openui() always reflects the current library. Ensure fragment(library) and openui(library) cannot observe different dialects or component metadata after a mutation.

Was this helpful? React with 👍 or 👎 to provide feedback.

LukasParke and others added 2 commits July 31, 2026 15:53

This comment was marked as resolved.

Sign in to view

LukasParke added 4 commits August 3, 2026 13:59

docs(agent): list getUiStream in the README stream table

e5759d3

The table of what each stream emits is the reference consumers use to pick one; getUiStream was absent.

Merge remote-tracking branch 'origin/main' into lukeparke/dev-773-typ…

4eb82c7

…escript-agent-openui-module-libraryfragment-builders

This comment was marked as resolved.

Sign in to view

LukasParke added the cortex-keep-updated cortex keeps this PR up to date with its base branch label Aug 3, 2026

synapse-github-agent Bot reviewed Aug 4, 2026

View reviewed changes

LukasParke added 2 commits August 3, 2026 19:07

Merge branch 'lukeparke/dev-773-typescript-agent-openui-module-librar…

6898806

…yfragment-builders' of https://github.com/OpenRouterTeam/typescript-agent into lukeparke/dev-773-typescript-agent-openui-module-libraryfragment-builders-2 # Conflicts: # packages/openui-playground/public/app.js

synapse-github-agent Bot approved these changes Aug 4, 2026

View reviewed changes

Merge branch 'main' into lukeparke/dev-773-typescript-agent-openui-mo…

20ab446

…dule-libraryfragment-builders

synapse-github-agent Bot added the cortex-merge-conflict cortex could not auto-merge; manual update needed label Aug 4, 2026

LukasParke removed cortex-merge-conflict cortex could not auto-merge; manual update needed cortex-keep-updated cortex keeps this PR up to date with its base branch labels Aug 6, 2026

Merge remote-tracking branch 'origin/main' into wt/pr92

20c0e5c

# Conflicts: # packages/agent/README.md # packages/agent/src/index.ts # packages/agent/src/lib/model-result.ts # packages/agent/src/lib/tool-types.ts # packages/agent/src/lib/tool.ts # pnpm-lock.yaml

devin-ai-integration Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/openui-playground/public/app.js Outdated

Comment thread packages/openui-playground/public/app.js

Comment thread packages/openui-playground/src/server.ts Outdated

Comment thread packages/agent/src/lib/openui/library.ts

This comment was marked as resolved.

Sign in to view

fix(openui): address review findings

3d4a582

This comment was marked as resolved.

Sign in to view

refactor(openui): simplify native diagnostics counting

3cefbdc

This comment was marked as resolved.

Sign in to view

LukasParke added 4 commits August 11, 2026 14:29

fix(agent): bound OpenUI fragment rendering

829a94c

fix(openui): finish review feedback

6701156

fix(openui): release timed-out UI renders

ff70982

fix(openui): drain renders added before close

e956e1e

This comment was marked as resolved.

Sign in to view

fix(agent): drain UI fragments before stream completion

0f7eb50

This comment was marked as resolved.

Sign in to view

LukasParke added 2 commits August 11, 2026 16:23

fix(agent): isolate UI stream lifecycle

bccd1ad

fix(agent): release exited UI consumers

cbfd884

This comment was marked as resolved.

Sign in to view

fix(agent): merge OpenUI event streams

1f68c5e

This comment was marked as resolved.

Sign in to view

LukasParke and others added 3 commits August 11, 2026 17:18

fix(agent): render deferred tool UI results

11eac75

Merge origin/main: resolve replay-buffer, correlated-event and OpenUI…

8f2cb86

… conflicts

chore: resolve origin/main merge conflicts

d0b4bb3

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot reviewed Aug 31, 2026

View reviewed changes

Uh oh!

Conversation

LukasParke commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New src/lib/openui/ module

tool() render surface

ModelResult.getUiStream()

Deferred (documented in the ticket)

Test plan

API example

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

synapse-github-agent Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cortex review — 6898806

Experience (DX · UX · A11y)

Performance

Uh oh!

This comment was marked as resolved.

Uh oh!

synapse-github-agent Bot left a comment

Choose a reason for hiding this comment

Uh oh!

synapse-github-agent Bot left a comment

Choose a reason for hiding this comment

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

synapse-github-agent Bot commented Aug 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

LukasParke commented Jul 31, 2026 •

edited

Loading

New `src/lib/openui/` module

`tool()` render surface

`ModelResult.getUiStream()`

synapse-github-agent Bot commented Aug 3, 2026 •

edited

Loading

cortex review — `6898806`