Spike: Investigate approaches for traversing DSL schema and extracting field metadata

[lornakelly]

Description

Investigate approach for traversing the Open Workflow DSL JSON Schema to extract comprehensive field metadata.

Motivation

In order to build a dynamic form and offer all available options for a task type we need to be able to pull that data out

Proposed Implementation

No response

Definition of Done

• Implementation: Fully implemented according to the Open Workflow spec.
• Unit Tests: Comprehensive unit tests are included and passing.
• Integration Tests: Verified within the monorepo and target environments (Web/VS Code).
• Documentation: Updated README.md, ADRs, or official docs.
• Performance: No significant regression in editor responsiveness.
• Accessibility: UI changes comply with accessibility standards.

[lornakelly]

The Problem

You cannot ask this schema "what fields does a Call task have?" and get an answer, because the schema doesn't store fields that way - it stores rules for checking a document, and those rules are scattered and conditional.

• Ask for #/$defs/callTask and you get no fields at all, you get a choice of seven.
• Open with and you reach endpoint, which is not a field either - it's a pointer to another definition that is itself a choice of three.

We need to investigate how to potentially implement something that walks that chain - follows the pointers, merges the inherited half with the specific half, so everything downstream can ask one simple question and get back a straight list of fields.

As of now, the schema has 47 $defs total.

13 of those are tasks:

callTask
doTask
emitTask
forTask
forkTask
listenTask
raiseTask
runTask
setTask
switchTask
tryTask
waitTask

Others:

taskBase
taskList
flowDirective
timeout
duration
input
output
export
schema
endpoint
externalResource
uriTemplate
catalog
runtimeExpression
authenticationPolicy
referenceableAuthenticationPolicy
secretBasedAuthenticationPolicy
oauth2AuthenticationProperties
oauth2Token
error
errorFilter
retryPolicy
eventProperties
eventFilter
eventConsumptionStrategy
subscriptionIterator
asyncApiServer
asyncApiSubscription 
asyncApiInboundMessage
asyncApiOutboundMessage
asyncApiMessageConsumptionPolicy
containerLifetime
processResult 
extension

[handreyrc]

Schema Filter — Spike Summary

Goal

The Serverless Workflow specification is described as a single large
workflow.json file (JSON Schema Draft 2020-12). Every definition lives inside
a shared $defs block and is composed together via $ref pointers and allOf
arrays.

The spike explored whether we could filter and produce a complete, standalone
schema for a single named definition — one that:

• has no remaining allOf constructs (fully resolved),
• bundles all transitive dependencies it still needs into its own $defs block,
• can be handed directly to a JSON Schema validator or a field
  documentation renderer without any access to the original full schema.

---

Library used for merging — allof-merge

allof-merge (v0.6.8) performs two
operations in a single pass over the schema:

1. $ref resolution — replaces every #/$defs/<name> pointer that
   participates in an allOf composition with the schema object it points to.
2. allOf flattening — merges allOf compositions into the parent object,
   so inherited taskBase fields (if, input, output, export, timeout,
   then, metadata) are inlined directly into each task's own properties
   map.

The merge is invoked once with the full schema as both input and reference
source:

import { merge } from "allof-merge";

_merged = merge(workflowSchema, { source: workflowSchema });

The result is stored in a module-level cache (_merged). All subsequent calls
reuse this cached object, so the expensive merge only happens once per process
lifetime.

Note: allof-merge does not inline standalone $refs that appear inside
properties values — those remain as pointers. That is intentional: those
refs are bundled into the output's own $defs block so consumers can resolve
them without the original schema.

---

How a definition is filtered — step by step

1. Merge the full schema (once, cached)

getMergedSchema() calls allof-merge on workflow.json. After this step,
every allOf in the schema is gone and every $ref that participated in an
allOf composition is inlined. The result is a new schema object kept in
_merged.

2. Look up the requested definition

The entry _merged.$defs[definitionName] is retrieved. If it does not exist an
error is thrown immediately.

3. Collect transitive $ref dependencies

collectTransitiveRefs walks the definition node recursively (using a
WeakSet to handle cycles safely) and collects the name of every
#/$defs/<name> pointer it finds — including pointers inside nested schemas
reachable from those names. The result is a Set<string> of every definition
the output will still need.

4. Deep-clone the definition

cloneDeepSafe deep-clones the definition node using a traversal-stack Set
to break true cyclic references without dropping non-cyclic shared DAG nodes.

5. Bundle transitive dependencies into $defs

Every name collected in step 3 is looked up in _merged.$defs and
deep-cloned into a $defs block on the result object. This makes the output
self-contained: a validator or documentation renderer can resolve all remaining
$ref pointers using only the returned object.

6. Strip inherited annotation titles

transformSchema does a single recursive pass and removes title values that
originated from taskBase property schemas — they are inherited during the
allOf merge and would otherwise shadow the correct task-level title.

7. Cache and return

The result is stored in a Map keyed by definition name. Repeated calls for
the same name return the identical cached object in O(1).

---

What we learned

• allof-merge handles the Serverless Workflow schema cleanly — all 12 task
  types flatten correctly, including the callTask discriminated oneOf with
  7 variants.
• A single full-schema merge followed by per-definition extraction is
  significantly simpler and faster than trying to resolve definitions
  individually.
• The output includes the full taskBase fields (if, input, output,
  export, timeout, then, metadata) alongside the task-specific own
  fields. This makes it correct for validation and field documentation. Further
  trimming (e.g. stripping base fields for a form renderer) can be layered on
  top if needed in a follow-up.
• The approach is entirely synchronous and dependency-free at runtime beyond
  allof-merge itself, which is declared as a proper package dependency.

---

getSchemaForDefinition

Resolves any named definition from the bundled workflow.json schema into a
fully formed, self-contained JSON Schema object. Every allOf composition
is merged — including the shared taskBase fields inherited by all tasks — and
all transitive $ref dependencies are bundled inline, so the result can be
handed directly to a validator or documentation renderer with no further
processing.

The function is synchronous. The full merge is computed once on the first
call and cached at module level; all subsequent calls are O(1) lookups.

import { getSchemaForDefinition } from "@openworkflowspec/diagram-editor";

// Resolve the full schema for the "setTask" definition
const schema = getSchemaForDefinition("setTask");

// schema is a plain Record<string, unknown> — no allOf remaining.
// Every field is inlined, including the inherited taskBase properties:
// schema.properties => { if, input, output, export, timeout, then, metadata, set }

Resolving a sub-definition

getReferencedDefinitions returns the names of all $defs entries that a
given task directly references (excluding structural scaffolding like taskBase
and taskList). Pass any of those names back to getSchemaForDefinition to
get a fully resolved, standalone schema for that sub-definition.

import {
  getSchemaForDefinition,
  getReferencedDefinitions,
} from "@openworkflowspec/diagram-editor";

// Which named sub-definitions does tryTask reference?
const subDefs = getReferencedDefinitions("tryTask");
// => ["errorFilter", "retryPolicy"]

// Resolve the retryPolicy schema on its own
const retryPolicySchema = getSchemaForDefinition("retryPolicy");

Source and PoC

Source schema-filter
Sandbox for Schema validation and form generation

Note: A story named "Schema Filter" was created under "Features" for testing purposes.

[lornakelly]

@handreyrc That looks good to me, really good starting point, great work! I will create a ticket to implement, think it is good to keep the story in as well temporarily as an aid for developing. Just a note when implementing, lets bump the sdk and pull the workflow json from there instead of adding to editor as well, that way we can stay in line with the sdk https://github.com/open-workflow-specification/sdk-typescript/blob/main/src/lib/schema.ts#L26

[lornakelly]

As part of our investigation, we developed a POC for schema resolution within the SDK, which proved as a potential option though not fully implemented. However, we ultimately decided against adopting this solution because our specific use case requires custom schema resolution for form building and our specific case. Once our implementation is fully complete, we will review whether any components of our approach would be valuable to contribute back to the SDK for the broader community.

POC can be found here https://github.com/lornakelly/sdk-typescript/tree/schema-resolution-poc

[fantonangeli]

Thanks @handreyrc for this investigation.
I agree with both of you to keep the schema resolution logic in the DE until we have tested it for a while, to make easier to modify and fix it.
I played a bit with @handreyrc POC and it works well 🚀
Only one small comment "Serverless Workflow specification" -> "Open Workflow specification" 😄
For the rest is perfect