fix(celld): make command lifecycle durable and fast

[patrickleet]

What this PR is

This is the hardening layer for the portable command host and celld support introduced in #206.

The core idea is that a domain command is declared once, then the application decides where to run it:

• in the normal service-oriented host (SOA), backed by the application's SQL repository, locks, and message bus;
• in a celld aggregate cell, backed by private SQLite and a single writer for that shard; or
• in both styles in the same application, selected per command/aggregate.

The domain crate does not import sqlx, QueuedRepository, Durable Objects, celld, NATS, or GraphQL transport code. Deployment topology is host configuration, not domain behavior.

This PR makes that model durable and responsive end to end: commands have fenced, restart-safe receipts; Eventual delivery no longer rebuilds the service graph after idle polls; exact command results settle client optimism without an unnecessary collection refetch; and authorization is preserved across the GraphQL-to-cell boundary.

Mental model

flowchart LR
    C[Generated client] --> G[GraphQL command field]
    G --> H{CommandHost route?}
    H -->|not selected| S[SOA / LocalCommandHost]
    H -->|selected| D[celld]
    S --> SQL[(application SQL)]
    D --> CELL[aggregate shard<br/>one writer + private SQLite]
    S --> O[outbox]
    CELL --> O
    O --> B[NATS / Kafka / RabbitMQ]
    B --> P[Eventual projectors]
    P --> RM[(SQL read models)]
    RM --> G

Loading

A cell is a consistency boundary, not another SQL dialect. One cell instance owns one aggregate shard, such as todo:{todo_id}. GraphQL, global queries, @live, projectors, and identity ingestors remain outside cells.

1. Declare the command once in the domain

The Todo domain's real complete command is representative:

portable_command! {
    name: "todo.complete",
    transition: domain_commands::Complete,
    aggregate: Todo,
    input: TodoCompleteInput,
    outcome: Eventual<TodoStatusPayload>,
    shard: |input| input.todo_id.clone(),
    load: required,
    roles: ["user", "admin"],
    field: "todos_complete",
    invoke: |todo, _input, principal| todo.complete(principal),
    payload: |todo| TodoStatusPayload::from_todo(&**todo),
}

That declaration carries everything both hosts must agree on:

• name is the stable command identity.
• aggregate and shard identify the consistency boundary. SOA aggregate loading and the celld route must resolve the same shard key.
• roles and field describe the generated GraphQL command surface.
• outcome preserves the application's Atomic<T> or Eventual<T> contract; choosing celld does not silently change the public consistency contract.
• invoke and payload are domain behavior and result mapping.

Simple transitions use load + invoke + payload. Commands needing custom creation, guards, or orchestration can use the handle escape hatch; for example, todo.create uses an authenticated guard, a custom handler, and a generated UUIDv7 default while remaining portable.

Handlers continue to receive CausalCommandContext<'_, A> and use ctx.repo(). There is deliberately no ctx.cell(): a handler must not know whether SOA or celld is executing it.

See the complete declarations in todo-domain/src/commands.rs.

2. Run the commands as a normal SOA service

Mount the portable declarations on ordinary aggregate routes:

Routes::for_aggregate::<R, L, Todo, S>(repo, locks, read_models)
    .mount(todo_domain::commands::create())
    .mount(todo_domain::commands::rename())
    .mount(todo_domain::commands::complete())
    .mount(todo_domain::commands::reopen())
    .mount(todo_domain::commands::archive())
    .mount(todo_domain::commands::purge())
    .modeled_projector(todo_projector)
    .handle(project_todos);

With the normal local command host, dispatch stays in the service process. The host supplies the configured repository, transaction/lock implementation, outbox, and bus. This is the simplest deployment when one database is the desired consistency boundary.

The runnable example is service/src/modules/todo.rs.

3. Run selected commands in celld

The cell worker mounts the same declarations:

let cell = AggregateCell::<Todo>::new(todo_id)?
    .mount(todo_domain::commands::create())
    .mount(todo_domain::commands::rename())
    .mount(todo_domain::commands::complete())
    .mount(todo_domain::commands::reopen())
    .mount(todo_domain::commands::archive());

The GraphQL host then declares which command names should wait-dispatch to that cell class:

const TODO_CELL_COMMANDS: &[&str] = &[
    "todo.create",
    "todo.rename",
    "todo.complete",
    "todo.reopen",
    "todo.archive",
    "todo.force_archive",
    "todo.purge",
];

pub fn celld_route() -> CelldRoute {
    CelldRoute::new(
        TODO_CELL_COMMANDS,
        "todo",
        todo_shard,
        graphql_todo_payload,
    )
}

todo_shard reads todo_id, so all transitions for one Todo reach the same todo:{todo_id} instance. That instance has one writer and private SQLite for events, snapshots, its durable command ledger, and its outbox.

The concrete route is in todo-service/src/host.rs, and the workers-rs cell mounts are in tests/celld/worker/src/lib.rs.

4. Mix SOA and celld in one application

Routing is additive and opt-in:

let host: SharedCommandHost = Arc::new(
    CelldCommandHost::new(celld_url, service, publisher)
        .route(todo::celld_route())
        .route(chat::celld_route()),
);

CelldCommandHost checks the registered routes for each command. A match goes to celld; a command with no celld route automatically falls back to the local SOA service.

The e2e application is intentionally hybrid:

Workload	Host
Todo transitions	celld, sharded by todo_id
chat.post	celld, sharded by message_id
Blob Atomic commands	local SOA fallback
GraphQL queries and @live	GraphQL service
Todo/Chat projectors	service consumers writing global SQL read models
Zitadel ingestion	service host

That means a team can move only a hot or contention-heavy aggregate to cells without rewriting its domain handlers, changing the GraphQL schema, or moving unrelated services. Adding or removing a CelldRoute changes placement; it does not fork domain behavior.

See the complete hybrid host in graphql-service/src/host.rs.

How a celld command completes

1. The generated client sends a stable command ID, input, and session through its GraphQL command field.
2. GraphQL authenticates the caller. Only the verified service identity and principal partition are forwarded on the trusted internal cell request; public input cannot spoof them.
3. CelldCommandHost selects a route and derives the cell shard from the command input.
4. The cell reserves the command ID in a durable, fenced ledger before execution.
5. The portable handler runs against the cell-local aggregate repository. Its events, snapshot state, receipt, and outbox work are persisted in the cell's SQLite boundary.
6. The host drains the cell outbox through the configured MessagePublisher; normal Eventual projectors consume those events and update global SQL read models.
7. The GraphQL host maps the cell result back to the generated command payload and seals the normal Distributed protocol/projection metadata.
8. The client replaces its optimistic preview with that exact accepted result. A terminal exact delta settles without issuing a command-triggered full query refetch.

The receipt is scoped by the named service and verified principal partition. Replaying the same command ID with the same input returns the committed result; reusing it with different input is a conflict. Because the ledger is in cell SQLite, that behavior survives worker/celld restarts rather than depending on process memory.

Why this fixes the latency and race failures

The original failure was not localhost network latency. Idle Eventual consumers were completing, causing service/projector graphs to be rebuilt, while the client also treated accepted commands as a reason to rerun broad queries. A late collection response could then temporarily overwrite a newer optimistic transition on another aggregate.

This PR changes that lifecycle:

• SQL and NATS consumer loops remain alive across idle polls.
• Every Todo transition uses the same shard rule, so create/complete/reopen/archive cannot accidentally split one aggregate across hosts or cells.
• Exact terminal command deltas settle locally; refetch remains a conservative recovery path only when the client cannot prove the affected projection or authorization membership.
• Accepted optimistic overlays remain ordered against comparable authoritative results, so a late response cannot resurrect an older Todo state during rapid multi-item complete/reopen flows.
• A newly created optimistic Todo is visibly pending (gray text with disabled actions) until its commit receipt is confirmed.
• Anonymous client-side navigation refreshes authority correctly, and direct Atomic collection membership (the Blob create case) is applied without requiring a page refresh.

The intended UX is immediate local feedback followed by a small authoritative settlement, not a several-second lockout followed by wholesale query replacement.

When to choose each host

Prefer SOA/local when

• one application database is already the desired consistency boundary;
• a workflow needs a transaction spanning several aggregates/tables in that database;
• aggregate contention is modest and the operational simplicity of a conventional service is more valuable than per-key isolation;
• the command is naturally served by a direct/Atomic projection.

Prefer celld when

• commands are naturally serialized by an aggregate key;
• hot keys or lock contention make a single-writer shard useful;
• per-aggregate isolation and independent placement/scaling are valuable;
• durable idempotent replay must live with the aggregate rather than one service process.

Prefer a hybrid when

• only some aggregates are hot;
• a system is being migrated incrementally;
• cell-owned command consistency is useful, while global read models, search, joins, GraphQL, and identity integration still belong in the service tier.

One important boundary: do not model a required atomic invariant as a distributed transaction across multiple cells. Put the operation on one parent shard (for example game:{game_id}) or keep it in an SOA database transaction. Cells are deliberately local consistency units.

What this PR hardens beyond #206

• Keeps long-running SQL and NATS consumers alive across idle polls instead of rebuilding the service and projector graph.
• Routes the full Todo lifecycle through one shard and adds fenced, durable command receipts with replay, conflict, and restart behavior.
• Preserves trusted service/principal identity across cell dispatch.
• Enforces role row policies for CellByKey reads and fails closed on malformed or unsupported policy material.
• Makes exact projection deltas the normal client settlement path, with conservative revalidation retained for ambiguous/recovery cases.
• Covers rapid Todo transitions, pending-create UI, anonymous soft navigation, and Blob Atomic creation without-refresh behavior.
• Runs authenticated Todo and Chat browser lifecycles against the real Azurite + celld + NATS stack in CI.

Verification

• GitHub CI: 23/23 checks passing.
• Rust quality, all-features, per-feature, Postgres, NATS, RabbitMQ, Kafka, and GraphQL identity matrices.
• e2e-celld workspace tests and live Azurite + worker + NATS lifecycle.
• e2e-ui offline suite and Playwright live stack, including rapid Todo complete/reopen, anonymous soft navigation, and Blob create.
• Distributed JS client quality on Node 20 and Node 24.
• wasm32 worker check, actionlint, and diff hygiene checks.

Stack

Stacked directly on #206 (tasks--portable-command-hosts-celld). Merge #206 first, then this PR.

Refs [[incidents/pr-206-e2e-ui-command-latency-1]]

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: aed36aa5-7bbd-4429-9ab9-a80dc5b9ac46

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}