Skip to content

mirror_worker: parse and verify add-entries (C4a)#239

Draft
lukevalenta wants to merge 1 commit into
lvalenta/mirror-worker-keys-statefrom
lvalenta/mirror-worker-add-entries-parse
Draft

mirror_worker: parse and verify add-entries (C4a)#239
lukevalenta wants to merge 1 commit into
lvalenta/mirror-worker-keys-statefrom
lvalenta/mirror-worker-add-entries-parse

Conversation

@lukevalenta
Copy link
Copy Markdown
Contributor

Summary

Implements the parse-and-verify portion of c2sp.org/tlog-mirror#add-entries. On a successful request the handler returns 200 with an empty body. Persistence, tile recomputation, committed-checkpoint advancement, and mirror cosignature emission land in C4b/C4c.

Towards #186 (tlog-mirror support); see the C1–C5 plan comment for how this fits into the broader mirror work. Stacked on #238; base will retarget to main as predecessors merge.

What the handler does

  1. Slurp the request body, capped at MAX_ADD_ENTRIES_BODY_SIZE = 10 MiB. A knob, not a spec value, sized to fit comfortable batches without letting a hostile client push the parser into base64-decoding tens of megabytes of garbage.
  2. Parse the AddEntriesRequestHeader via tlog_mirror::wire. Malformed headers, oversized bodies, or trailing bytes after the last package surface as 400.
  3. Origin lookup: 404 if log_origin isn't configured.
  4. Snapshot per-origin DO state via the C2 POST /get-state RPC.
  5. Resolve target pending checkpoint:
    • upload_end == snapshot.pending.size: use the current pending.
    • Otherwise: open the request ticket via TicketMacer::open, re-parse the plaintext as a signed Note, re-verify against the trusted log keys, and adopt the embedded checkpoint iff its origin and size match. This implements the spec's "store the signed checkpoint in the ticket" recommendation.
    • Neither matches: 409 with text/x.tlog.mirror-info body carrying the current (pending.size, committed.size, fresh_ticket_for_current_pending).
  6. Validate upload_start: <= committed.size and (C4a constraint) % 256 == 0. Non-aligned upload_start requires reading previously-committed leaves from R2 to verify the first package's subtree, which lands in C4b. C4a surfaces non-aligned starts as 409 so the client can re-fetch and realign.
  7. Per-package verification via package_ranges. For each:
    • Parse wire bytes via EntryPackage::read_from.
    • Replay leaves through tlog_core::stored_hashes_for_record_hash into a sparse HashMap<u64, Hash> store. Because the package is aligned, no pre-package leaves are referenced during replay.
    • Reconstruct the package's subtree hash via Subtree::hash.
    • Verify the subtree consistency proof against the target pending checkpoint via tlog_core::verify_subtree_consistency_proof. 422 on failure.

Implementation notes

  • MapReader: a sparse HashMap<u64, Hash>-backed HashReader. The store is built incrementally as we replay leaves, with a fresh borrow per iteration so the borrow checker is satisfied without unsafe.
  • Indexes via tlog_core::stored_hash_index(0, n) directly — already public on the spec crate.
  • mirror_info_409 seals a fresh ticket from the current pending signed-note bytes whenever it returns a 409. If MIRROR_TICKET_KEY is missing or malformed, we still return 409 but with an empty ticket and log an error so an operator notices.
  • The handler trusts the streaming-friendly shape of EntryPackage::read_from. Workers' req.bytes().await doesn't actually stream, but the per-package iteration model means future Workers streaming support won't require a rewrite.

What this PR does NOT do

  • Persist entries to R2 (C4b).
  • Recompute tiles from received entries (C4b).
  • Advance the committed checkpoint via the C2 /commit RPC (C4c).
  • Emit mirror cosignature lines in the response body (C4c).
  • Lift the upload_start % 256 == 0 constraint (requires C4b's R2 reads).
  • Add an integration test for add-entries end-to-end. C4a's success-is-empty-body shape doesn't observably differ from a stub against wrangler dev; the meaningful integration test lands with C4c.

Stats

  • 3 files changed, +475 / −5.
  • 1 new file: crates/mirror_worker/src/add_entries.rs (467 LOC).
  • 1 new route on the frontend, 1 new module declaration.
  • 18 unit tests in mirror_worker (unchanged from mirror_worker: load cosigner + ticket keys, /metadata, extended state DO #238; verification logic exercised end-to-end in C4b/C4c integration tests rather than via narrow unit tests over Merkle math plumbing).

Verification

All five pre-push checks pass locally on the head commit:

cargo clippy --workspace --all-targets -- -Dwarnings -Dclippy::pedantic
cargo test
cargo fmt --all --check
cargo machete
cargo check -p mirror_worker --target wasm32-unknown-unknown

Implements the parse-and-verify portion of the
[c2sp.org/tlog-mirror#add-entries][add-e] endpoint. On a successful
request the handler returns 200 with an empty body. C4a does not yet
persist entries, recompute tiles, advance the committed checkpoint,
or emit a mirror cosignature; those land in C4b and C4c
respectively.

Slice references are anchored in
#186 (comment).

What the handler does

- Slurps the request body (capped at MAX_ADD_ENTRIES_BODY_SIZE = 10
  MiB; a knob, not a spec value, sized to fit comfortable batches).
- Parses the AddEntriesRequestHeader. Malformed headers, oversized
  bodies, or trailing bytes after the last package surface as 400.
- Looks up the log by `log_origin`; 404 on unknown.
- Snapshots the per-origin DO state via `POST /get-state` (the C2
  RPC).
- Resolves the *target pending checkpoint*:
  * `upload_end == snapshot.pending.size`: use the current pending.
  * Otherwise, opens the request ticket via `TicketMacer::open`,
    re-parses the plaintext as a signed `Note`, re-verifies against
    the trusted log keys, and adopts the embedded checkpoint iff
    its origin and size match. The ticket round-trip is what
    implements the spec's "store the signed checkpoint in the
    ticket" recommendation.
  * Neither matches: 409 with text/x.tlog.mirror-info body
    carrying the current `(pending.size, committed.size,
    fresh_ticket_for_current_pending)`.
- Validates `upload_start <= committed.size` and (C4a constraint)
  `upload_start % 256 == 0`. Non-aligned `upload_start` requires
  reading previously-committed leaves from R2 to verify the first
  package's subtree, which lands in C4b; C4a surfaces non-aligned
  starts as 409 so the client can re-fetch and realign.
- Iterates each EntryPackage via `package_ranges`. For each:
  * Parses the wire bytes.
  * Replays the leaves through `stored_hashes_for_record_hash` to
    populate an in-memory store covering only the package's
    range. Because the package is aligned, no pre-package leaves
    are referenced during replay.
  * Reconstructs the package's subtree hash via `Subtree::hash`.
  * Verifies the subtree consistency proof against the target
    pending checkpoint via
    `tlog_core::verify_subtree_consistency_proof`. 422 on failure.

Implementation notes

- `MapReader` is a sparse `HashMap<u64, Hash>`-backed `HashReader`
  used during proof verification. The store is built incrementally
  as we replay leaves, and a fresh borrow is taken on each
  iteration so the borrow checker is satisfied without unsafe.
- Uses `tlog_core::stored_hash_index(0, n)` directly for indexing
  the per-package store; the function is already part of the
  public `tlog_core` surface.
- `mirror_info_409` seals a fresh ticket from the *current* pending
  signed-note bytes whenever it returns a 409. If
  `MIRROR_TICKET_KEY` is missing or malformed, we still return 409
  but with an empty ticket and log an error so an operator
  notices.
- The handler trusts the streaming-friendly shape of
  `EntryPackage::read_from`. Workers' `req.bytes().await`
  doesn't actually stream, but the per-package iteration model
  means future Workers streaming support won't require a
  rewrite.

Pre-push checks all green: clippy-pedantic, test, fmt, machete,
and `cargo check -p mirror_worker --target wasm32-unknown-unknown`.
18 unit tests in `mirror_worker` (unchanged from base; the
verification logic is exercised end-to-end in C4b/C4c integration
tests rather than via narrow unit tests over Merkle math
plumbing).

[add-e]: https://c2sp.org/tlog-mirror#add-entries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant