mirror_worker: parse and verify add-entries (C4a)#239
Draft
lukevalenta wants to merge 1 commit into
Draft
Conversation
Implements the parse-and-verify portion of the [c2sp.org/tlog-mirror#add-entries][add-e] endpoint. On a successful request the handler returns 200 with an empty body. C4a does not yet persist entries, recompute tiles, advance the committed checkpoint, or emit a mirror cosignature; those land in C4b and C4c respectively. Slice references are anchored in #186 (comment). What the handler does - Slurps the request body (capped at MAX_ADD_ENTRIES_BODY_SIZE = 10 MiB; a knob, not a spec value, sized to fit comfortable batches). - Parses the AddEntriesRequestHeader. Malformed headers, oversized bodies, or trailing bytes after the last package surface as 400. - Looks up the log by `log_origin`; 404 on unknown. - Snapshots the per-origin DO state via `POST /get-state` (the C2 RPC). - Resolves the *target pending checkpoint*: * `upload_end == snapshot.pending.size`: use the current pending. * Otherwise, opens the request ticket via `TicketMacer::open`, re-parses the plaintext as a signed `Note`, re-verifies against the trusted log keys, and adopts the embedded checkpoint iff its origin and size match. The ticket round-trip is what implements the spec's "store the signed checkpoint in the ticket" recommendation. * Neither matches: 409 with text/x.tlog.mirror-info body carrying the current `(pending.size, committed.size, fresh_ticket_for_current_pending)`. - Validates `upload_start <= committed.size` and (C4a constraint) `upload_start % 256 == 0`. Non-aligned `upload_start` requires reading previously-committed leaves from R2 to verify the first package's subtree, which lands in C4b; C4a surfaces non-aligned starts as 409 so the client can re-fetch and realign. - Iterates each EntryPackage via `package_ranges`. For each: * Parses the wire bytes. * Replays the leaves through `stored_hashes_for_record_hash` to populate an in-memory store covering only the package's range. Because the package is aligned, no pre-package leaves are referenced during replay. * Reconstructs the package's subtree hash via `Subtree::hash`. * Verifies the subtree consistency proof against the target pending checkpoint via `tlog_core::verify_subtree_consistency_proof`. 422 on failure. Implementation notes - `MapReader` is a sparse `HashMap<u64, Hash>`-backed `HashReader` used during proof verification. The store is built incrementally as we replay leaves, and a fresh borrow is taken on each iteration so the borrow checker is satisfied without unsafe. - Uses `tlog_core::stored_hash_index(0, n)` directly for indexing the per-package store; the function is already part of the public `tlog_core` surface. - `mirror_info_409` seals a fresh ticket from the *current* pending signed-note bytes whenever it returns a 409. If `MIRROR_TICKET_KEY` is missing or malformed, we still return 409 but with an empty ticket and log an error so an operator notices. - The handler trusts the streaming-friendly shape of `EntryPackage::read_from`. Workers' `req.bytes().await` doesn't actually stream, but the per-package iteration model means future Workers streaming support won't require a rewrite. Pre-push checks all green: clippy-pedantic, test, fmt, machete, and `cargo check -p mirror_worker --target wasm32-unknown-unknown`. 18 unit tests in `mirror_worker` (unchanged from base; the verification logic is exercised end-to-end in C4b/C4c integration tests rather than via narrow unit tests over Merkle math plumbing). [add-e]: https://c2sp.org/tlog-mirror#add-entries
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the parse-and-verify portion of c2sp.org/tlog-mirror#add-entries. On a successful request the handler returns 200 with an empty body. Persistence, tile recomputation, committed-checkpoint advancement, and mirror cosignature emission land in C4b/C4c.
Towards #186 (tlog-mirror support); see the C1–C5 plan comment for how this fits into the broader mirror work. Stacked on #238; base will retarget to
mainas predecessors merge.What the handler does
MAX_ADD_ENTRIES_BODY_SIZE = 10 MiB. A knob, not a spec value, sized to fit comfortable batches without letting a hostile client push the parser into base64-decoding tens of megabytes of garbage.AddEntriesRequestHeaderviatlog_mirror::wire. Malformed headers, oversized bodies, or trailing bytes after the last package surface as 400.log_originisn't configured.POST /get-stateRPC.upload_end == snapshot.pending.size: use the current pending.TicketMacer::open, re-parse the plaintext as a signedNote, re-verify against the trusted log keys, and adopt the embedded checkpoint iff its origin and size match. This implements the spec's "store the signed checkpoint in the ticket" recommendation.text/x.tlog.mirror-infobody carrying the current(pending.size, committed.size, fresh_ticket_for_current_pending).upload_start:<= committed.sizeand (C4a constraint)% 256 == 0. Non-alignedupload_startrequires reading previously-committed leaves from R2 to verify the first package's subtree, which lands in C4b. C4a surfaces non-aligned starts as 409 so the client can re-fetch and realign.package_ranges. For each:EntryPackage::read_from.tlog_core::stored_hashes_for_record_hashinto a sparseHashMap<u64, Hash>store. Because the package is aligned, no pre-package leaves are referenced during replay.Subtree::hash.tlog_core::verify_subtree_consistency_proof. 422 on failure.Implementation notes
MapReader: a sparseHashMap<u64, Hash>-backedHashReader. The store is built incrementally as we replay leaves, with a fresh borrow per iteration so the borrow checker is satisfied withoutunsafe.tlog_core::stored_hash_index(0, n)directly — already public on the spec crate.mirror_info_409seals a fresh ticket from the current pending signed-note bytes whenever it returns a 409. IfMIRROR_TICKET_KEYis missing or malformed, we still return 409 but with an empty ticket and log an error so an operator notices.EntryPackage::read_from. Workers'req.bytes().awaitdoesn't actually stream, but the per-package iteration model means future Workers streaming support won't require a rewrite.What this PR does NOT do
/commitRPC (C4c).upload_start % 256 == 0constraint (requires C4b's R2 reads).add-entriesend-to-end. C4a's success-is-empty-body shape doesn't observably differ from a stub againstwrangler dev; the meaningful integration test lands with C4c.Stats
crates/mirror_worker/src/add_entries.rs(467 LOC).mirror_worker(unchanged from mirror_worker: load cosigner + ticket keys, /metadata, extended state DO #238; verification logic exercised end-to-end in C4b/C4c integration tests rather than via narrow unit tests over Merkle math plumbing).Verification
All five pre-push checks pass locally on the head commit: