feat: add pallet-block-forwarder for offchain HTTP indexer forwarding#185
Open
rishitesh-snt wants to merge 13 commits intomainfrom
Open
feat: add pallet-block-forwarder for offchain HTTP indexer forwarding#185rishitesh-snt wants to merge 13 commits intomainfrom
rishitesh-snt wants to merge 13 commits intomainfrom
Conversation
Offchain worker that forwards table lifecycle and data events (SchemaUpdated, TablesCreatedWithCommitments, TableDropped, QuorumReached) to an external HTTP indexer service via protobuf-over-HTTP. Architecture: - Producer (on_finalize): extracts events during block execution, writes block-indexed entries to the offchain DB. Runs during both live blocks and historical sync, enabling full backfill for new nodes. - Consumer (offchain_worker): drains the offchain DB queue in deposit order, POSTs to the HTTP server, checkpoints, deletes consumed entries. - Server checkpoint is the sole source of truth (no local cursor). - No new host functions; raw DDL and postcard bytes shipped as-is. - Processes up to 100 blocks per OCW invocation for catch-up. - Dynamic indexing pallet detection: IndexingPallet (PalletInfoAccess) resolves the pallet index from construct_runtime!; QuorumReachedVariantIndex is a Get<u8> the runtime provides (currently ConstU8<1>). Includes the proto definition, an HTTP+protobuf client, the producer/consumer pipeline, an axum-based mock server for local testing, helper scripts, and seven unit/integration tests covering the full pipeline.
1.69.0Bug Fixes
Features
Performance Improvements
|
The pallet ships QuorumReached.data bytes (which pallet_indexing validates as Arrow IPC single-batch stream via record_batch_bytes_dimensions) through the HTTP adapter verbatim. Block-forwarder never decodes them. Updates the stale doc comment in lib.rs and the mock-server log field from 'postcard OnChainTable' to 'Arrow IPC stream'. No runtime behavior change.
Replaces the hard-coded `QuorumReachedVariantIndex = ConstU8<1>` wiring with a `DynamicQuorumReachedIndex` struct that looks up the variant's index on `pallet_indexing::Event` by name via `scale_info::TypeInfo` metadata that FRAME already derives on every pallet event. Motivation: the hard-coded `1` was a silent-drift hazard. If anyone reorders `pallet_indexing::Event` (e.g., prepends a new variant), the old index becomes wrong and the block-forwarder silently stops matching QuorumReached events — data quietly stops flowing to indexers. Looking it up by name survives reorderings and fails loudly at node boot with a clear message if the variant is ever renamed or removed. The lookup lives entirely in the runtime crate. The pallet's `Config` contract is unchanged (`QuorumReachedVariantIndex: Get<u8>`); the pallet and its mock stay exactly as they were. No new deps; scale-info is already a direct runtime dep. Adds a `dynamic_quorum_reached_index_resolves` unit test so a future rename of `QuorumReached` fails at `cargo test` time rather than node boot.
Adds an `--indexer-url <URL>` CLI flag to sxt-node. When set, the node writes the URL into the block-forwarder OCW's persistent local storage (key: `block_forwarder::indexer_url`) during startup, as a SCALE-encoded `Vec<u8>` under the standard offchain STORAGE_PREFIX. Equivalent to running `pallets/block_forwarder/scripts/configure-ocw.sh` via the offchain_localStorageSet RPC, but: - no --rpc-methods=unsafe required; - no second process / second terminal; - seeds before the first block is authored, so no events are missed. If omitted, behaviour is unchanged: OCW is a no-op until the URL is written by some other means (RPC, manual offchain_localStorageSet, etc.). Touches node/src/cli.rs (new CLI arg), node/src/service.rs (`configure_indexer_url` helper invoked from new_full_base).
The block-forwarder producer writes events via sp_io::offchain_index::set,
which is a silent no-op unless --enable-offchain-indexing=true is passed.
Until now, forgetting that flag caused forwarding to look healthy on the
producer side while nothing arrived at the indexer — a hard-to-debug
silent-failure mode.
Two changes to turn silent failure into loud failure:
1. node/src/service.rs (new_full_base):
- --indexer-url set + --enable-offchain-indexing=false → hard startup
error. Unambiguous misconfiguration; boot refuses to proceed.
- --indexer-url unset + --enable-offchain-indexing=false → stderr
warning at boot. Forwarding will never work even if the URL is
written via RPC later; worth surfacing even if the operator hasn't
opted into the CLI flag path.
2. pallets/block_forwarder/README.md:
New README explaining the three node flags that must be set for
forwarding to work (--enable-offchain-indexing, --offchain-worker,
--indexer-url), the runtime wiring, the wire data format, the dedup
key contract, and the testing options. One-command dev setup example.
No behavior change for correctly-configured nodes. Tests unchanged:
pallet-block-forwarder 7/7, sxt-runtime 4/4.
tlovell-sxt
reviewed
Apr 21, 2026
| }; | ||
| index.events.push(BlockEvent::Data(DataEntry { | ||
| table: quorum.table, | ||
| data: data.to_vec(), |
Contributor
There was a problem hiding this comment.
BTW I'm sure you've noticed but the data in the quorum reached event isn't a record batch but a native table type we made for no-std compatibility. The source batch for it doesn't have the META_ROW_NUMBER column, so we can't just use that. I think you mentioned this last week actually. We might unfortunately need to add a native interface to convert it back to a record batch like you said last week, for the offchain workers to transform the processed data back to a format that the PRB can read. Or I guess the PRB could be made to read OnChainTables but that seems wrong.
Contributor
Author
There was a problem hiding this comment.
Yes, right now reads the OnChainTable. The interface is not very generic but is optimised for our use case and to keep OCW dumb. If ever PRB evolves to be more than a ingestion component, we can add interfaces separately.
| } | ||
|
|
||
| fn try_extract_indexing_event( | ||
| event: &<T as polkadot_sdk::frame_system::Config>::RuntimeEvent, |
Contributor
There was a problem hiding this comment.
BTW what's the benefit of processing events in this pallet as opposed to just writing to local storage w/ off-chain-indexing as part of the pallet-tables/pallet-indexing extrinsics? It seems like there's a little bit of headache to decoding the events in this separate place. Not saying it's wrong, just wanna consider both options
Contributor
There was a problem hiding this comment.
Still curious what your thoughts are on this.
Contributor
Author
There was a problem hiding this comment.
You're right that it would be less code — but the cost shows up across pallet-indexing and pallet-tables call paths instead, and couples their internals to block_forwarder's storage. I preferred to pay the decode tax in one place than spread writes across three pallets' extrinsics.
Real integration path is now: - prb-service with --features indexer implements the HTTP surface. - --indexer-url flag on sxt-node seeds the OCW storage at startup. - sxt-int-harness (standalone crate) drives chain actions via TOML. Removing the now-redundant local-testing artifacts: - pallets/block_forwarder/mock-server/ (axum stub HTTP server that logged calls; superseded by running the real prb-service). - pallets/block_forwarder/scripts/configure-ocw.sh (SCALE-encoded the URL and called offchain_localStorageSet via RPC; superseded by the --indexer-url CLI flag). - pallets/block_forwarder/scripts/run-local-demo.sh (tmux orchestrator that chained the mock server + node + configure-ocw.sh; no longer has a job). - Workspace-member entry "pallets/block_forwarder/mock-server". README updated to describe the new integration path. Comments in node/src/cli.rs and node/src/service.rs no longer reference configure-ocw.sh. pallet-block-forwarder tests unchanged: 7/7 pass — they use sp_core::offchain::testing::TestOffchainExt, not the mock-server.
`DynamicQuorumReachedIndex::get()` is called once per pallet_indexing
event in block-forwarder's per-event filter (`try_extract_indexing_event`),
which is the hot path during on_finalize. Before this change each call
re-ran `scale_info::TypeInfo::type_info()`, which builds a fresh `Type`
struct with heap-allocated children on every invocation.
Wraps the one-time lookup in `lazy_static!` so subsequent calls are a
single atomic load. Lookup still runs on first access (panics on
rename/remove, as intended) — just doesn't run 10+ times per block.
Adds `lazy_static = { workspace = true }` to runtime's deps. The
workspace already pins it with `spin_no_std` feature, so this works in
both std (native) and no_std (WASM) builds.
No behavior change. sxt-runtime 4/4 tests pass, including
`dynamic_quorum_reached_index_resolves`.
…foAccess The filter had two different mechanisms for resolving a pallet's construct_runtime! index: - pallet_indexing → `type IndexingPallet: PalletInfoAccess;` (introduced earlier this session, FRAME-idiomatic, clean). - pallet_tables → `fn tables_pallet_index()` that fabricated a dummy `TableDropped(None, Community, empty_ident, Source::default())`, routed it through the `From<pallet_tables::Event>` for RuntimeEvent impl, SCALE-encoded the result, and peeked at byte 0. Brittle — every time `TableDropped`'s variant shape changes (it gained a 4th Source field recently), the dummy constructor has to track it. Now both use `PalletInfoAccess`. Adds `TablesPallet` to Config parallel to `IndexingPallet`; runtime wires `type TablesPallet = Tables;` and the mock wires `type TablesPallet = Tables;` as well. The `tables_pallet_index` function is deleted entirely. Separately fixes a typo the pallet picked up during an earlier edit: `TableTzype::Community` → `TableType::Community`. The line is now gone with the dummy constructor, so the typo no longer matters, but the deletion implicitly resolves it. No behavior change. Tests: pallet-block-forwarder 7/7 pass; sxt-runtime checks clean.
…row IPC
Earlier this session I incorrectly updated these docs to say the
forwarded row-data bytes were Arrow IPC. Re-tracing the pallet_indexing
flow shows that's not what QuorumReached.data carries:
indexer → submit_data.data = Arrow IPC bytes
chain:
validate_data: parse Arrow IPC header (weight accounting, check only)
host fn record_batch_to_onchain: Arrow IPC → RecordBatch → OnChainTable
process_insert_and_update_commitments: attach meta columns
postcard::to_allocvec(&insert_with_meta_columns) ← POSTCARD from here
QuorumReached { data: <postcard bytes> }
block-forwarder: opaque relay of postcard bytes to /v1/put_batches
The Arrow IPC format only lives on the indexer-side input; the on-chain
event data and everything downstream is postcard-encoded OnChainTable.
Module header and README data-format section now reflect that.
No code change — this is purely a documentation correction. The
companion fix on the sxtdb side restores the prb-service decoder to
postcard.
- Allow dead_code/missing_docs on the auto-generated pallet module and the prost-generated proto submodule. - Allow enum_variant_names on http_client::Error's IoError variant. - Drop unused TableType import in the OCW tests. - Attach the existing result_large_err expectation to configure_indexer_url so CI's -Dclippy::all doesn't regress on it. - cargo f (imports_granularity=Module, group_imports=StdExternalCrate, imports_layout=HorizontalVertical) across our PR files.
tlovell-sxt
reviewed
Apr 22, 2026
| } | ||
|
|
||
| fn try_extract_indexing_event( | ||
| event: &<T as polkadot_sdk::frame_system::Config>::RuntimeEvent, |
Contributor
There was a problem hiding this comment.
Still curious what your thoughts are on this.
| } | ||
|
|
||
| Ok(response.body().collect()) | ||
| } |
Contributor
There was a problem hiding this comment.
I would appreciate this PR being split into more. Perhaps 1 PR for configuration, 1 for the http client, one for the indexing, one for the OCW flushing the indexed items to the http client. It's still nice to see it all put together here still but yeah we do usually strive for smaller PRs in this repo
Contributor
Author
There was a problem hiding this comment.
Sure let me try that, first I would like to get a generic feedback on the approach.
Address review comments on the offchain HTTP forwarder pallet: - Rename pallet from block-forwarder to prover-db-indexer; the old name was too generic for what is specifically forwarding to a prover-db backend (directory, crate, runtime wiring, storage-key prefixes, log targets, mock/test idents, README all moved). - Rename proto file: indexer.proto -> prover-db.proto. - Rename CLI flag and OCW storage key: --indexer-url -> --prover-db-url (PROVER_DB_URL_KEY const + "prover_db_indexer::prover_db_url" key). - Strengthen --prover-db-url type from String to url::Url so an invalid value is rejected at clap-parse time rather than failing on the OCW's first HTTP request. - Drop debug eprintln on successful storage seed (the Result already propagates errors; success is silent).
Address PR review: instead of SCALE-encoding each runtime event and peeking pallet/variant tag bytes, supertrait the source pallets and let `construct_runtime!`'s generated `TryInto<pallet_X::Event>` conversions do typed downcasts. Variant-rename safety is now compile-time, not a `lazy_static!` panic at startup. - Make pallet instanced (`Pallet<T, I = ()>`, `Config<I>`, `Event<T, I>`) so we can supertrait `pallet_indexing::Config<I>` (which is itself instanced). - Drop `TablesPallet`, `IndexingPallet`, `QuorumReachedVariantIndex` associated types. Drop runtime's `DynamicQuorumReachedIndex`, `lazy_static!` cache, `Get<u8>` impl, and `find_event_variant_index` helper. Drop `lazy_static` dep from runtime. - Add `native_pallet` aliasing module so `construct_runtime!` can refer to `Pallet<Runtime>` instead of carrying the instance type parameter. - Bridge `frame_system::Config::RuntimeEvent` -> our `Config<I>::RuntimeEvent` via explicit `From::from(...)`; same underlying value, distinct types to the type system, joined by `IsType`. - Mock expands from 78 to ~225 lines (mirrors pallet_indexing's mock) — pure trait-impl boilerplate to satisfy the supertrait chain. No test exercised the producer side either before or after this change.
Address PR review: prefer immutability. The two extraction helpers no longer take `&mut BlockIndex` — they return what would be appended (`Vec<BlockEvent>` for tables since one event can yield N creates, `Option<BlockEvent>` for indexing since it's at most one). `extract_block_index` flattens them into a single `collect`, and `BlockIndex` is constructed once from the result rather than mutated in a loop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
sxt-node already emits SchemaUpdated, TablesCreatedWithCommitments, TableDropped, and QuorumReached events. Today there's no first-party path for indexer-paired nodes to replay those events off-chain. Operators
hook that up themselves with ad-hoc scripts. This pallet closes that gap with:
Architecture
keyed by block number. Running during sync means fresh nodes backfill their indexer automatically.
MAX_BLOCKS_PER_INVOCATION = 100 per OCW tick → ~100× realtime catch-up). Forwards each block's events in deposit order, checkpoints the block, deletes the consumed offchain-DB entry.
forwarder relays them verbatim. No new host functions; the runtime does no Arrow decoding.
Runtime wires type IndexingPallet = Indexing. Variant byte for QuorumReached is a Get the runtime provides as ConstU8<1> — constant for now, swap in real decoding only when the enum becomes volatile.
module docs.
producer side.
Test coverage
Runtime integration surface
Workspace-level: pallet-block-forwarder and pallet-block-forwarder/mock-server added as members; pallet-block-forwarder added to runtime/Cargo.toml with the /std feature.