Skip to content

Bound /store/*/subspace ABCI queries - #4009

Merged
amir-deris merged 8 commits into
mainfrom
amir/plt-799-abci-query-exhaustion-dos
Aug 28, 2026
Merged

Bound /store/*/subspace ABCI queries #4009
amir-deris merged 8 commits into
mainfrom
amir/plt-799-abci-query-exhaustion-dos

Conversation

@amir-deris

@amir-deris amir-deris commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes PLT-799. The /store/{name}/subspace ABCI path on storev2 SS nodes iterated an entire KV prefix into memory with no empty-data guard, no result cap, and no concurrency limit.

This PR makes /subspace fail closed on unbounded work:

  • Reject empty prefix in storev2/query.ScanSubspace before any iteration (covers state and commitment paths).
  • Cap during iteration via the shared storev2/query helper: stop once pair count or accumulated key+value bytes exceed configured limits (defaults: 1,000 pairs / 4 MiB). Returns typed store.ErrSubspaceCapExceeded — no silent truncation.
  • Context-aware iteration via IteratorOn so SS MVCC skip loops honor request cancellation and deadlines (not just between yielded pairs).
  • Concurrency semaphore on the SS fast path (subspaceQuerySem, separate from histProofSem): saturated callers get ErrConflict, same as historical proof.
  • OTel metric subspace_query_rejected{reason=semaphore|cap_exceeded} for operator visibility.

Query-only change — no AppHash or chain-upgrade impact.

Operator note: wide /subspace queries that previously returned large result sets may now fail with ErrSubspaceCapExceeded. Narrow prefixes or raise sc-subspace-max-pairs / sc-subspace-max-bytes before upgrading if indexers or tooling depend on wide scans.

New config ([state-commit])

Key Default Notes
sc-subspace-query-max-inflight 2 <=0 resolves to default
sc-subspace-max-pairs 1,000 <=0 resolves to default; no unlimited setting
sc-subspace-max-bytes 4 MiB (4,194,304) <=0 resolves to default; no unlimited setting

Wired through parseSCConfigs, app.toml template, and config characterization goldens.

Test plan

  • go test ./sei-cosmos/storev2/query/... — pair/byte caps, ctx cancel, typed error + ABCI code preservation
  • go test ./sei-cosmos/storev2/commitment/... — empty prefix rejected, narrow prefix succeeds, /key unaffected
  • go test ./sei-cosmos/storev2/rootmulti/... — semaphore saturation, integration paths, empty prefix
  • go test ./app/ -run 'TestDefaultsMatchTheRecordedValues|TestKeyNamesMatchTheRecordedNames|FuzzParseSCConfigs'
  • go test ./sei-db/config/...

Reject empty prefixes, cap pair/byte accumulation during iteration, and limit concurrent SS fast-path scans so unauthenticated callers cannot OOM a node with wide subspace queries.

Co-authored-by: Cursor <cursoragent@cursor.com>
@amir-deris amir-deris self-assigned this Aug 25, 2026
@amir-deris amir-deris changed the title Bound /subspace ABCI queries to prevent memory-exhaustion DoS (PLT-799). Bound /store/*/subspace ABCI queries against memory-exhaustion DoS (PLT-799) Aug 25, 2026
@amir-deris
amir-deris marked this pull request as ready for review August 25, 2026 15:22
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedAug 28, 2026, 11:04 AM

@cursor

cursor Bot commented Aug 25, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes default ABCI query behavior for wide /subspace scans and adds SS-path concurrency limits, which can break indexers or tooling that relied on unbounded prefix reads but does not alter consensus state.

Overview
/store/*/subspace no longer materializes an entire KV prefix unbounded. State-commitment and SS query paths now share storev2/query.ScanSubspace, which rejects empty prefixes, stops before exceeding configurable pair and byte limits (defaults 1,000 pairs / 4 MiB), and uses context-aware iteration so cancellations and SS MVCC aborts fail closed instead of returning truncated success. Over-cap scans return subspace result exceeds limit (ErrSubspaceCapExceeded); marshal failures become query errors instead of panics.

On the SS fast path, a separate subspaceQuerySem (default max 2 concurrent scans, configurable via sc-subspace-query-max-inflight) rejects saturated callers with subspace query busy, mirroring historical-proof backpressure. subspace_query_rejected{reason=semaphore|cap_exceeded} OTel counters were added for visibility.

Three new [state-commit] knobs are wired through app.toml, parsing, and config goldens: sc-subspace-query-max-inflight, sc-subspace-max-pairs, and sc-subspace-max-bytes (values <= 0 resolve to defaults; no unlimited mode). Query-only — no AppHash impact; indexers doing wide /subspace scans must narrow prefixes or raise caps before upgrade.

Reviewed by Cursor Bugbot for commit 172be6a. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread sei-cosmos/storev2/query/subspace.go
@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.84211% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.29%. Comparing base (17cac18) to head (172be6a).

Files with missing lines Patch % Lines
sei-cosmos/storev2/query/subspace.go 80.95% 4 Missing and 4 partials ⚠️
sei-cosmos/storev2/rootmulti/store.go 85.10% 5 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4009      +/-   ##
==========================================
- Coverage   61.26%   60.29%   -0.97%     
==========================================
  Files        2153     2055      -98     
  Lines      188485   176850   -11635     
==========================================
- Hits       115477   106640    -8837     
+ Misses      62265    60430    -1835     
+ Partials    10743     9780     -963     
Flag Coverage Δ
sei-chain-pr 58.65% <86.84%> (?)
sei-db 69.80% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
app/config_register.go 100.00% <100.00%> (ø)
app/seidb.go 91.48% <100.00%> (+0.58%) ⬆️
sei-cosmos/storev2/commitment/store.go 65.21% <100.00%> (+13.17%) ⬆️
sei-cosmos/storev2/rootmulti/metrics.go 50.00% <ø> (ø)
sei-cosmos/storev2/state/store.go 41.66% <100.00%> (+11.28%) ⬆️
sei-db/config/sc_config.go 100.00% <100.00%> (ø)
sei-cosmos/storev2/rootmulti/store.go 70.45% <85.10%> (+0.43%) ⬆️
sei-cosmos/storev2/query/subspace.go 80.95% <80.95%> (ø)

... and 99 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid, well-tested DoS hardening of the storev2 /subspace ABCI path: pair/byte caps, an empty-prefix rejection, an SS-path semaphore, and complete config-characterization wiring (flags, defaults, app.toml template, both goldens, and the fuzz seed re-indexing all line up). Three non-blocking issues: the scan ignores the repo's context-aware iterator helper, the empty-prefix guard is duplicated at callers instead of the shared choke point, and the 0-means-default cap semantics are undocumented and contradict the neighbouring rate-limit key.

Findings: 0 blocking | 5 non-blocking | 3 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • [suggestion] The new in-flight semaphore only covers the SS fast path. Since RequireProof("/subspace") is false, /subspace reaches the commitment path whenever rs.ssStore == nil (SS disabled), and that path gets no concurrency limit — only the pair/byte caps. Either extend tryAcquireSubspaceQueryPermit to cover it, or state the SS-only scope in a comment on subspaceQuerySem so the gap reads as deliberate.
  • [suggestion] This is a fail-closed behavior change on a public query surface: a /store/{name}/subspace query that previously returned 5k pairs now returns ErrSubspaceCapExceeded instead of data. Worth an explicit release note so operators running indexers or tooling on wide prefixes know to narrow their prefixes or raise sc-subspace-max-pairs / sc-subspace-max-bytes before upgrading.
  • 3 suggestion(s)/nit(s) flagged inline on specific lines.

Comment thread sei-cosmos/storev2/query/subspace.go Outdated
Comment thread sei-cosmos/storev2/query/subspace.go
Comment thread sei-db/config/toml.go
amir-deris and others added 2 commits August 26, 2026 14:48
Thread request context through IteratorOn on the SS path, centralize the
empty-prefix guard in ScanSubspace, and document <=0 default semantics
for subspace limits in app.toml.

Co-authored-by: Cursor <cursoragent@cursor.com>
@amir-deris amir-deris changed the title Bound /store/*/subspace ABCI queries against memory-exhaustion DoS (PLT-799) Bound /store/*/subspace ABCI queries Aug 26, 2026
Comment thread sei-cosmos/storev2/query/subspace.go
amir-deris and others added 3 commits August 26, 2026 17:51
Cancelled or timed-out /subspace queries were panicking on the SS path
because state.Store fatally handles iterator-construction failures.

Co-authored-by: Cursor <cursoragent@cursor.com>
@amir-deris
amir-deris enabled auto-merge August 26, 2026 16:46

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 260ccf0. Configure here.

Comment thread sei-cosmos/storev2/query/subspace.go
amir-deris and others added 2 commits August 28, 2026 12:55
An SS iterator that stops mid-prefix must surface its error instead of marshaling a truncated result as success. Register the three subspace keys in config characterization and document that the in-flight semaphore applies only on the SS fast path.

Co-authored-by: Cursor <cursoragent@cursor.com>
@amir-deris

Copy link
Copy Markdown
Contributor Author

@seidroid review

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well-scoped DoS hardening of the storev2 /subspace ABCI path: the empty-prefix guard now lives in the single ScanSubspace choke point, iteration is context-aware via IteratorOn, aborted scans are no longer marshaled as successful truncated results, and the config wiring (flags, defaults, template, both goldens, fuzz seed re-indexing, key-count comment) is complete and internally consistent. Two non-blocking points: the pair/byte defaults are now defined twice with nothing tying them together, and no test proves an operator-configured non-default cap actually reaches the SS query path.

Findings: 0 blocking | 2 non-blocking | 2 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • None at the file/PR level.
  • 2 suggestion(s)/nit(s) flagged inline on specific lines.

)

const (
DefaultMaxSubspacePairs = 1_000

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] DefaultMaxSubspacePairs/DefaultMaxSubspaceBytes duplicate DefaultSCSubspaceMaxPairs/DefaultSCSubspaceMaxBytes in sei-db/config/sc_config.go, and nothing ties the two pairs together.

That matters because effective() is the runtime authority for a <= 0 value, not the config package. The app.toml comment this PR adds ("<=0 resolves to default 1000") and the CHANGELOG upgrade note both promise a specific number; if someone later retunes DefaultSCSubspaceMaxPairs, toml_test.go will be updated to match the new template output while a node with sc-subspace-max-pairs = 0 silently keeps resolving to the stale value here — and no test compares the two.

sei-db/config has no dependency on sei-cosmos, so sei-cosmos/storev2/query can import it (sei-cosmos/storev2/state already imports sei-db/db_engine/types) and define these as = config.DefaultSCSubspaceMaxPairs / = config.DefaultSCSubspaceMaxBytes, leaving one source for the documented number.

histProofSem: make(chan struct{}, maxInFlight),
histProofLimiter: limiter,
subspaceQuerySem: make(chan struct{}, subspaceMaxInFlight),
subspaceLimits: query.Limits{

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] This is the only place SubspaceMaxPairs/SubspaceMaxBytes cross from scConfig into the query path, and no test covers it: the rootmulti tests exercise the semaphore, the empty prefix, and a narrow-prefix success, but none sets a non-default cap and observes it take effect.

Because Limits.effective() substitutes the package defaults for any non-positive field, a mis-wiring here (dropping the fields, passing query.Limits{}, or swapping pairs/bytes in a future edit) leaves every existing test green — the scans just fall back to 1000 / 4 MiB. A rootmulti-level test in the shape of TestQuery_SubspaceNarrowPrefixAndKeyUnaffected with scCfg.SubspaceMaxPairs = 1, two seeded keys, and an assertion on query.IsCapExceededResponse(resp) would pin the wiring and cover the reason=cap_exceeded metric branch at the same time.

@amir-deris
amir-deris added this pull request to the merge queue Aug 28, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 28, 2026
@amir-deris
amir-deris added this pull request to the merge queue Aug 28, 2026
Merged via the queue into main with commit 5ef8bb2 Aug 28, 2026
85 checks passed
@amir-deris
amir-deris deleted the amir/plt-799-abci-query-exhaustion-dos branch August 28, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants