fix(cubestore): expose CUBESTORE_CACHESTORE_WAL_TTL_SECONDS to bound the cachestore WAL#11166
Open
killzoner wants to merge 1 commit into
Open
fix(cubestore): expose CUBESTORE_CACHESTORE_WAL_TTL_SECONDS to bound the cachestore WAL#11166killzoner wants to merge 1 commit into
killzoner wants to merge 1 commit into
Conversation
github-actions
Bot
added
cube store
killzoner
force-pushed
the
fix/cachestore-wal-ttl-seconds
branch
from
0c5fa17 to
96189e6
Compare
killzoner
force-pushed
the
fix/cachestore-wal-ttl-seconds
branch
from
96189e6 to
8b1e5b3
Compare
killzoner
force-pushed
the
fix/cachestore-wal-ttl-seconds
branch
from
8b1e5b3 to
d0ea25f
Compare
killzoner
force-pushed
the
fix/cachestore-wal-ttl-seconds
branch
from
d0ea25f to
0410c00
Compare
…the cachestore WAL The cachestore opens RocksDB with a hardcoded WAL ttl of meta_store_snapshot_interval + meta_store_log_upload_interval (~330s), retained so get_updates_since can read recent batches for log upload. Under sustained high-cardinality write churn the archived WAL is kept for that whole window and grows large; on a Kubernetes emptyDir this can fill the volume and the kubelet evicts the pod. Expose the ttl as CUBESTORE_CACHESTORE_WAL_TTL_SECONDS. When unset it falls back to the existing default (the snapshot + log-upload sum), so behavior is unchanged; deployments that do not use cachestore log upload can lower it to bound the archived WAL -- RocksDB purges archived WAL older than the ttl on a wal_ttl/2 cycle, so a lower value gives a lower on-disk plateau. Floored at 2s (0 would disable WAL-ttl purge).
killzoner
force-pushed
the
fix/cachestore-wal-ttl-seconds
branch
from
0410c00 to
e489bf8
Compare
Author
|
From what I've seen in the commits, probably @ovr would be best to have a look when you have time. Thanks in advance |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Check List
cargo test -p cubestore cachestore+ the new config tests pass)cargo fmt --checkclean;cargo clippyshows no new issues in the changed files)CUBESTORE_CACHESTORE_WAL_TTL_SECONDSto the environment variables reference)Issue Reference this PR resolves
Fixes #11168
Description of Changes Made
The cachestore's RocksDB keeps old write-ahead-log (WAL) files for a retention time that is currently hardcoded (~330s).
emptyDir), which gets the pod evicted.CUBESTORE_CACHESTORE_WAL_TTL_SECONDSto make that retention configurable:0would turn WAL cleanup off entirely).WAL_size_limit_MBwas deliberately not used: enabling it alongside the ttl makes RocksDB switch from the fast cleanup cycle to a 10-minute one, so it keeps more WAL, not less.