[fix](fe) Avoid blocking external meta cache refresh on slow miss load#64705
Merged
yiguolei merged 10 commits intoJun 24, 2026
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29015 ms |
Contributor
TPC-DS: Total hot run time: 172348 ms |
Contributor
ClickBench: Total hot run time: 25.14 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
reviewed
Jun 23, 2026
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28812 ms |
Contributor
TPC-DS: Total hot run time: 172896 ms |
Contributor
ClickBench: Total hot run time: 25.34 s |
yiguolei
previously approved these changes
Jun 23, 2026
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29494 ms |
Contributor
TPC-DS: Total hot run time: 172718 ms |
Contributor
ClickBench: Total hot run time: 25.21 s |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
Bot
removed
the
approved
Contributor
Author
|
run buildall |
yiguolei
approved these changes
Jun 24, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
TPC-H: Total hot run time: 29076 ms |
Contributor
TPC-DS: Total hot run time: 172802 ms |
Contributor
ClickBench: Total hot run time: 25.26 s |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29239 ms |
Contributor
TPC-DS: Total hot run time: 172951 ms |
Contributor
ClickBench: Total hot run time: 25.41 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Requesting changes for one concurrency issue in the manual miss-load invalidation path. The PR mostly keeps the external meta-cache change focused and adds useful FE/unit and external regression coverage, but the accepted finding means a value loaded before REFRESH CATALOG/entry invalidation can still become visible to a concurrent reader after the invalidation returns.
Critical checkpoint conclusions:
- Goal/test coverage: The PR aims to avoid blocking external meta-cache refresh on slow miss loads and adds focused FE unit tests plus an external JDBC regression case; the tests do not cover the remaining visible
put-before-remove race. - Scope/focus: The implementation is localized to
MetaCacheEntry, one FE config, and tests. - Concurrency/lifecycle: The main remaining issue is concurrency between in-flight manual loads, invalidation, and the fast
getIfPresenthit path. - Configuration: The new config is mutable and read dynamically by existing entries; no additional config propagation issue found.
- Compatibility/persistence: No wire/storage/EditLog compatibility issue found.
- Parallel paths: Because
MetaCacheEntryis shared by external schema/partition/file/table caches, the accepted race applies to enabled entries across engines, not only JDBC. - Observability/performance: No additional blocking observability or performance issue found beyond the accepted correctness race.
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 24, 2026
#64705) ### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: This PR avoids blocking external meta cache invalidation on slow miss loads in FE. Previously, `MetaCacheEntry` relied on Caffeine's synchronous loading path for cache misses. When an external metadata loader became slow, operations that invalidate the same cache, such as `REFRESH CATALOG` and the corresponding replay path, could wait on the slow load and block the replay-related invalidation flow. Implementation summary: - Keep the existing `LoadingCache` to preserve current hit-path behavior and `refreshAfterWrite` support. - Add a manual miss-load path behind a new FE config switch, using `getIfPresent()` instead of synchronous `LoadingCache.get()` for misses. - Deduplicate concurrent miss loads with striped locks inside `MetaCacheEntry`. - Add an entry-level `invalidateGeneration` counter. Each invalidate increments the generation before clearing cache state. - Record the generation before a manual miss load, check it once before `put()`, and check it again after `put()`. If invalidation happens during the race window, the just-loaded value is removed so stale data is not kept in cache. - Keep null miss-load results uncached so the manual path does not attempt to put null into Caffeine. Configuration: - Add FE config `enable_external_meta_cache_manual_miss_load`, default `false`. - When it is `false`, `MetaCacheEntry` keeps the original synchronous Caffeine miss-load behavior. - When it is `true`, `MetaCacheEntry` uses the manual miss-load path plus `invalidateGeneration` protection. Scope and limitations: - This change applies to `MetaCacheEntry` used by external metadata cache paths in FE. It does not cover the legacy `MetaCache`. - `LegacyMetaCacheFactory` is intentionally not refactored in this PR. A follow-up PR will rework that path with `MetaCache`, and the legacy factory changes are left to that dedicated refactor. - The protection is designed for manual miss loads. It does not make Caffeine's asynchronous `refreshAfterWrite` reload generation-aware. - As a result, `refreshAfterWrite` is still preserved, but an async refresh result may still write back after an invalidate. That is an intentional trade-off in this version. - The new regression case is valuable as a reference and for suitable environments, but it may be skipped in standard CI because it depends on JDBC regression setup, FE debug points, and an external MySQL/JDBC environment. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Manual test: 1. Reproduced the blocking path with `REFRESH CATALOG` against a JDBC external catalog and a debug point that sleeps in `PluginDrivenExternalTable.initSchema`. 2. Repeated the baseline scenario 5 times with `enable_external_meta_cache_manual_miss_load=false` and observed `REFRESH CATALOG` blocked for about 14s while `DESC` stayed slow. 3. Repeated the optimized scenario 5 times with `enable_external_meta_cache_manual_miss_load=true` and observed `REFRESH CATALOG` return within about 1s while `DESC` remained slow. 4. Added a regression case as a manual-test reference because its execution depends on JDBC regression environment and FE debug-point availability. Unit test: - `FE_UT_PARALLEL=1 ./run-fe-ut.sh --run org.apache.doris.datasource.metacache.MetaCacheEntryTest` - Behavior changed: - [x] Yes. Behavior change: - `REFRESH CATALOG` and the corresponding FE invalidation path are no longer blocked by slow external metadata miss loads in this `MetaCacheEntry` implementation. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
Closed
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
This PR avoids blocking external meta cache invalidation on slow miss loads in FE. Previously,
MetaCacheEntryrelied on Caffeine's synchronous loading path for cache misses. When an external metadata loader became slow, operations that invalidate the same cache, such asREFRESH CATALOGand the corresponding replay path, could wait on the slow load and block the replay-related invalidation flow.Implementation summary:
LoadingCacheto preserve current hit-path behavior andrefreshAfterWritesupport.getIfPresent()instead of synchronousLoadingCache.get()for misses.MetaCacheEntry.invalidateGenerationcounter. Each invalidate increments the generation before clearing cache state.put(), and check it again afterput(). If invalidation happens during the race window, the just-loaded value is removed so stale data is not kept in cache.Configuration:
enable_external_meta_cache_manual_miss_load, defaultfalse.false,MetaCacheEntrykeeps the original synchronous Caffeine miss-load behavior.true,MetaCacheEntryuses the manual miss-load path plusinvalidateGenerationprotection.Scope and limitations:
MetaCacheEntryused by external metadata cache paths in FE. It does not cover the legacyMetaCache.LegacyMetaCacheFactoryis intentionally not refactored in this PR. A follow-up PR will rework that path withMetaCache, and the legacy factory changes are left to that dedicated refactor.refreshAfterWritereload generation-aware.refreshAfterWriteis still preserved, but an async refresh result may still write back after an invalidate. That is an intentional trade-off in this version.Release note
None
Check List (For Author)
Manual test:
REFRESH CATALOGagainst a JDBC external catalog and a debug point that sleeps inPluginDrivenExternalTable.initSchema.enable_external_meta_cache_manual_miss_load=falseand observedREFRESH CATALOGblocked for about 14s whileDESCstayed slow.enable_external_meta_cache_manual_miss_load=trueand observedREFRESH CATALOGreturn within about 1s whileDESCremained slow.Unit test:
FE_UT_PARALLEL=1 ./run-fe-ut.sh --run org.apache.doris.datasource.metacache.MetaCacheEntryTestBehavior changed:
Behavior change:
REFRESH CATALOGand the corresponding FE invalidation path are no longer blocked by slow external metadata miss loads in thisMetaCacheEntryimplementation.Does this need documentation?
Check List (For Reviewer who merge this PR)