Cap external scale IT at 100k and fix external-cluster test flakiness#30202
Open
mohityadav766 wants to merge 1 commit into
Open
Cap external scale IT at 100k and fix external-cluster test flakiness#30202mohityadav766 wants to merge 1 commit into
mohityadav766 wants to merge 1 commit into
Conversation
The nightly "Java Integration Tests (External Cluster)" workflow failed on every run. Root causes, all distinct: - scale-500k was cancelled every night — it runs after 100k (max-parallel: 1), took >5h, and never completed. Default the scale matrix fallback to [100000]; larger cohorts remain available on demand via the scaleSeeds dispatch input. - scale-100k died mid-seed with "AuthenticationException (401): Expired token". ExternalTokenRefresher renewed 10m before expiry, but a ~10m login-endpoint outage landing on that window consumed the whole buffer and the ~1h token expired. Renew at half the token's remaining life (~30m) so a login blip has ~half the TTL of retry runway instead of 10m. - SearchRelevancyPreviewIT.matchType/searchFields asserted exact result sets (hasSize/containsExactly) on fuzzy name.ngram queries. On the shared external cluster a foreign ngram-colliding table inflates the set, flaking a different test each run. Assert the seeded ids are present / the excluded id is absent (contains + doesNotContain), matching the existing maxResultHits fix. Verified 7/7 green against the external cluster. - DataQualityPaginationReindexUIIT timed out opening the page-size dropdown: the unscoped .ant-dropdown-menu first-matched a hidden nav-bar menu portaled to <body> (NextPrevious is a shared component). Scope to .ant-dropdown:not(.ant-dropdown-hidden), mirroring the .ant-select-dropdown idiom already used in the page object. NoDuplicatesDuringReindexIT (alias briefly empty during recreate) is not addressed here: current DefaultRecreateHandler is zero-downtime (staged index + atomic swapAliases), so the read alias is never empty. The 0-count is the test correctly catching a read-gap in the older build deployed on the cluster; silencing it would drop a valid regression guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
Code Review ✅ ApprovedStabilizes the external integration test suite by capping scale jobs at 100k, implementing proactive token renewal, and hardening test assertions and UI selectors. No issues found. OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
github-actions
Bot
added
backend
safe to test
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if (exp != null) { | ||
| final long now = System.currentTimeMillis() / 1000; | ||
| delaySeconds = Math.max(60, exp - now - EXPIRY_BUFFER.toSeconds()); | ||
| delaySeconds = Math.max(60, (exp - now) / 2); |
Contributor
There was a problem hiding this comment.
When the configured token is already partway through its lifetime, this calculation can reduce the retry runway. A token with 10 minutes remaining previously refreshed after the 60-second floor, leaving about 9 minutes for retries; it now waits 5 minutes, so a five-minute login outage can let the token expire and cause external-cluster requests to return 401.
Context Used: CLAUDE.md (source)
Contributor
🔴 Playwright Results — 1 failure(s), 27 flaky✅ 4543 passed · ❌ 1 failed · 🟡 27 flaky · ⏭️ 95 skipped
Genuine Failures (failed on all attempts)❌
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes:
The nightly External Cluster workflow failed every run, for four unrelated reasons. This fixes each:
max-parallel: 1), took >5h, and never completed. The scale matrix fallback now defaults to[100000]; larger cohorts stay available on demand via thescaleSeedsdispatch input.401 Expired tokenmid-seed —ExternalTokenRefresherrenewed only 10m before expiry, so a ~10m login-endpoint outage on the renewal window let the ~1h token die. It now renews at half the token's remaining life (~30m runway). Validated live against the cluster (refresh scheduled in 1795s).SearchRelevancyPreviewITflaky (matchType/searchFields) — exact-set assertions (hasSize/containsExactly) on fuzzyname.ngramqueries were inflated by foreign ngram-colliding tables on the shared cluster. Switched tocontains/doesNotContain(matching the existingmaxResultHitsfix). Verified 7/7 green againstmohitcorp2.getcollate.io.DataQualityPaginationReindexUIITdropdown timeout — the unscoped.ant-dropdown-menufirst-matched a hidden nav-bar menu portaled to<body>(NextPreviousis a shared component). Scoped to.ant-dropdown:not(.ant-dropdown-hidden), mirroring the existing.ant-select-dropdownidiom.Not changed —
NoDuplicatesDuringReindexIT: currentDefaultRecreateHandleris zero-downtime (staged index + atomicswapAliases), so the read alias is never empty. Thecount=0is the test correctly catching a read-gap in the older build deployed on the cluster (revf1f786a8, not in this repo). Silencing it would remove a valid regression guard — the cluster needs redeploying instead.Type of change:
High-level design:
N/A — CI config + test/selector robustness only; no production code changed.
Tests:
Manual testing performed
SearchRelevancyPreviewIT(-P search-it -Dit.test=...) againsthttps://mohitcorp2.getcollate.io— reproduced the flake, applied the fix, re-ran 7/7 green.NextPrevious.tsxAnt<Dropdown>markup (Playwright not runnable in this environment).Backend integration tests
Playwright (UI) tests
UI screen recording / screenshots:
Not applicable.
Checklist:
Greptile Summary
This PR stabilizes the nightly external-cluster integration tests. The main changes are:
Confidence Score: 4/5
The aged-token refresh path needs a fix before merging.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/util/ExternalTokenRefresher.java
Important Files Changed
Reviews (1): Last reviewed commit: "Cap external scale IT at 100k and fix ex..." | Re-trigger Greptile
Context used (3)