Skip to content

TEZ-4725: Fix flaky tests in TestAMRecoveryAggregationBroadcast - #520

Merged
abstractdog merged 4 commits into
apache:masterfrom
maheshrajus:TEZ-4725_flaky
Jul 29, 2026
Merged

TEZ-4725: Fix flaky tests in TestAMRecoveryAggregationBroadcast#520
abstractdog merged 4 commits into
apache:masterfrom
maheshrajus:TEZ-4725_flaky

Conversation

@maheshrajus

Copy link
Copy Markdown
Contributor

Root causes and fixes:

  1. TestAMRecoveryAggregationBroadcast.testMapJoinTemporalFailure (race condition)
    Replace fixed Thread.sleep(10s) before AM kill with a deterministic
    waitForVertexSucceeded() helper that polls DAGClient.getVertexStatus()
    every 500ms (up to 60s) until the target vertices reach SUCCEEDED state.
    Each test now waits for only the vertices it logically depends on before
    killing the AM, ensuring the recovery log assertions always see the
    expected counts.
    Make OUT_PATH unique per test run (random suffix) to eliminate cross-test.

  2. DAGClientRPCImpl / TezClientUtils: port out of range:-1 (YARN-808 gap)
    YARN sets rpcPort=-1 when an AM container is allocated (state=RUNNING)
    but the AM has not yet bound its RPC listener. The existing guard only
    checked rpcPort==0 (protobuf default), so rpcPort==-1 reached
    NetUtils.createSocketAddrForHost(), which threw
    IllegalArgumentException: port out of range:-1.
    Fix DAGClientRPCImpl.createAMProxyIfNeeded(): rpcPort == 0 → rpcPort <= 0.
    Fix TezClientUtils.getAMProxy(FrameworkClient,...): add the same
    rpcPort <= 0 guard

@tez-yetus

Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 4m 11s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 35s Maven dependency ordering for branch
+1 💚 mvninstall 4m 33s master passed
+1 💚 compile 3m 59s master passed
+1 💚 checkstyle 1m 4s master passed
+1 💚 javadoc 1m 11s master passed
+0 🆗 spotbugs 1m 23s tez-api in master has 92 extant spotbugs warnings.
+0 🆗 spotbugs 0m 53s tez-tests in master has 6 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 9s Maven dependency ordering for patch
+1 💚 mvninstall 3m 51s the patch passed
+1 💚 codespell 1m 37s No new issues.
+1 💚 compile 4m 5s the patch passed
+1 💚 javac 4m 5s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 30s /results-checkstyle-tez-api.txt tez-api: The patch generated 2 new + 62 unchanged - 0 fixed = 64 total (was 62)
+1 💚 javadoc 1m 10s the patch passed
+1 💚 spotbugs 2m 33s the patch passed
_ Other Tests _
+1 💚 unit 75m 58s root in the patch passed.
+1 💚 asflicense 1m 0s The patch does not generate ASF License warnings.
111m 9s
Subsystem Report/Notes
Docker ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/1/artifact/out/Dockerfile
Optional Tests dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname Linux c15aef73b5a9 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality tez-personality.sh
git revision master / 4bccf3a
Default Java Eclipse Adoptium-21.0.11+10-LTS
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/1/testReport/
Max. process+thread count 1525 (vs. ulimit of 5500)
modules C: tez-api tez-tests U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/1/console
versions git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Comment thread tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java Outdated
Comment thread tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java Outdated
Comment thread tez-api/src/main/java/org/apache/tez/dag/api/client/rpc/DAGClientRPCImpl.java Outdated
Comment thread tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java Outdated
@maheshrajus

Copy link
Copy Markdown
Contributor Author

@abstractdog I have addressed your review comments. Could you please review and approve the PR at your convenience? Thanks !

@tez-yetus

Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 3m 59s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 33s Maven dependency ordering for branch
+1 💚 mvninstall 4m 32s master passed
+1 💚 compile 4m 3s master passed
+1 💚 checkstyle 1m 3s master passed
+1 💚 javadoc 1m 12s master passed
+0 🆗 spotbugs 1m 23s tez-api in master has 92 extant spotbugs warnings.
+0 🆗 spotbugs 0m 51s tez-tests in master has 6 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 9s Maven dependency ordering for patch
+1 💚 mvninstall 3m 40s the patch passed
+1 💚 codespell 1m 36s No new issues.
+1 💚 compile 4m 2s the patch passed
+1 💚 javac 4m 2s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 31s /results-checkstyle-tez-api.txt tez-api: The patch generated 2 new + 62 unchanged - 0 fixed = 64 total (was 62)
+1 💚 javadoc 1m 10s the patch passed
+1 💚 spotbugs 2m 37s the patch passed
_ Other Tests _
+1 💚 unit 75m 54s root in the patch passed.
+1 💚 asflicense 0m 56s The patch does not generate ASF License warnings.
110m 40s
Subsystem Report/Notes
Docker ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/2/artifact/out/Dockerfile
Optional Tests dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname Linux d02fa7af64e9 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality tez-personality.sh
git revision master / efab6a1
Default Java Eclipse Adoptium-21.0.11+10-LTS
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/2/testReport/
Max. process+thread count 1479 (vs. ulimit of 5500)
modules C: tez-api tez-tests U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/2/console
versions git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

@abstractdog abstractdog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @maheshrajus , only a few minor comments left

Comment thread tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java
@maheshrajus
maheshrajus requested a review from abstractdog July 29, 2026 12:35

@abstractdog abstractdog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 more thing, otherwise it's okay

@maheshrajus
maheshrajus requested a review from abstractdog July 29, 2026 13:33
@abstractdog
abstractdog merged commit d348bf1 into apache:master Jul 29, 2026
6 checks passed
@tez-yetus

Copy link
Copy Markdown

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 12s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 34s Maven dependency ordering for branch
+1 💚 mvninstall 4m 31s master passed
+1 💚 compile 4m 5s master passed
+1 💚 checkstyle 1m 2s master passed
+1 💚 javadoc 1m 13s master passed
+0 🆗 spotbugs 1m 23s tez-api in master has 92 extant spotbugs warnings.
+0 🆗 spotbugs 0m 53s tez-tests in master has 6 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 10s Maven dependency ordering for patch
+1 💚 mvninstall 4m 0s the patch passed
+1 💚 codespell 1m 36s No new issues.
+1 💚 compile 4m 3s the patch passed
+1 💚 javac 4m 3s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 30s /results-checkstyle-tez-api.txt tez-api: The patch generated 2 new + 62 unchanged - 0 fixed = 64 total (was 62)
+1 💚 javadoc 1m 10s the patch passed
+1 💚 spotbugs 2m 38s the patch passed
_ Other Tests _
+1 💚 unit 78m 8s root in the patch passed.
+1 💚 asflicense 0m 57s The patch does not generate ASF License warnings.
109m 33s
Subsystem Report/Notes
Docker ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/4/artifact/out/Dockerfile
Optional Tests dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname Linux dc30764bbeb3 5.15.0-181-generic #191-Ubuntu SMP Fri May 22 19:09:02 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality tez-personality.sh
git revision master / 84e634a
Default Java Eclipse Adoptium-21.0.11+10-LTS
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/4/testReport/
Max. process+thread count 1287 (vs. ulimit of 5500)
modules C: tez-api tez-tests U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-520/4/console
versions git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants