IGNITE-28738 Implement basic Rolling Upgrade test using testcontainers by wernerdv · Pull Request #13184 · apache/ignite

wernerdv · 2026-05-27T07:17:11Z

Thank you for submitting the pull request to the Apache Ignite.

In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:

The Contribution Checklist

There is a single JIRA ticket related to the pull request.
The web-link to the pull request is attached to the JIRA ticket.
The JIRA ticket has the Patch Available state.
The pull request body describes changes that have been made.
The description explains WHAT and WHY was made instead of HOW.
The pull request title is treated as the final commit message.
The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
A reviewer has been mentioned through the JIRA comments
(see the Maintainers list)
The pull request has been checked by the Teamcity Bot and
the green visa attached to the JIRA ticket (see tab PR Check at TC.Bot - Instance 1 or TC.Bot - Instance 2)

Notes

If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.

This reverts commit 0b6e0ef.

…path, faster shutdown (#1) - testClassesJar: inject each listed class together with its nested classes (the plugin provider's anonymous classes) and add DisabledRollingUpgradeProcessor / DisabledValidationProcessor. Without this the containerized nodes fail to start (NoClassDefFoundError) and the test times out after 600s on a clean checkout. - OS-aware networking. Keep the macOS path (host<->container via published ports + ContainerAddressResolver + host.docker.internal) and add a direct Linux path: containers are reached at their Docker bridge IP, the host node is reached at the Docker network gateway. Removes the Docker-Desktop VM-proxy latency on Linux/CI. - Container shutdown wait 60s -> 10s. Graceful SIGTERM does not complete (the image entrypoint does not forward it), so the node is force-stopped anyway; this saved ~50s per node (~423s -> ~250s end to end on macOS). - SOURCE_COMMIT_HASH overridable via -Dru.source.commit.hash. - assertTrue on the post-upgrade topology waitForCondition (was silently ignored); compare verified values as boxed Integer to avoid an NPE on a missing key. Verified green on macOS. The Linux path compiles and its addressing primitives are validated, but a full Linux/CI run is still required. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

- IgniteContainer.stop(): drop the unconditional "shut down gracefully" log. The image entrypoint does not forward SIGTERM, so the await times out and the node is force-stopped via super.stop(); the message contradicted the preceding "Proceeding with forceful stop." warning. - IgniteContainer.testClassesJar(): null-check File.listFiles() (returns null on an unreadable dir) so a missing class directory fails with a clear IOException instead of an NPE in the loop. - IgniteRebalanceOnUpgradeTest: compare the post-put value as a boxed Integer too (the get(1000)-loop already does), so a missing key fails the assertion instead of NPE-ing on unboxing. - Scope the "bind communication to loopback" comment to macOS (on Linux it binds to 0.0.0.0) and drop a trailing-whitespace blank line. No behavioral change on the macOS path. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

anton-vinogradov

Final review. Built locally on Java 17: the compatibility module test-compiles against testcontainers 2.0.5 and passes strict checkstyle (-Pcheckstyle, 0 errors); RAT is fine (it treats .sh as binary). I could not run the end-to-end test (no Docker on my machine, and the source image has to be built manually).

Inline I've left a handful of ready-to-apply suggestions (unused param, a misleading comment, an unclosed Files.list stream, and the missing ASF header in build_docker_image.sh) plus a few higher-level points: a unit test + log for the core TcpDiscoverySpi change, a Docker-availability guard so a stray mvn test doesn't fail hard, the WIP default for ru.source.commit.hash, and stale jars in the in-place libs swap. Nothing blocking — mostly polish. Thanks for the thorough DEVNOTES!

anton-vinogradov · 2026-06-30T18:22:25Z

        if (!node.equals(locNode))
-            addrs.removeIf(addr -> addr.getAddress().isLoopbackAddress() && locNode.socketAddresses().contains(addr));
+            addrs.removeIf(addr -> addr.getAddress() == null ||
+                (addr.getAddress().isLoopbackAddress() && locNode.socketAddresses().contains(addr)));


This is the only production change in the PR, and it's a sensible NPE guard for unresolved addresses. A few asks:

It's currently exercised only by the Docker smoke test, which isn't part of RunAll — could you add a focused unit test for getEffectiveNodeAddresses with an unresolved InetSocketAddress (getAddress() == null)?

Dropping the address silently makes a fully-unresolvable node hard to diagnose; a log.debug(...) on the removed address would help.

Since the rest of the PR is test infrastructure, consider landing this core change as its own commit / sub-ticket so it's reviewed independently.

Without this change, the test will fail:

[c1099d16-e7d7-49f4-925c-53329286c444] java.lang.NullPointerException: Cannot invoke "java.net.InetAddress.isLoopbackAddress()" because the return value of "java.net.InetSocketAddress.getAddress()" is null [c1099d16-e7d7-49f4-925c-53329286c444] at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.lambda$getEffectiveNodeAddresses$0(TcpDiscoverySpi.java:1272) [c1099d16-e7d7-49f4-925c-53329286c444] at java.base/java.util.ArrayList.removeIf(Unknown Source) [c1099d16-e7d7-49f4-925c-53329286c444] at java.base/java.util.ArrayList.removeIf(Unknown Source) [c1099d16-e7d7-49f4-925c-53329286c444] at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.getEffectiveNodeAddresses(TcpDiscoverySpi.java:1272) [c1099d16-e7d7-49f4-925c-53329286c444] at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.getEffectiveNodeAddresses(TcpDiscoverySpi.java:1253) [c1099d16-e7d7-49f4-925c-53329286c444] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.pingJoiningNode(ServerImpl.java:7665) [c1099d16-e7d7-49f4-925c-53329286c444] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.processJoinRequestMessage(ServerImpl.java:7602) [c1099d16-e7d7-49f4-925c-53329286c444] at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7076) [c1099d16-e7d7-49f4-925c-53329286c444] at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)

Thanks, but the NPE being real was never the question — the guard is correct and I'm happy for it to land. My asks were about how it lands, not whether:

Unit test. This production path is currently exercised only by the Docker RU smoke test, which isn't part of RunAll — so nothing guards it in CI. A focused getEffectiveNodeAddresses test with an unresolved InetSocketAddress (getAddress() == null) asserting "no NPE + address filtered out" would lock the behavior in.

Diagnostics. Silently dropping a fully-unresolvable node is hard to debug; a log.debug(...) on the removed address would help — this very stack trace is a good example of why, since otherwise you only see the join failure downstream.

Own sub-ticket / commit. Since it's the only production change in an otherwise test-infra PR, splitting it out gets it a discovery-maintainer review independent of the harness.

And please drop this stack trace into the JIRA ticket / commit message as the reproducer — it's exactly the context a reviewer needs.

Added unit test org.apache.ignite.spi.discovery.tcp.TcpDiscoverySelfTest#testGetEffectiveNodeAddressesFiltersUnresolvedAddressWithoutNpe

Great, thanks — the test wraps the node so socketAddresses() returns an unresolved InetSocketAddress and asserts both no-NPE and that it's filtered out, which is exactly the case. That closes ask (1).

Still open, though non-blocking from my side: (2) a log.debug(...) on the dropped address for diagnosability, and (3) landing this core change under its own sub-ticket so it gets a discovery-maintainer review independent of the harness.

ignitetcbot · 2026-07-01T13:04:37Z

TCBot Test Analysis

TeamCity: --> Run :: All Results
Branch: pull/13184/head
Base: master
Tested commit: 4e691cd

Possible Blockers (0)

No blockers found.

New Tests (4)

SPI (Discovery): 2 tests
- IgniteSpiDiscoverySelfTestSuite: TcpDiscoveryMdcSelfTest.testGetEffectiveNodeAddressesFiltersUnresolvedAddressWithoutNpe - PASSED
- IgniteSpiDiscoverySelfTestSuite: TcpDiscoverySslTrustedSelfTest.testGetEffectiveNodeAddressesFiltersUnresolvedAddressWithoutNpe - PASSED
SPI (Discovery) 2: 2 tests
- IgniteSpiDiscoverySelfTestSuite2: TcpDiscoverySslSelfTest.testGetEffectiveNodeAddressesFiltersUnresolvedAddressWithoutNpe - PASSED
- IgniteSpiDiscoverySelfTestSuite2: TcpDiscoverySelfTest.testGetEffectiveNodeAddressesFiltersUnresolvedAddressWithoutNpe - PASSED

maksaska

Step 5/11 : COPY apache-ignite* apache-ignite
When using COPY with more than one source file, the destination must be a directory and end with a /

Error during Docker image build

Fix step COPY apache-ignite* apache-ignite for x86_64

Could we build the Docker image using HEAD instead of a specific pinned commit?

Currently, to build an older Ignite version, I have to manually modify step 5 of the Dockerfile directly on that older commit. Using HEAD would make this process dynamic and much simpler.

maksaska · 2026-07-13T12:05:03Z

+                .setBackups(1)
+                .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
+
+            ClientCache<Integer, Integer> cache = client(cluster.containers().get(0).clientAddress()).createCache(cfg);


2 issues here:

If test failed to removed the work dir from previous run, the next try will reuse the data. Hence the errors:

[2026-07-13T13:49:58,072][ERROR][main][] Test failed [test=IgniteRebalanceOnUpgradeTest#testRollingUpgrade, duration=99234] org.apache.ignite.client.ClientException: Ignite failed to process request [3]: Failed to start cache (a cache with the same name is already started): ru-test-cache (server status code [1001]) ... org.apache.ignite.testframework.junits.GridAbstractTest$6.run(GridAbstractTest.java:2488) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.apache.ignite.internal.client.thin.ClientServerError: Ignite failed to process request [3]: Failed to start cache (a cache with the same name is already started): ru-test-cache (server status code [1001]) at org.apache.ignite.internal.client.thin.TcpClientChannel.processNextMessage(TcpClientChannel.java:581) at ...

I fail to remove the local work dir with my user without root.

Fixed.

On Linux, the container is now started with the current host user's UID:GID (detected via id -u:id -g).
Docker supports numeric uid:gid in --user without the user existing inside the container's /etc/passwd, so this works out of the box.

maksaska · 2026-07-13T13:09:57Z

+        LOGGER.info("Graceful stop of node {}", hostname);
+
+        getDockerClient().stopContainerCmd(getContainerId())
+            .withTimeout(30)


This approach might cause issues with Ignite's internal lifecycle. We should explicitly stop Ignite first to ensure a predictable shutdown procedure.

Currently, the test log show 'Failed to check connection to previous node', which indicates that a backward connection check was initiated during the teardown.

The root cause of "Failed to check connection to previous node" is not that docker stop bypasses Ignite's lifecycle — it's that the 30-second timeout was too short for Ignite to complete its graceful shutdown.

How shutdown actually works:

The container entrypoint (run.sh) uses exec java, meaning the JVM replaces the shell process and becomes PID 1. When docker stop sends SIGTERM, the JVM receives it directly and triggers all registered shutdown hooks. Ignite's internal lifecycle (persistence flush, cluster leave, discovery notification, socket close) runs as part of these hooks.

Why the warning appeared:

"Failed to check connection to previous node" is logged inside TcpDiscoverySpi's internal implementation (ServerImpl.SocketReader.checkConnection) on remaining neighbor nodes. When they detect that a node left the cluster, they attempt to verify the connection to it. If the stopping node gets SIGKILLED (after the 30s timeout) before it can properly leave the discovery ring, neighbors perceive it as a sudden disappearance and initiate this check — hence the warning.

Fix:

Increased the docker stop timeout from 30 → 120 seconds (the standard recommended value for docker-java shutdown hooks used by eclipse-temurin images). This gives Ignite enough time to:

Flush persistence data to disk

Leave the discovery ring cleanly

Notify neighboring nodes

Close all socket connections

The 120s value is configurable via the SHUTDOWN_TIMEOUT_SEC constant.

No explicit "stop Ignite first" is needed because SIGTERM to JVM PID 1 is the explicit stop signal — it's the standard Docker way to trigger application shutdown hooks.

maksaska · 2026-07-13T13:34:16Z

+    public void upgradeAndRestart() throws Exception {
+        LOGGER.info("Cleaning up old libs in container {}", hostname);
+
+        ExecResult result = execInContainer("sh", "-c", "rm -f " + LIBS_DIR_PATH + "*");


Why we remove libs on alive ignite node?

The short answer: because you can't exec into a stopped Docker container.

The current sequence is:

1. execInContainer("rm -rf libs/*") ← container must be running 2. stopGraceful() ← container stops 3. copy new libs + restart ← container starts again

Why not reverse the order (stop first, then clean)?

If we stop the container first, we could still copy new jars via the Docker API (copyFileToContainer works on stopped containers). The order could be:

1. stopGraceful() 2. exec is NOT possible → but copyFileToContainer OVERWRITES files anyway 3. restart

However, copyFileToContainer copies individual files — it doesn't clean up jars that existed in the source image but are absent in the target. If the source version has extra jars the target doesn't need, they would linger.

wernerdv · 2026-07-16T13:35:36Z

Step 5/11 : COPY apache-ignite* apache-ignite
When using COPY with more than one source file, the destination must be a directory and end with a /

Error during Docker image build
Fix step COPY apache-ignite* apache-ignite for x86_64

Could we build the Docker image using HEAD instead of a specific pinned commit?

Currently, to build an older Ignite version, I have to manually modify step 5 of the Dockerfile directly on that older commit. Using HEAD would make this process dynamic and much simpler.

I've addressed both points in build_docker_image.sh:

COPY directive fix: The script now automatically patches all Dockerfiles (appending / to the destination of COPY apache-ignite* apache-ignite) right before building the Docker image.
This patch is applied dynamically and automatically reverted after the script finishes, so you no longer need to manually modify the Dockerfile on older commits.
Using HEAD: The script already supports building from HEAD out of the box.
If you run it without passing a specific commit hash (./build_docker_image.sh), it will use the latest commit (git rev-parse HEAD) to build and tag the Docker image.

ignitetcbot · 2026-07-23T16:31:31Z

TCBot Test Analysis

TeamCity: --> Run :: All Results
Branch: pull/13184/head
Base: master
Tested commit: 234facd

Possible Blockers (0)

No blockers found.

New Tests (4)

SPI (Discovery) 2: 2 tests
- IgniteSpiDiscoverySelfTestSuite2: TcpDiscoverySslSelfTest.testGetEffectiveNodeAddressesFiltersUnresolvedAddressWithoutNpe - PASSED
- IgniteSpiDiscoverySelfTestSuite2: TcpDiscoverySelfTest.testGetEffectiveNodeAddressesFiltersUnresolvedAddressWithoutNpe - PASSED
SPI (Discovery): 2 tests
- IgniteSpiDiscoverySelfTestSuite: TcpDiscoveryMdcSelfTest.testGetEffectiveNodeAddressesFiltersUnresolvedAddressWithoutNpe - PASSED
- IgniteSpiDiscoverySelfTestSuite: TcpDiscoverySslTrustedSelfTest.testGetEffectiveNodeAddressesFiltersUnresolvedAddressWithoutNpe - PASSED

wernerdv force-pushed the testcontainers branch 4 times, most recently from 0ce5477 to b7493e9 Compare May 29, 2026 05:15

wernerdv changed the title ~~WIP Use testcontainers for RU tests~~ IGNITE-28738 Implement basic Rolling Upgrade test using testcontainers Jun 1, 2026

wernerdv force-pushed the testcontainers branch from b7493e9 to 1f27f57 Compare June 1, 2026 12:40

wernerdv force-pushed the testcontainers branch from 3b203f9 to e43c95b Compare June 17, 2026 18:55

chesnokoff reviewed Jun 22, 2026

View reviewed changes

Comment thread ...atibility/src/test/java/org/apache/ignite/compatibility/ru/IgniteRebalanceOnUpgradeTest.java Outdated

wernerdv force-pushed the testcontainers branch 2 times, most recently from 4f23dfa to 7c2a550 Compare June 25, 2026 16:28

wernerdv added 20 commits June 25, 2026 21:40

PoC

b17ef7f

refactoring

a90e8c5

add logger impl

132b1d5

move to compatibility

fceb804

copy plugin to libs

fffb2f7

add sh script

6c9b724

vibe

a02aa70

WORK

5ebd23d

def data region fix

71f01ca

WIP

3c4c0d0

WIP

1b35495

WIP

7c79071

qwen

ac295b2

Revert "qwen"

bdbbfca

This reverts commit 0b6e0ef.

fix

386c34e

cleanup

5ab41fe

WIP

75bd37e

WIP

9ae1437

WIP

0ad4656

WIP

8334558

anton-vinogradov mentioned this pull request Jun 25, 2026

IGNITE-28738 RU testcontainers test: fix class injection, Linux fast-path, faster shutdown wernerdv/ignite#1

Merged

anton-vinogradov and others added 4 commits June 26, 2026 11:39

add docker upgrade mode

c623971

minor

99da189

anton-vinogradov reviewed Jun 30, 2026

View reviewed changes

wernerdv added 3 commits July 1, 2026 12:43

fix claude comments

75f3117

Merge branch 'master' into testcontainers

4e691cd

fix claude comments v2

89bd83f

maksaska reviewed Jul 13, 2026

View reviewed changes

maksaska suggested changes Jul 13, 2026

View reviewed changes

Merge branch 'master' into testcontainers

5c3df28

wernerdv and others added 2 commits July 16, 2026 18:36

fix review comments

a685072

Merge branch 'master' into testcontainers

8fc4ace

wernerdv force-pushed the testcontainers branch from 3510386 to d6567a7 Compare July 21, 2026 12:24

fix run on linux

e52df0c

wernerdv force-pushed the testcontainers branch from d6567a7 to e52df0c Compare July 22, 2026 14:40

wernerdv and others added 4 commits July 23, 2026 12:05

clean

b7044ae

speed up

feb3550

fix group id is too big

dc2785f

Merge branch 'master' into testcontainers

234facd

maksaska approved these changes Jul 23, 2026

View reviewed changes

minor

04c2219

wernerdv force-pushed the testcontainers branch from 2bc6fd1 to 04c2219 Compare July 24, 2026 17:02

minor update DEVNOTES

377969b

wernerdv merged commit 3dfb922 into apache:master Jul 24, 2026
7 checks passed

wernerdv deleted the testcontainers branch July 24, 2026 17:33

Uh oh!

Conversation

wernerdv commented May 27, 2026

The Contribution Checklist

Notes

Uh oh!

Uh oh!

anton-vinogradov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ignitetcbot commented Jul 1, 2026

TCBot Test Analysis

Possible Blockers (0)

New Tests (4)

Uh oh!

maksaska left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wernerdv Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wernerdv commented Jul 16, 2026

Uh oh!

ignitetcbot commented Jul 23, 2026

TCBot Test Analysis

Possible Blockers (0)

New Tests (4)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wernerdv Jul 16, 2026 •

edited

Loading