Skip to content

fix(store): refresh gRPC channels after address changes - #3130

Open
bitflicker64 wants to merge 1 commit into
apache:masterfrom
bitflicker64:fix/hstore-channel-refresh-3124
Open

fix(store): refresh gRPC channels after address changes#3130
bitflicker64 wants to merge 1 commit into
apache:masterfrom
bitflicker64:fix/hstore-channel-refresh-3124

Conversation

@bitflicker64

Copy link
Copy Markdown
Contributor

Purpose of the PR

HStore caches gRPC channels and blocking/asynchronous stubs by Store target.
When a stable target resolves to a different address, those pools can retain the
failed transport indefinitely even though the replacement Store is reachable.

This is the channel-lifecycle half of #3124. Complete recovery behind a stable
DNS name still requires the finite positive DNS TTL from #3126; this patch does
not bypass an indefinitely stale JVM DNS cache.

Main Changes

  • Fingerprint each target's sorted resolved address set before selecting a
    channel.
  • Atomically shut down and replace the channel pool when that fingerprint
    changes or when an initially unresolved target first resolves.
  • Rebuild blocking and asynchronous stub pools when their channels no longer
    belong to the current pool.
  • Order concurrent resolutions and retry in-flight stub builds so stale work
    cannot replace or return retired channels.
  • Add four self-contained refresh regression tests without a running PD or
    Store cluster.

The validated implementation was developed through commits bf9086ee,
4dd41233, 26b333ab, and c71c77a0.

Verifying these changes

  • Need tests and can be verified as follows:
    • mvn test -pl hugegraph-store/hg-store-test -am -Dtest=AbstractGrpcClientTest -Dsurefire.failIfNoSpecifiedTests=false: 4 distinct tests passed with 0 failures, errors, or skips on Java 11.
    • mvn test -pl hugegraph-store/hg-store-test -am -P store-client-test -Dsurefire.failIfNoSpecifiedTests=false: the refresh-only client suite ran 4 tests with 0 failures, errors, or skips on Java 11.
    • mvn editorconfig:format: passed with 0 files changed.
    • mvn clean compile -Dmaven.javadoc.skip=true: passed across all 38 reactor modules on Java 11.

Does this PR potentially affect the following parts?

  • Dependencies
  • Modify configurations
  • The public API
  • Other affects
  • Nope

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need

@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. store Store module tests Add or improve test cases labels Jul 30, 2026
Refresh cached channel pools when a Store target resolves to a new address. Rebuild stale blocking and async stub pools, and guard concurrent resolution and publication races.

Fixes apache#3124
@bitflicker64
bitflicker64 force-pushed the fix/hstore-channel-refresh-3124 branch from 218b681 to 26218cb Compare July 30, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL This PR changes 500-999 lines, ignoring generated files. store Store module tests Add or improve test cases

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

[Bug][HStore][Kubernetes] Data writes remain unavailable after Store pod replacement due to stale DNS resolution

1 participant