[AMORO-4037] Add support for dynamic refresh interval for table metadata refreshing in TableRuntimeRefreshExecutor #4038

zhangwl9 · 2026-01-14T06:46:41Z

Why are the changes needed?

Close #4037.

Brief change log

The new feature involves two methods in "TableRuntimeRefreshExecutor".
- #getNextExecutingTime()
  - When dynamic refresh interval is enabled，retrieving the latest refresh interval from "DefaultTableRuntime" to determine the next execution time for "TableRuntimeRefreshExecutor#execute"
- #execute()
  - Depending on the evaluated results, determine whether the table requires optimization.
  - When the evaluation table requires no optimization, extend the interval for its next refresh (incrementally increasing the interval by a fixed step).
  - When evaluating table requiring optimization, shorten its next refresh interval (halving the interval each time).
dynamic refresh interval in [ refresh-tables.interval(1 min)，Long.MAX_VALUE]

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

zhangwl9 · 2026-01-16T01:59:22Z

@zhoujinsong @turboFei @xxubai Could you help me review it when you are free? Thanks!

Copilot

Pull request overview

This PR implements a dynamic refresh interval feature for table metadata refreshing in TableRuntimeRefreshExecutor. The feature uses an AIMD (Additive Increase Multiplicative Decrease) algorithm to adjust refresh intervals based on whether tables need optimization, helping to reduce resource consumption for healthy tables while maintaining responsiveness for tables requiring optimization.

Changes:

Added configuration properties for adaptive refresh intervals with configurable maximum interval and increase step
Implemented adaptive interval calculation logic that increases intervals for healthy tables and decreases them for tables needing optimization
Added comprehensive test coverage for the adaptive interval functionality

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/user-guides/configurations.md	Added documentation for three new configuration properties related to adaptive refresh intervals
amoro-format-iceberg/src/main/java/org/apache/amoro/table/TableProperties.java	Defined constants for adaptive refresh table properties and their default values
amoro-common/src/main/java/org/apache/amoro/config/OptimizingConfig.java	Added fields, getters, setters, and updated equals/hashCode/toString methods for adaptive refresh configuration
amoro-ams/src/main/java/org/apache/amoro/server/table/TableConfigurations.java	Integrated parsing of adaptive refresh properties from table properties into OptimizingConfig
amoro-ams/src/main/java/org/apache/amoro/server/table/DefaultTableRuntime.java	Added fields and methods to track latest refresh interval and optimization evaluation status
amoro-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java	Implemented core adaptive interval logic with AIMD algorithm and updated execute method to apply adaptive intervals
amoro-ams/src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java	Added comprehensive test cases covering various adaptive interval scenarios including boundary conditions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/user-guides/configurations.md

klion26

Thanks for the contribution. I left some comments; please let me know what you think. thanks.

IMHO, this feature is valuable(especially when users use HMS as the catalog), as we can't use the event-based trigger as RestCatalog, but when there are many "static table" in the catalog, it will be a great burden for AMS to evaluate these tables, and changing the interval for every table is not a doable(as there are so many tables), and keep the default behavior as the same as before, would not add any inconvenience if we don't need it.

klion26 · 2026-01-27T11:29:28Z

docs/user-guides/configurations.md

+| self-optimizing.min-plan-interval                    | 60000            | The minimum time interval between two self-optimizing planning action                                                                                                                                                               |
+| self-optimizing.filter                               | NULL             | Filter conditions for self-optimizing, using SQL conditional expressions, without supporting any functions. For the timestamp column condition, the ISO date-time formatter must be used. For example: op_time > '2007-12-03T10:15:30'. |
+| self-optimizing.refresh-table.adaptive.enabled       | false            | Whether to enable adaptive refresh interval for refreshing table metadata                                                                                                                                                           |
+| self-optimizing.refresh-table.adaptive.max-interval  | 3600000(1 hour)  | The maximum time interval in milliseconds to refresh table metadata                                                                                                                                                                 |


Maybe we can set the max-interval to an unset value, so that we can keep the behavior same as before the current pr.

klion26 · 2026-01-27T11:31:06Z

amoro-format-iceberg/src/main/java/org/apache/amoro/table/TableProperties.java

  public static final int SNAPSHOT_MIN_COUNT_DEFAULT = 1;

+  public static final String SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_ENABLED =
+      "self-optimizing.refresh-table.adaptive.enabled";


If we need to set self-optimizing.refresh-table.adaptive.enabled and self-optimizing.refresh-table.adaptive.max-interval, how about using only self-optimizing.refresh-table.adaptive.max-interval? Some value (like -1 or zero) means that disable it, and some other value means that to enable it.

klion26 · 2026-01-27T11:31:41Z

amoro-format-iceberg/src/main/java/org/apache/amoro/table/TableProperties.java

+  public static final boolean SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_ENABLED_DEFAULT = false;
+  public static final String SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_MAX_INTERVAL =
+      "self-optimizing.refresh-table.adaptive.max-interval";
+  public static final long SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_MAX_INTERVAL_DEFAULT =


How about add the time unit in the name, SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_MAX_INTERVAL_DEFAULT -> SELF_OPTIMIZING_REFRESH_TABLE_ADAPTIVE_MAX_INTERVAL_MS_DEFAULT

Same as the other places

klion26 · 2026-01-28T02:56:31Z

...-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java

      if (optimizingConfig.isMetadataBasedTriggerEnabled()
          && !MetadataBasedEvaluationEvent.isEvaluatingNecessary(
              optimizingConfig, table, tableRuntime.getLastPlanTime())) {
+        tableRuntime.setLatestEvaluatedNeedOptimizing(false);


Is it better that we change the return value of tryEvaluatingPendingInput to boolean -- indicate whether we need to optimize after evaluating?
So that we can call tableRuntime.setLatestEvaluteNeedOptimizing in one place in execute below,

Please let me what do you think about it.

klion26 · 2026-01-28T02:58:46Z

...-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java

+      // Update adaptive interval according to evaluated result.
+      if (defaultTableRuntime.getOptimizingConfig().getRefreshTableAdaptiveEnabled()) {
+        long newInterval = getAdaptiveExecutingInterval(defaultTableRuntime);
+        defaultTableRuntime.setLatestRefreshInterval(newInterval);


Maybe we can call tableRuntime.setLatestEvaluatedNeedOptimizing(true); and defaultTableRuntime.setLatestRefreshInterval(newInterval); both in this if block, so that we wouldn't miss either

klion26 · 2026-01-28T03:01:39Z

.../src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java

+    executor.execute(mockTableRuntime);
+
+    // Verify that setLatestRefreshInterval was called with the expected value
+    verify(mockTableRuntime, Mockito.times(1)).setLatestRefreshInterval(expectedInterval);


Instead using verify here to validate, is there any way we can validate the value of the true interval after executor.execute in L147.

The true interval is the value keeped in the table runtime.

klion26 · 2026-01-28T03:05:07Z

.../src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java

+
+    // Test unhealthy table (need optimizing) - interval should decrease
+    mockTableRuntime =
+        getMockTableRuntimeWithAdaptiveInterval(tableRuntime, true, true, INTERVAL * 2);


Is there a way to reuse this tableRuntime so we can observe the change in interval before and after two consecutive execute calls? Reusing a single tableRuntime function would also be more suitable for production scenarios.

…g in the TableRuntimeRefreshExecutor.

klion26

Thanks for the update, I've left some comments, please take another look.

klion26 · 2026-01-30T09:17:07Z

...-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java

+          tableRuntime.getTableIdentifier());
+      return false;
+    } else {
+      logger.debug(


Could we add some comments to describle the "semantics" of the return value, and why would this case would be true?

From below, the name is needOptimizing, looks like whethere we need to do an opitmizing, but in this branch, we don't need to optimizing in current loop? as there is an ongoing optimizing process

klion26 · 2026-01-30T09:19:34Z

...-ams/src/main/java/org/apache/amoro/server/scheduler/inline/TableRuntimeRefreshExecutor.java

+      }
+
+      // Update adaptive interval according to evaluated result.
+      if (defaultTableRuntime.getOptimizingConfig().getRefreshTableAdaptiveMaxIntervalMs() > 0) {


Maybe we can add a function in OptimizingConfig which return true/false show whethere we enabled this feature

klion26 · 2026-01-30T09:31:40Z

docs/user-guides/configurations.md

+| self-optimizing.full.rewrite-all-files                  | true             | Whether full optimizing rewrites all files or skips files that do not need to be optimized                                                                                                                                              |
+| self-optimizing.min-plan-interval                       | 60000            | The minimum time interval between two self-optimizing planning action                                                                                                                                                                   |
+| self-optimizing.filter                                  | NULL             | Filter conditions for self-optimizing, using SQL conditional expressions, without supporting any functions. For the timestamp column condition, the ISO date-time formatter must be used. For example: op_time > '2007-12-03T10:15:30'. |
+| self-optimizing.refresh-table.adaptive.max-interval-ms  | -1               | The maximum time interval in milliseconds to refresh table metadata. The default value is -1, which disables the adaptive refresh.                                                                                                                |


Maybe we need to add some description here: after this has been enabled, the interval may be larger than defaultTableRuntime.getOptimizingConfig().getMinorLeastInterval() * 4L / 5

klion26 · 2026-01-30T09:41:02Z

.../src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java

+    createDatabase();
+    createTable();
+
+    DefaultTableRuntime tableRuntime =


Could you please reuse the same table runtime, in the current test, we'll create every new table runtime in buildTableRuntimeWithAdaptiveRefresh

If we need to update the config for an existing table runtime, maybe we can add a new class who extend DefaultTableRuntime and overrides the getStore function.

class TestTableRuntime extends DefaultTableRuntime { TableRuntimeStore testStore; TestTableRuntime(TableRuntimeStore store) { ... this.testStore = store } void updateRuntimeStore(TableRuntimeStore newStore) { ... this.testStore = newStore; } @Override TableConfiguration getStore() { this.testStore; }

klion26 · 2026-02-02T02:18:09Z

.../src/test/java/org/apache/amoro/server/scheduler/inline/TestTableRuntimeRefreshExecutor.java

+    // Test minimum boundary - interval should not below INTERVAL
+    // The unhealthy table (need optimizing) - interval should decrease half
+    // The currentInterval is INTERVAL + STEP, the latest interval is INTERVAL not (INTERVAL + STEP)
+    // / 2


Place (INTERVAL + STEP) / 2 in the same line is better than 2 lines

github-actions bot added type:docs Improvements or additions to documentation module:ams-server Ams server module module:common labels Jan 14, 2026

zhangwl9 changed the title ~~[AMORO-4037] Add support for dynamic refresh interval for table metadata refreshing~~ [AMORO-4037] Add support for dynamic refresh interval for table metadata refreshing in TableRuntimeRefreshExecutor Jan 14, 2026

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch from 60e829e to 298152d Compare January 15, 2026 07:06

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch from 298152d to 4fec76a Compare January 21, 2026 05:42

turboFei requested a review from Copilot January 22, 2026 04:46

Copilot AI reviewed Jan 22, 2026

View reviewed changes

docs/user-guides/configurations.md Outdated Show resolved Hide resolved

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch 2 times, most recently from 16b59c2 to 3298bcf Compare January 27, 2026 06:15

klion26 reviewed Jan 28, 2026

View reviewed changes

zhangwl9 force-pushed the AMORO-dynamic-refresh-interval-dev branch 3 times, most recently from 1a658f8 to 4f0ca86 Compare January 29, 2026 09:25

张文领 added 2 commits January 30, 2026 16:23

Add support for dynamic refresh interval for table metadata refreshin…

b43db25

…g in the TableRuntimeRefreshExecutor.

fixup

72aad3b

klion26 force-pushed the AMORO-dynamic-refresh-interval-dev branch from 4f0ca86 to 72aad3b Compare January 30, 2026 08:23

Merge branch 'master' into AMORO-dynamic-refresh-interval-dev

cb81daf

klion26 reviewed Feb 2, 2026

View reviewed changes

[AMORO-4037] Add support for dynamic refresh interval for table metadata refreshing in TableRuntimeRefreshExecutor #4038

Are you sure you want to change the base?

[AMORO-4037] Add support for dynamic refresh interval for table metadata refreshing in TableRuntimeRefreshExecutor #4038

Uh oh!

Conversation

zhangwl9 commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

Uh oh!

zhangwl9 commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

klion26 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

klion26 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhangwl9 commented Jan 14, 2026 •

edited

Loading

zhangwl9 commented Jan 16, 2026 •

edited

Loading