Skip to content

[core] Support latest snapshot delta scan in batch - #9142

Open
wwj6591812 wants to merge 1 commit into
apache:masterfrom
wwj6591812:agent/add-latest-delta-scan-mode
Open

[core] Support latest snapshot delta scan in batch#9142
wwj6591812 wants to merge 1 commit into
apache:masterfrom
wwj6591812:agent/add-latest-delta-scan-mode

Conversation

@wwj6591812

Copy link
Copy Markdown
Contributor

Purpose

Batch pipelines sometimes need to process only the newest committed snapshot, but currently must first query its ID and construct incremental-between boundaries.

This adds scan.mode = 'latest-delta' for batch sources. At planning time it selects the literal latest snapshot N and reuses the existing DELTA scanner for (N - 1, N].

It deliberately does not search backwards for an APPEND snapshot. Therefore, if the latest snapshot is COMPACT or OVERWRITE, the query returns no records instead of treating rewritten files as new logical data. Streaming use is rejected, and boundary or time-travel companion options are rejected rather than ignored.

This is a convenience mode, not a persistent consumer cursor. Workflows that must consume multiple snapshots reliably should continue to use explicit incremental-between boundaries or tags.

SELECT *
FROM t /*+ OPTIONS('scan.mode' = 'latest-delta') */;

Tests

  • Added core coverage for latest APPEND, latest COMPACT, empty table, streaming rejection, level-0 preservation, incompatible options, and postpone-merge rejection.
  • Added a Flink SQL integration test proving two appends return only the latest snapshot.
  • Added Flink catalog option coverage for batch and streaming modes.
  • Regenerated configuration docs with the official docs profile.

@wwj6591812
wwj6591812 marked this pull request as ready for review August 10, 2026 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant