Skip to content

Core: Fix REST scan tasks dropping content file sequence numbers - #17835

Open
waterWang wants to merge 1 commit into
apache:mainfrom
waterWang:fix-17833-content-file-seq-numbers
Open

Core: Fix REST scan tasks dropping content file sequence numbers#17835
waterWang wants to merge 1 commit into
apache:mainfrom
waterWang:fix-17833-content-file-seq-numbers

Conversation

@waterWang

Copy link
Copy Markdown

ContentFileParser serialized neither the data sequence number nor the file sequence number of a content file, and the REST ContentFile schema did not define fields for them. Any file scan task that crosses the REST scan planning API therefore arrived with null sequence numbers on its data file and delete files.

Since delete files apply only to data files with an equal or smaller data sequence number, engines consuming those tasks could not determine delete applicability at read time, breaking merge-on-read for tables with delete files.

Changes:

  • ContentFileParser.toJson: serialize data-sequence-number and file-sequence-number when present
  • ContentFileParser.fromJson: deserialize both fields and set them on the constructed file objects
  • rest-catalog-open-api.yaml: add data-sequence-number and file-sequence-number to the ContentFile schema
  • Tests: round-trip tests for data files, delete files, and backward-compatible null defaults

Fixes #17833

@JohnEarle

Copy link
Copy Markdown

Note: this duplicates #17834, which I opened for my issue #17833 a few minutes earlier.

My PR also includes the regenerated OpenAPI Python model (required for spec changes per the open-api README) and a scan-task round-trip test.

@yangshangqing95

Copy link
Copy Markdown
Contributor

I looked through this account’s commit history @waterWang, and I strongly suspect that it is a bot. It appears to scan open issues across many repositories—not just Iceberg—and may be using AI to generate code changes in an attempt to address those issues, without any meaningful follow-up afterward.

I don’t know what the motivation or purpose behind this is, but I don’t think we should encourage this kind of behavior, at the very least.

ContentFileParser serialized neither the data sequence number nor the file
sequence number of a content file, and the REST ContentFile schema did not
define fields for them. Any file scan task that crosses the REST scan
planning API therefore arrived with null sequence numbers on its data file
and delete files. Since delete files apply only to data files with an equal
or smaller data sequence number, engines consuming those tasks could not
determine delete applicability at read time, breaking merge-on-read for
tables with delete files.

This change serializes and deserializes data-sequence-number and
file-sequence-number in ContentFileParser (following the existing first-row-id
pattern), and adds both fields to the REST ContentFile schema.

Fixes apache#17833
@ebyhr

ebyhr commented Aug 28, 2026

Copy link
Copy Markdown
Member

The issue #17833 is closed as a bug on the engine side, not Iceberg design. We can close this PR now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

server-side scan planning - REST scan tasks drop content file sequence numbers breaking merge-on-read

4 participants