Core: Fix REST scan tasks dropping content file sequence numbers - #17835
Open
waterWang wants to merge 1 commit into
Open
Core: Fix REST scan tasks dropping content file sequence numbers#17835waterWang wants to merge 1 commit into
waterWang wants to merge 1 commit into
Conversation
Contributor
|
I looked through this account’s commit history @waterWang, and I strongly suspect that it is a bot. It appears to scan open issues across many repositories—not just Iceberg—and may be using AI to generate code changes in an attempt to address those issues, without any meaningful follow-up afterward. I don’t know what the motivation or purpose behind this is, but I don’t think we should encourage this kind of behavior, at the very least. |
ContentFileParser serialized neither the data sequence number nor the file sequence number of a content file, and the REST ContentFile schema did not define fields for them. Any file scan task that crosses the REST scan planning API therefore arrived with null sequence numbers on its data file and delete files. Since delete files apply only to data files with an equal or smaller data sequence number, engines consuming those tasks could not determine delete applicability at read time, breaking merge-on-read for tables with delete files. This change serializes and deserializes data-sequence-number and file-sequence-number in ContentFileParser (following the existing first-row-id pattern), and adds both fields to the REST ContentFile schema. Fixes apache#17833
Member
|
The issue #17833 is closed as a bug on the engine side, not Iceberg design. We can close this PR now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ContentFileParser serialized neither the data sequence number nor the file sequence number of a content file, and the REST ContentFile schema did not define fields for them. Any file scan task that crosses the REST scan planning API therefore arrived with null sequence numbers on its data file and delete files.
Since delete files apply only to data files with an equal or smaller data sequence number, engines consuming those tasks could not determine delete applicability at read time, breaking merge-on-read for tables with delete files.
Changes:
ContentFileParser.toJson: serializedata-sequence-numberandfile-sequence-numberwhen presentContentFileParser.fromJson: deserialize both fields and set them on the constructed file objectsrest-catalog-open-api.yaml: adddata-sequence-numberandfile-sequence-numberto the ContentFile schemaFixes #17833