Core, OpenAPI: Preserve avg value sizes in REST JSON - #17855
Draft
huan233usc wants to merge 7 commits into
Draft
Conversation
Keep average value sizes in REST content-file JSON round trips and drop stale path-size statistics when rewrite_table_path rewrites position deletes. Generated-by: Cursor
singhpk234
reviewed
Aug 28, 2026
Comment on lines
+5126
to
+5143
| IntegerMap: | ||
| type: object | ||
| properties: | ||
| keys: | ||
| type: array | ||
| items: | ||
| $ref: '#/components/schemas/IntegerTypeValue' | ||
| description: "List of integer column ids for each corresponding value" | ||
| values: | ||
| type: array | ||
| items: | ||
| $ref: '#/components/schemas/IntegerTypeValue' | ||
| description: "List of integer values, matched to 'keys' by index" | ||
| example: | ||
| { | ||
| "keys": [ 1, 2 ], | ||
| "values": [ 8, 16 ] | ||
| } |
Contributor
There was a problem hiding this comment.
can't we use ValueMap ? like what we use for other stats
iceberg/open-api/rest-catalog-open-api.yaml
Line 5126 in ef8a69d
Comment on lines
+5244
to
+5249
| avg-value-sizes: | ||
| allOf: | ||
| - $ref: '#/components/schemas/IntegerMap' | ||
| description: | ||
| Map of column id to average value size in memory (uncompressed) in | ||
| bytes over non-null values |
Contributor
There was a problem hiding this comment.
would this be part of data file ? like lower / upper bound are ?
huan233usc
changed the title
Leave REST JSON round-trip changes here; rewrite_table_path handling moves to a separate PR. Generated-by: Cursor
huan233usc
changed the title
huan233usc
marked this pull request as draft
August 28, 2026 05:45
2 tasks
Contributor
Author
|
Hi @singhpk234 for quick review -- I just realize that this PR was not marked as draft when I am double checking the generated code vs spec. Will asked for another review when I finish my self validation. Sry for confusion |
added 5 commits
August 27, 2026 22:53
avg_value_size_in_bytes lives in v4 content_stats field structs, not a v1-v3 metrics map. REST JSON now uses content-stats keyed by field ID. Generated-by: Cursor
Use the spec wording for content_stats and avg_value_size_in_bytes. Generated-by: Cursor
Match other optional ContentFileParser fields: explicit JSON null is missing, and invalid field IDs get a contextual error. Generated-by: Cursor
Satisfy checkstyle by creating the parsed content stats map with the relocated Maps utility. Generated-by: Cursor
Generated-by: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #17451. Average value sizes were dropped when content files crossed the planned REST scan JSON boundary.
Preserve
avg-value-sizesfor both data and delete files inContentFileParserJSON round trips, and document the optional field in the REST OpenAPI schema.Test plan
./gradlew :iceberg-core:test --tests org.apache.iceberg.TestContentFileParser --tests org.apache.iceberg.TestFileScanTaskParser --tests org.apache.iceberg.TestDataTaskParser./gradlew :iceberg-core:spotlessCheckAI Disclosure