Skip to content

HIVE-29525: Iceberg partitioned columns are not reflected by RelOptHiveTable #6600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kasakrisz wants to merge 5 commits into
base: master
Choose a base branch
from
Open

HIVE-29525: Iceberg partitioned columns are not reflected by RelOptHiveTable #6600

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7309923
HIVE-29525: Iceberg partitioned columns are not reflected by RelOptHi…
kasakrisz Jul 17, 2026
1aaf48a
use List.copyOf
kasakrisz Aug 6, 2026
8a9550c
set column stats status to Completed when all projected columns from …
kasakrisz Aug 6, 2026
706af48
set column stats status to Completed when all projected columns from …
kasakrisz Aug 6, 2026
e671c7b
update q.out
kasakrisz Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions iceberg/iceberg-handler/src/test/results/positive/bucket_map_join_9.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,22 +35,22 @@ Stage-0
Stage-1
Map 1 vectorized
File Output Operator [FS_23]
Map Join Operator [MAPJOIN_22] (rows=2 width=530)
Map Join Operator [MAPJOIN_22] (rows=4 width=912)
BucketMapJoin:true,Conds:SEL_21._col1, _col2=RS_19._col1, _col2(Inner),Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
<-Map 2 [CUSTOM_EDGE] vectorized
MULTICAST [RS_19]
PartitionCols:_col2, _col1
Select Operator [SEL_18] (rows=2 width=265)
Select Operator [SEL_18] (rows=2 width=456)
Output:["_col0","_col1","_col2"]
Filter Operator [FIL_17] (rows=2 width=265)
Filter Operator [FIL_17] (rows=2 width=456)
predicate:(id is not null and part is not null)
TableScan [TS_3] (rows=2 width=265)
TableScan [TS_3] (rows=2 width=456)
default@tbl,tbl2,Tbl:COMPLETE,Col:COMPLETE,Output:["foid","part","id"]
<-Select Operator [SEL_21] (rows=2 width=265)
<-Select Operator [SEL_21] (rows=2 width=456)
Output:["_col0","_col1","_col2"]
Filter Operator [FIL_20] (rows=2 width=265)
Filter Operator [FIL_20] (rows=2 width=456)
predicate:(id is not null and part is not null)
TableScan [TS_0] (rows=2 width=265)
TableScan [TS_0] (rows=2 width=456)
default@tbl,tbl,Tbl:COMPLETE,Col:COMPLETE,Grouping Num Buckets:100,Grouping Partition Columns:["id","part"],Output:["foid","part","id"]

PREHOOK: query: SELECT * FROM tbl JOIN tbl tbl2 ON tbl.id = tbl2.id AND tbl.part = tbl2.part
Expand Down
760 changes: 359 additions & 401 deletions .../iceberg-handler/src/test/results/positive/delete_iceberg_copy_on_write_partitioned.q.out
View file
Open in desktop

Large diffs are not rendered by default.

212 changes: 106 additions & 106 deletions iceberg/iceberg-handler/src/test/results/positive/dynamic_partition_pruning.q.out
View file
Open in desktop

Large diffs are not rendered by default.

120 changes: 60 additions & 60 deletions iceberg/iceberg-handler/src/test/results/positive/iceberg_insert_into_partition.q.out
View file
Open in desktop

Large diffs are not rendered by default.

120 changes: 60 additions & 60 deletions iceberg/iceberg-handler/src/test/results/positive/iceberg_insert_overwrite_partition.q.out
View file
Open in desktop

Large diffs are not rendered by default.

20 changes: 10 additions & 10 deletions iceberg/iceberg-handler/src/test/results/positive/iceberg_partition_pruner_cache_key.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,22 +293,22 @@ STAGE PLANS:
alias: tbl_ice_pp_key
As of version: s1
filterExpr: (a > 2) (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (a > 2) (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
minReductionHashAggr: 0.99
minReductionHashAggr: 0.4
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: COMPLETE
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
null sort order:
sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: COMPLETE
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
Execution mode: vectorized
Reducer 2
Expand All @@ -325,7 +325,7 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 99 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
Statistics: Num rows: 2 Data size: 107 Basic stats: PARTIAL Column stats: COMPLETE
Statistics: Num rows: 2 Data size: 198 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Expand All @@ -337,14 +337,14 @@ STAGE PLANS:
aggregations: count(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: COMPLETE
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: 'asof_s1' (type: string), _col0 (type: bigint)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: COMPLETE
Statistics: Num rows: 1 Data size: 99 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
Statistics: Num rows: 2 Data size: 107 Basic stats: PARTIAL Column stats: COMPLETE
Statistics: Num rows: 2 Data size: 198 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Expand Down
6 changes: 3 additions & 3 deletions iceberg/iceberg-handler/src/test/results/positive/iceberg_pcr_null_partition.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,14 +109,14 @@ STAGE PLANS:
TableScan
alias: ice_01
filterExpr: ds is not null (type: boolean)
Statistics: Num rows: 2 Data size: 530 Basic stats: COMPLETE Column stats: COMPLETE
Statistics: Num rows: 2 Data size: 710 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string), ds (type: string)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 2 Data size: 530 Basic stats: COMPLETE Column stats: COMPLETE
Statistics: Num rows: 2 Data size: 710 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
Statistics: Num rows: 2 Data size: 530 Basic stats: COMPLETE Column stats: COMPLETE
Statistics: Num rows: 2 Data size: 710 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
Expand Down
42 changes: 21 additions & 21 deletions ...iceberg-handler/src/test/results/positive/iceberg_truncate_partition_with_evolution.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,14 +88,14 @@ STAGE PLANS:
Statistics: Num rows: 3 Data size: 276 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
predicate: (a = 22) (type: boolean)
Statistics: Num rows: 3 Data size: 276 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 2 Data size: 184 Basic stats: COMPLETE Column stats: PARTIAL
Select Operator
expressions: PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string), 22 (type: int), b (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 3 Data size: 1440 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 2 Data size: 960 Basic stats: COMPLETE Column stats: PARTIAL
File Output Operator
compressed: false
Statistics: Num rows: 3 Data size: 1440 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 2 Data size: 960 Basic stats: COMPLETE Column stats: PARTIAL
table:
input format: org.apache.iceberg.mr.hive.HiveIcebergInputFormat
output format: org.apache.iceberg.mr.hive.HiveIcebergOutputFormat
Expand Down Expand Up @@ -283,14 +283,14 @@ STAGE PLANS:
Statistics: Num rows: 3 Data size: 288 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
predicate: (a = 226784902765739L) (type: boolean)
Statistics: Num rows: 3 Data size: 288 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 2 Data size: 192 Basic stats: COMPLETE Column stats: PARTIAL
Select Operator
expressions: PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string), 226784902765739L (type: bigint), b (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 3 Data size: 1452 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 2 Data size: 968 Basic stats: COMPLETE Column stats: PARTIAL
File Output Operator
compressed: false
Statistics: Num rows: 3 Data size: 1452 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 2 Data size: 968 Basic stats: COMPLETE Column stats: PARTIAL
table:
input format: org.apache.iceberg.mr.hive.HiveIcebergInputFormat
output format: org.apache.iceberg.mr.hive.HiveIcebergOutputFormat
Expand Down Expand Up @@ -501,17 +501,17 @@ STAGE PLANS:
TableScan
alias: test_ice_str
filterExpr: (b = 'ddd') (type: boolean)
Statistics: Num rows: 7 Data size: 665 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 7 Data size: 1344 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
predicate: (b = 'ddd') (type: boolean)
Statistics: Num rows: 7 Data size: 665 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 4 Data size: 768 Basic stats: COMPLETE Column stats: PARTIAL
Select Operator
expressions: PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string), a (type: bigint), 'ddd' (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 7 Data size: 3381 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 4 Data size: 1932 Basic stats: COMPLETE Column stats: PARTIAL
File Output Operator
compressed: false
Statistics: Num rows: 7 Data size: 3381 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 4 Data size: 1932 Basic stats: COMPLETE Column stats: PARTIAL
table:
input format: org.apache.iceberg.mr.hive.HiveIcebergInputFormat
output format: org.apache.iceberg.mr.hive.HiveIcebergOutputFormat
Expand Down Expand Up @@ -742,17 +742,17 @@ STAGE PLANS:
TableScan
alias: test_ice_date
filterExpr: (b = DATE'2022-02-07') (type: boolean)
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: NONE
Statistics: Num rows: 2 Data size: 4438 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (b = DATE'2022-02-07') (type: boolean)
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: NONE
Statistics: Num rows: 2 Data size: 4438 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string), a (type: bigint), DATE'2022-02-07' (type: date)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: NONE
Statistics: Num rows: 2 Data size: 4438 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: NONE
Statistics: Num rows: 2 Data size: 4438 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.iceberg.mr.hive.HiveIcebergInputFormat
output format: org.apache.iceberg.mr.hive.HiveIcebergOutputFormat
Expand Down Expand Up @@ -933,14 +933,14 @@ STAGE PLANS:
Statistics: Num rows: 3 Data size: 192 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
predicate: (a = 1.156748927566759E11D) (type: boolean)
Statistics: Num rows: 3 Data size: 192 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: PARTIAL
Select Operator
expressions: PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string), 1.156748927566759E11D (type: double), b (type: date)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 3 Data size: 1356 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 2 Data size: 904 Basic stats: COMPLETE Column stats: PARTIAL
File Output Operator
compressed: false
Statistics: Num rows: 3 Data size: 1356 Basic stats: COMPLETE Column stats: PARTIAL
Statistics: Num rows: 2 Data size: 904 Basic stats: COMPLETE Column stats: PARTIAL
table:
input format: org.apache.iceberg.mr.hive.HiveIcebergInputFormat
output format: org.apache.iceberg.mr.hive.HiveIcebergOutputFormat
Expand Down Expand Up @@ -1112,17 +1112,17 @@ STAGE PLANS:
TableScan
alias: test_ice_double_date
filterExpr: ((a = 1.156748927566759E11D) and (b = DATE'2022-02-07')) (type: boolean)
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: NONE
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: PARTIAL
Filter Operator
predicate: ((a = 1.156748927566759E11D) and (b = DATE'2022-02-07')) (type: boolean)
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: NONE
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: PARTIAL
Select Operator
expressions: PARTITION__SPEC__ID (type: int), PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: bigint), PARTITION__PROJECTION (type: string), 1.156748927566759E11D (type: double), DATE'2022-02-07' (type: date)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: NONE
Statistics: Num rows: 2 Data size: 904 Basic stats: COMPLETE Column stats: PARTIAL
File Output Operator
compressed: false
Statistics: Num rows: 2 Data size: 128 Basic stats: COMPLETE Column stats: NONE
Statistics: Num rows: 2 Data size: 904 Basic stats: COMPLETE Column stats: PARTIAL
table:
input format: org.apache.iceberg.mr.hive.HiveIcebergInputFormat
output format: org.apache.iceberg.mr.hive.HiveIcebergOutputFormat
Expand Down
12 changes: 6 additions & 6 deletions iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_2.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,9 +259,9 @@ Stage-3
<-Map 2 [CUSTOM_SIMPLE_EDGE] vectorized, llap
File Output Operator [FS_32]
table:{"name:":"default.bucketmapjoin_tmp_result"}
Select Operator [SEL_31] (rows=785 width=366)
Select Operator [SEL_31] (rows=238000 width=366)
Output:["_col0","_col1","_col2"]
Map Join Operator [MAPJOIN_30] (rows=785 width=186)
Map Join Operator [MAPJOIN_30] (rows=238000 width=186)
BucketMapJoin:true,Conds:RS_27._col0=SEL_29._col0(Inner),Output:["_col0","_col1","_col3"]
<-Map 1 [CUSTOM_EDGE] vectorized, llap
MULTICAST [RS_27]
Expand All @@ -281,7 +281,7 @@ Stage-3
PARTITION_ONLY_SHUFFLE [RS_35]
Group By Operator [GBY_34] (rows=1 width=704)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12"],aggregations:["max(length(key))","avg(COALESCE(length(key),0))","count(1)","count(key)","compute_bit_vector_hll(key)","max(length(value1))","avg(COALESCE(length(value1),0))","count(value1)","compute_bit_vector_hll(value1)","max(length(value2))","avg(COALESCE(length(value2),0))","count(value2)","compute_bit_vector_hll(value2)"]
Select Operator [SEL_33] (rows=785 width=366)
Select Operator [SEL_33] (rows=238000 width=366)
Output:["key","value1","value2"]
Please refer to the previous Select Operator [SEL_31]

Expand Down Expand Up @@ -423,9 +423,9 @@ Stage-3
<-Map 2 [CUSTOM_SIMPLE_EDGE] vectorized, llap
File Output Operator [FS_32]
table:{"name:":"default.bucketmapjoin_tmp_result"}
Select Operator [SEL_31] (rows=809 width=366)
Select Operator [SEL_31] (rows=124712 width=366)
Output:["_col0","_col1","_col2"]
Map Join Operator [MAPJOIN_30] (rows=809 width=186)
Map Join Operator [MAPJOIN_30] (rows=124712 width=186)
BucketMapJoin:true,Conds:RS_27._col0=SEL_29._col0(Inner),Output:["_col0","_col1","_col3"]
<-Map 1 [CUSTOM_EDGE] vectorized, llap
MULTICAST [RS_27]
Expand All @@ -445,7 +445,7 @@ Stage-3
PARTITION_ONLY_SHUFFLE [RS_35]
Group By Operator [GBY_34] (rows=1 width=704)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12"],aggregations:["max(length(key))","avg(COALESCE(length(key),0))","count(1)","count(key)","compute_bit_vector_hll(key)","max(length(value1))","avg(COALESCE(length(value1),0))","count(value1)","compute_bit_vector_hll(value1)","max(length(value2))","avg(COALESCE(length(value2),0))","count(value2)","compute_bit_vector_hll(value2)"]
Select Operator [SEL_33] (rows=809 width=366)
Select Operator [SEL_33] (rows=124712 width=366)
Output:["key","value1","value2"]
Please refer to the previous Select Operator [SEL_31]

Expand Down
Loading
Loading