HIVE-29525: Iceberg partitioned columns are not reflected by RelOptHiveTable by kasakrisz · Pull Request #6600 · apache/hive

kasakrisz · 2026-07-13T11:38:53Z

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

mvn test -Dtest.output.overwrite -Dtest=TestIcebergCliDriver -Dqfile=merge_iceberg_copy_on_write_partitioned.q,merge_iceberg_copy_on_write_unpartitioned.q,merge_iceberg_orc.q,merge_iceberg_partitioned_orc.q,merge_with_null_check_on_joining_col.q,update_iceberg_copy_on_write_partitioned.q,update_iceberg_copy_on_write_unpartitioned.q,update_iceberg_partitioned_avro.q,update_iceberg_partitioned_orc.q,update_iceberg_partitioned_parquet.q,update_iceberg_unpartitioned_parquet.q -pl itests/qtest-iceberg -Pitests

rubenada · 2026-07-24T13:26:17Z

-                    TableScan [TS_0] (rows=238 width=89)
-                      default@srcbucket_mapjoin_part_1_n1,a,Tbl:COMPLETE,Col:COMPLETE,Grouping Num Buckets:2,Grouping Partition Columns:["key"],Output:["key","part"]
+                    TableScan [TS_0] (rows=238 width=188)
+                      default@srcbucket_mapjoin_part_1_n1,a,Tbl:COMPLETE,Col:PARTIAL,Grouping Num Buckets:2,Grouping Partition Columns:["key"],Output:["key","part"]


Is this (and other tests) moving from Col:COMPLETE to Col:PARTIAL because partitioned columns are now considered, and there are no stats for them? Or is it another reason?

The explanation fits my understanding. One of the partitioned columns doesn't support statistics. Maybe the bucket(2, key) in srcbucket_mapjoin_part_1_n1.

In the expression bucket(2, key) currently Hive can't treat key as a partition column. In the first version of the patch I tried but this leads to incomplete stats.

deniskuzZ · 2026-07-27T13:56:02Z

                    File Output Operator [FS_32]
                      table:{"name:":"default.bucketmapjoin_tmp_result"}
-                      Select Operator [SEL_31] (rows=785 width=366)
+                      Select Operator [SEL_31] (rows=238000 width=366)


seems like an over-estimate?
cc @okumin

deniskuzZ · 2026-07-27T13:56:39Z

-                      TableScan [TS_3] (rows=238 width=89)
-                        default@srcbucket_mapjoin_part_2_n4,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","part"]
+                      TableScan [TS_3] (rows=238 width=188)
+                        default@srcbucket_mapjoin_part_2_n4,b,Tbl:COMPLETE,Col:PARTIAL,Output:["key","part"]


is this expected change?

same as #6600 (comment)

deniskuzZ · 2026-07-27T13:59:49Z

@@ -306,17 +306,17 @@ STAGE PLANS:
                TableScan
                  alias: srcpart_iceberg
                  filterExpr: ds is not null (type: boolean)
-                  Statistics: Num rows: 2000 Data size: 188000 Basic stats: COMPLETE Column stats: COMPLETE
+                  Statistics: Num rows: 2000 Data size: 368000 Basic stats: COMPLETE Column stats: PARTIAL


is this expected change? why Column stats is PARTIAL?

#6600 (comment)

deniskuzZ · 2026-07-27T14:05:49Z

                  As of version: s1
                  filterExpr: (a > 2) (type: boolean)
-                  Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE
+                  Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: PARTIAL


partitioned by spec (a), where a > 2 why it's PARTIAL ?

#6600 (comment)

deniskuzZ · 2026-07-27T14:10:11Z

      throw new CalciteCteException("Failed to create temporary location", e);
    }
    Table hiveTable = new Table(metaTable);
+    hiveTable.setFields(columns.stream()


seems unrelated to the PR, do we have tests for this?

This is called when a CTE is materialized at the CBO level. A temporary table is created to store the CTE result set and the table's fields were not set.

deniskuzZ · 2026-07-27T14:20:35Z

-  public static <T> ImmutableMap<Integer, T> getColInfoMap(List<T> hiveCols,
-      int startIndx) {
-    Builder<Integer, T> bldr = ImmutableMap.<Integer, T> builder();
+  public static ImmutableMap<Integer, ColumnInfo> getColInfoMap(List<ColumnInfo> hiveCols, Table table) {


how about

public static ImmutableMap<Integer, ColumnInfo> getColInfoMap( List<ColumnInfo> hiveCols, ToIntFunction<String> indexByName) { return Maps.uniqueIndex(hiveCols, ci -> indexByName.applyAsInt(ci.getInternalName())); }

and then

this.hiveNonPartitionColsMap = HiveCalciteUtil.getColInfoMap(hiveNonPartitionCols, hiveTblMetadata::getColumnIndexByName);

deniskuzZ · 2026-07-27T14:26:45Z

+        int allColCount = tabMetaData.getAllCols().size();
+        List<ColumnInfo> colInfoList = new ArrayList<>(Collections.nCopies(allColCount, null));
+        Set<String> partColNames = new HashSet<>(tabMetaData.getPartColNames());
+        ArrayList<ColumnInfo> nonPartitionColumns = new ArrayList<>(fields.size());


code to interface

change to

List<ColumnInfo> nonPartitionColumns...

deniskuzZ · 2026-07-27T14:36:51Z

        }

-        final TableType tableType = obtainTableType(tabMetaData);
+        List<ColumnInfo> partitionColumns = new ArrayList<>(partitionColumnSet);


List.copyOf(partitionColumnSet)

deniskuzZ · 2026-07-27T14:42:00Z

        // 3.2 Add column info corresponding to partition columns
+        // Normally, the column names in a schema should be unique, but in the case of Iceberg v1 tables,
+        // updating the partition spec doesn't remove the existing partition keys, so we can end up with a
+        // partition spec containing multiple columns with the same name.


can you elaborate? shouldn't the type be void in case column is dropped as part of schema evolution?

…veTable

…a TS is partition column and the stats are available

sonarqubecloud · 2026-08-06T14:13:22Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.3% Duplication on New Code

See analysis details on SonarQube Cloud

kasakrisz marked this pull request as draft July 13, 2026 11:38

asf-ci-hive added tests pending tests unstable and removed tests pending labels Jul 13, 2026

kasakrisz force-pushed the HIVE-29525-master-relopthivetable branch from 9cae808 to 40c9252 Compare July 21, 2026 13:34

asf-ci-hive added tests pending and removed tests unstable labels Jul 21, 2026

kasakrisz force-pushed the HIVE-29525-master-relopthivetable branch from 40c9252 to 481b616 Compare July 21, 2026 13:38

asf-ci-hive added tests failed tests pending tests unstable and removed tests pending tests failed labels Jul 21, 2026

kasakrisz force-pushed the HIVE-29525-master-relopthivetable branch from 481b616 to 726f582 Compare July 22, 2026 04:38

asf-ci-hive added tests pending tests passed and removed tests unstable tests pending labels Jul 22, 2026

kasakrisz marked this pull request as ready for review July 22, 2026 11:44

rubenada reviewed Jul 24, 2026

View reviewed changes

deniskuzZ reviewed Jul 27, 2026

View reviewed changes

HIVE-29525: Iceberg partitioned columns are not reflected by RelOptHi…

7309923

…veTable

kasakrisz force-pushed the HIVE-29525-master-relopthivetable branch from 726f582 to 9fab0a3 Compare August 5, 2026 14:43

asf-ci-hive added tests pending tests failed tests unstable and removed tests passed tests pending tests failed labels Aug 5, 2026

kasakrisz added 4 commits August 6, 2026 09:14

use List.copyOf

1aaf48a

set column stats status to Completed when all projected columns from …

8a9550c

…a TS is partition column and the stats are available

set column stats status to Completed when all projected columns from …

706af48

…a TS is partition column and the stats are available

update q.out

e671c7b

kasakrisz force-pushed the HIVE-29525-master-relopthivetable branch from f7f65ba to e671c7b Compare August 6, 2026 13:22

asf-ci-hive added tests pending and removed tests unstable labels Aug 6, 2026

asf-ci-hive added tests unstable and removed tests pending labels Aug 6, 2026

Uh oh!

Conversation

kasakrisz commented Jul 13, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Aug 6, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

deniskuzZ Jul 27, 2026 •

edited

Loading

deniskuzZ Jul 27, 2026 •

edited

Loading

deniskuzZ Jul 27, 2026 •

edited

Loading

deniskuzZ Jul 27, 2026 •

edited

Loading