feat: project Spark 4 VARIANT columns in native Parquet scans - #5407

Open

peterxcli wants to merge 20 commits into

apache:mainfrom

peterxcli:feat/native-variant-proj

Commits on Aug 21, 2026

feat: project Spark 4 VARIANT columns in native Parquet scans
peterxcli
committed
45a0ed4

Commits on Aug 22, 2026

review
peterxcli
committed
784c316

Commits on Aug 23, 2026

review
peterxcli
committed
c355fef
add link about spark and arrow-rs issue that need to be fixed so we can get cleaner code
peterxcli
committed
c556a52
update spark issue link
peterxcli
committed
9742a7b

Commits on Aug 24, 2026

Merge branch 'main' into feat/native-variant-proj
peterxcli
authored
0a079de
widen unsigned shredded field
peterxcli
committed
33e513c

Commits on Aug 25, 2026

fix: preserve Spark Variant compatibility when unshredding
peterxcli
committed
ed39c4c

Commits on Aug 26, 2026

Merge remote-tracking branch 'upstream/main' into feat/native-variant-proj
peterxcli
committed
8fb17b1
fix: normalize shredded Variant types before unshredding
peterxcli
committed
360a653
refactor: isolate Parquet Variant normalization
peterxcli
committed
a800bc8
fix: preserve strict Variant Parquet validation
peterxcli
committed
ba37a68
review
peterxcli
committed
1e832c1
fix: normalize remaining Variant scan encodings
peterxcli
committed
a784bea

Commits on Aug 27, 2026

fix: align native Variant Parquet scan semantics
peterxcli
committed
389b134
style: remove redundant interpolation
peterxcli
committed
43e7e17
fix: preserve Spark semantics for Variant scans
peterxcli
committed
23bfd83
fix: preserve Parquet Variant physical semantics
peterxcli
committed
22f588a
Merge branch 'main' of https://github.com/apache/datafusion-comet into feat/native-variant-proj
peterxcli
committed
c2778b7
fix: narrow Variant Unicode scan fallback
peterxcli
committed
a352f46