Table partitioning optimizations in Parquet and other formats - #14

Merged

tsafin merged 44 commits into

tsafin/parquet_partitioning

Jul 18, 2026

Commits on Jul 14, 2026

docs: add parallel-partition optimization plan + SF=5 baseline
tsafin
committed
7d4b59d
perf(tpch): add --batch-rows, set Parquet batch default to 65536
tsafin
committed
456a4d2
perf(tpch): add --parquet-output-buffer-bytes, default 32 MiB buffered Parquet output
tsafin
committed
ba2e016
perf(tpch): schedule largest tables first (LPT) in parallel all-table path
tsafin
committed
bd5362a
docs: record Phase 3 commit hash
tsafin
committed
a45f987
perf(tpch): pipeline dbgen+convert with Parquet write, default queue capacity 1
tsafin
committed
cdfee35
docs: record Phase 4 commit hash
tsafin
committed
f602c6b
docs: defer dbgen partitioning pending RNG compatibility
tsafin
committed
b3923d2
docs: record terminal outcomes for remaining partition ideas
tsafin
committed
439459c

Commits on Jul 15, 2026

feat(tpch): preserve dbgen state across serial partitions

tsafin
and
claude
committed
3fcb376
docs: accept exact serial dbgen partitioning

tsafin
and
claude
committed
b8fd522
feat(tpch): add deterministic parallel partition files

tsafin
and
claude
committed
189d8c8
docs: record parallel partition file results

tsafin
and
claude
committed
273535c
docs: reject adaptive partition selection

tsafin
and
claude
committed
688c96d
docs: defer ordered Parquet merge

tsafin
and
claude
committed
12b904a
docs: reject fused lineitem construction

tsafin
and
claude
committed
c6ca486
docs: defer narrow physical types

tsafin
and
claude
committed
fb80a6f
docs: finalize optimization phase status

tsafin
and
claude
committed
c64f836
docs: plan all-format table optimization
tsafin
committed
a4d6c46
docs: record Phase 11 all-format all-table baseline matrix
tsafin
committed
5707297
perf(tpcds): add --batch-rows, generalize from TPC-H Phase 1
tsafin
committed
713e023
perf(tpch): pipeline writer in all-table parallel child path
tsafin
committed
c04b858
perf(tpcds): add --pipeline-queue-capacity bounded write pipeline
tsafin
committed
c0bcb7e
docs: record Phase 13 common bounded write pipeline results
tsafin
committed
d19a742
perf(paimon): add opt-in output buffering for internal Parquet data files
tsafin
committed
87cb350
perf(iceberg): add opt-in output buffering for internal Parquet data files
tsafin
committed
75388cb
perf(tpch): wire existing buffer-bytes flag into Paimon/Iceberg writers
tsafin
committed
5d732bc
perf(tpcds): add --output-buffer-bytes for Parquet/Paimon/Iceberg
tsafin
committed
136984c
docs: record Phase 14 writer-specific buffering results
tsafin
committed
da3a512
perf(tpcds): schedule largest tables first in all-table generation
tsafin
committed
0112d46
docs: record Phase 15 all-table scheduling results
tsafin
committed
0ade382
docs: reject further deterministic partitioning after investigation
tsafin
committed
6ba43d2
perf(paimon,iceberg): inject io_uring into internal Parquet data-file writes
tsafin
committed
934b22e
docs: record Phase 17 writer-specific async I/O results
tsafin
committed
9a54d83
docs: record Phase 18 final cross-format verification
tsafin
committed
e087eed
perf(tpch): add partsupp partition support for SF=10 re-evaluation

tsafin
and
claude
committed
0e39de8
docs: correct Phase 16 partsupp extrapolation with real SF=10 measurement
tsafin
committed
858c708

Commits on Jul 16, 2026

feat(parquet): add ordered single-file merge prototype (Phase 8 revisit)
tsafin
committed
4cf1fe4
docs: record Phase 8 ordered-merge revisit results
tsafin
committed
b6f8688
docs: re-check stale/optional deferred items (Workstream C)
tsafin
committed
1b46b8d
perf(tpch): retest Phase 9/10 with real code instead of profiling only
tsafin
committed
160ee98
docs: record Phase 9/10 retest results (real code, not just profiling)
tsafin
committed
2f435e5
docs: add optimization results waterfall diagram

tsafin
and
claude
committed
4e406fb
fix(parquet): address PR review feedback on footer merge and tests
tsafin
committed
b3fbdfa