feat: support co-partitioned range right-side equi hash joins#23484
feat: support co-partitioned range right-side equi hash joins#23484gmhelmold wants to merge 3 commits into
Conversation
github-actions
Bot
added
core
|
Done in 852a619: the end-to-end matrix now lives in |
gene-bordegaray
left a comment
gene-bordegaray
left a comment
There was a problem hiding this comment.
few nits but this is great, thank you. Feel free to ping on another round 😄
| // Same rows as `range_partitioned` but split into only three range | ||
| // partitions on `range_key`. Used to exercise the co-partition check when | ||
| // two Range inputs disagree on partition count. | ||
| let narrow_output_partitioning = Partitioning::Range( |
| NULL 35 350 | ||
|
|
||
| ########## | ||
| # TEST 22: Mark Join Marker Semantics over Range Partitioned Inputs |
There was a problem hiding this comment.
nit: can we move this as the last test in this block?
| physical_plan | ||
| 01)FilterExec: non_range_key@1 = 2 OR mark@3, projection=[range_key@0, value@2] | ||
| 02)--HashJoinExec: mode=Partitioned, join_type=LeftMark, on=[(range_key@0, range_key@0)] | ||
| 03)----RepartitionExec: partitioning=Hash([range_key@0], 4), input_partitions=4 |
There was a problem hiding this comment.
to be clear this should disappear after #23453 ? Just putting here to show reationale on why the unit is good
| 30 600 | ||
| 35 700 | ||
|
|
||
| ########## |
There was a problem hiding this comment.
can we add one subset test for sanity check 👍
2 participants
Which issue does this PR close?
Partitioning::Rangeinputs for right-side hash joins #23453.Rationale for this change
#23184 let compatible range-partitioned inputs satisfy inner partitioned hash joins without repartitioning. Right-side partitioned equi joins have the same locality guarantee:
Right,RightSemi,RightAntiandRightMarkall anchor every output row on the probe (right) partition (on_lr_is_preservedis probe-side for all four), so when both inputs are co-partitioned by the join keys, execution stays partition-local and the hash repartition is unnecessary.This removes unnecessary
RepartitionExecs for already-co-located inputs, extending the inner-join behavior from #23184 to the right-side variants. No micro-benchmark included, consistent with #23184; correctness is demonstrated by matched/unmatched execution results below.What changes are included in this PR?
The only production change is widening the existing inner-only gate in
HashJoinExec::input_distribution_requirementsfromjoin_type == JoinType::Innertomatches!(join_type, Inner | Right | RightSemi | RightAnti | RightMark)(underPartitionMode::Partitioned). Theco_partitioned/ range-satisfaction machinery from #23184 is unchanged — the sanity checker and enforce_distribution consume range satisfaction join-type-agnostically.One behavior note for #23376: range co-partitioned right joins now stay
Range/Range, so partitioned dynamic filters are disabled for them (has_partitioned_dynamic_filter_routingreturns false), the same safe delta #23184 introduced for inner joins. These variants are probe-not-preserved for pruning, so dynamic filters were not eligible to prune their probe rows regardless.Are these changes tested?
Yes.
enforce_distribution.rs): 4 reuse tests (one per join type) proving compatible range/range inputs keepRangepartitioning with noRepartitionExec; 4 incompatibility tests proving that mismatched split points, sort options, partition counts, or join-key expressions still insert a hash repartition; sanity-check pairs mirroring Support co-partitioned range inner equi joins #23184.range_partitioning.slt):EXPLAINplan pins plus matched/unmatched result checks forRight(incl.NULLleft values),RightSemi,RightAnti, and an incompatible-layoutRightjoin (repartitions, correct results).mainwithout the gate change (verified by reverting the production diff: exactly the 4 reuse tests fail, everything else passes).RightMarktesting note:RightMarkis not reachable from SQL in sqllogictest (IN-subquery decorrelation emitsLeftMark; physicalRightMarkonly appears via a statistics-based swap), so its co-partitioning behavior is pinned at the optimizer/plan level, and mark null-marker semantics (matched/unmatched/NULLbuild keys) are pinned via theLeftMarkpath in the slt.Are there any user-facing changes?
No API changes. Plans over compatible range-partitioned inputs avoid a hash repartition for right-side equi hash joins.