Skip to content

[UUID 6/8] UUID multi-stage engine (planner + runtime)#18874

Open
xiangfu0 wants to merge 5 commits into
apache:masterfrom
xiangfu0:uuid-split/06-mse-planner-runtime
Open

[UUID 6/8] UUID multi-stage engine (planner + runtime)#18874
xiangfu0 wants to merge 5 commits into
apache:masterfrom
xiangfu0:uuid-split/06-mse-planner-runtime

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

What

Multi-stage (v2) engine support for UUID.

Changes

  • expressions.proto UUID / UUID_ARRAY enum values + planner type resolution, Rex parsing, proto serde, TypeFactory
  • HashJoinOperator / EnrichedHashJoinOperator + UuidLookupTable (LookupTable.normalizeKey), OneUuidKeyGroupIdGenerator, ServerPlanRequestUtils

Enables

UUID equi-joins and group-by in the multi-stage engine.

⚠️ Rolling-upgrade note: the new proto enum values are unknown to older brokers/servers; multi-stage queries with UUID literals fail on mixed-version clusters. Atomic-upgrade feature.

Depends on

#18869 and #18871.

About this PR / how to review

This is part 6 of 8 splitting #18140 (first-class logical UUID type) into layered PRs, as requested there.

The split is enabled by the v1 design: DataType.UUID has stored type BYTES, so most paths handle it automatically; each PR adds explicit UUID semantics to one subsystem. Head branch lives on xiangfu0/pinot.

This PR is stacked on #18873 (branch uuid-split/05-agg-groupby-distinct). Because GitHub PRs to apache must base on master, the Files-changed tab is cumulative (it includes layers 1–6) until the PRs below it merge. Review the commit titled [UUID 6/8] UUID multi-stage engine (planner + runtime) — that is this layer's change. Each parent merge shrinks this diff after a rebase.

Full stack (merge bottom → top)

  1. [UUID 1/8] Add logical UUID type foundation (pinot-spi) #18869 — [UUID 1/8] logical UUID type foundation (pinot-spi)
  2. [UUID 2/8] UUID ingest and segment storage #18870 — [UUID 2/8] UUID ingest and segment storage
  3. [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) #18871 — [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders)
  4. [UUID 4/8] UUID server-side predicates, CAST and transforms #18872 — [UUID 4/8] UUID server-side predicates, CAST and transforms
  5. [UUID 5/8] UUID aggregation, group-by and distinct #18873 — [UUID 5/8] UUID aggregation, group-by and distinct
  6. [UUID 6/8] UUID multi-stage engine (planner + runtime) #18874 — [UUID 6/8] UUID multi-stage engine (planner + runtime)
  7. [UUID 7/8] UUID scalar UDFs and partitioning #18875 — [UUID 7/8] UUID scalar UDFs and partitioning
  8. [UUID 8/8] UUID integration tests, benchmarks and docs #18876 — [UUID 8/8] UUID integration tests, benchmarks and docs

After #18869 lands, PRs 2/4/7 (#18870, #18872, #18875) only depend on it and can be reviewed in parallel; PRs 5/6 (#18873, #18874) also need #18871; PR 8 (#18876) needs #18874.

Full feature description, v1 design contract, scope exclusions, and benchmark numbers: #18140.

@codecov-commenter

codecov-commenter commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 52.20713% with 563 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.79%. Comparing base (9ceca0f) to head (763c96a).

Files with missing lines Patch % Lines
...ot/common/request/context/RequestContextUtils.java 55.30% 48 Missing and 11 partials ⚠️
...java/org/apache/pinot/common/utils/DataSchema.java 55.55% 36 Missing and 8 partials ⚠️
...ator/transform/function/CastTransformFunction.java 34.92% 40 Missing and 1 partial ⚠️
...sform/function/ScalarTransformFunctionWrapper.java 17.07% 34 Missing ⚠️
...pby/NoDictionarySingleColumnGroupKeyGenerator.java 42.00% 23 Missing and 6 partials ⚠️
.../function/DistinctCountULLAggregationFunction.java 0.00% 26 Missing ⚠️
...perator/query/NonScanBasedAggregationOperator.java 20.00% 22 Missing and 2 partials ⚠️
...ion/DistinctCountCPCSketchAggregationFunction.java 0.00% 24 Missing ⚠️
...upby/NoDictionaryMultiColumnGroupKeyGenerator.java 26.66% 19 Missing and 3 partials ⚠️
...e/operator/groupby/OneUuidKeyGroupIdGenerator.java 0.00% 20 Missing ⚠️
... and 35 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18874      +/-   ##
============================================
- Coverage     64.81%   64.79%   -0.02%     
  Complexity     1347     1347              
============================================
  Files          3396     3399       +3     
  Lines        212503   213512    +1009     
  Branches      33484    33703     +219     
============================================
+ Hits         137732   138351     +619     
- Misses        63610    63932     +322     
- Partials      11161    11229      +68     
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-21 64.79% <52.20%> (-0.02%) ⬇️
temurin 64.79% <52.20%> (-0.02%) ⬇️
unittests 64.79% <52.20%> (-0.02%) ⬇️
unittests1 56.74% <41.08%> (-0.05%) ⬇️
unittests2 37.15% <18.50%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0 xiangfu0 force-pushed the uuid-split/06-mse-planner-runtime branch 5 times, most recently from bdd6933 to 69c36b3 Compare July 5, 2026 19:24
xiangfu0 added 5 commits July 6, 2026 17:56
Part 2/8 of splitting apache#18140 (logical UUID type). Rebased onto master (now includes the merged apache#18869 type foundation); stacked on 5a451fa.

Downstream references use the UuidKey class merged in apache#18869.
Part 3/8 of splitting apache#18140 (logical UUID type). Rebased onto master (now includes the merged apache#18869 type foundation); stacked on uuid-split/02-ingest-storage.

Downstream references use the UuidKey class merged in apache#18869.
Part 4/8 of splitting apache#18140 (logical UUID type). Rebased onto master (now includes the merged apache#18869 type foundation); stacked on uuid-split/03-result-rendering.

Downstream references use the UuidKey class merged in apache#18869.
Part 5/8 of splitting apache#18140 (logical UUID type). Rebased onto master (now includes the merged apache#18869 type foundation); stacked on uuid-split/04-sse-predicates-cast.

Downstream references use the UuidKey class merged in apache#18869.
Part 6/8 of splitting apache#18140 (logical UUID type). Rebased onto master (now includes the merged apache#18869 type foundation); stacked on uuid-split/05-agg-groupby-distinct.

Downstream references use the UuidKey class merged in apache#18869.
@xiangfu0 xiangfu0 force-pushed the uuid-split/06-mse-planner-runtime branch from 69c36b3 to 763c96a Compare July 7, 2026 00:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants