HIVE-29322: Avoid TopNKeyOperator When Map-Side LIMIT Pushdown Provides Better Pruning for ORDER BY LIMIT Queries #6202
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Indhumathi27
force-pushed
the
disable_topn
branch
from
3a7b050 to
cbb41cf
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
Indhumathi27
changed the title
Indhumathi27
force-pushed
the
disable_topn
branch
from
cbb41cf to
75f9861
Compare
Contributor
Author
|
@kasakrisz @deniskuzZ Can you help to review this PR. Thanks |
Indhumathi27
force-pushed
the
disable_topn
branch
4 times, most recently
from
db493f2 to
a003d03
Compare
Indhumathi27
changed the title
asf-ci-hive
added
tests pending
tests passed
and removed
tests unstable
tests pending
labels
Indhumathi27
force-pushed
the
disable_topn
branch
from
a003d03 to
bc90bb4
Compare
asf-ci-hive
added
tests pending
tests passed
and removed
tests passed
tests pending
labels
Contributor
Author
|
@ayushtkn @zabetak @kasakrisz @deniskuzZ Can you help to review the PR. thanks |
deniskuzZ
reviewed
Dec 6, 2025
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
Outdated
Show resolved
Hide resolved
deniskuzZ
reviewed
Dec 6, 2025
| // Skip the current optimization when a simple global ORDER BY...LIMIT is present | ||
| // (topN > -1 and hasOnlyOrderByLimit()). | ||
| // This plan structure is handled more efficiently by the specialized 'TopN In Reducer' optimization. | ||
| if (reduceSinkDesc.getTopN() > -1 && reduceSinkDesc.hasOnlyOrderByLimit()) { |
Member
There was a problem hiding this comment.
WDYT @zabetak, @kasakrisz , @thomasrebele, @okumin, @ngsg
Contributor
There was a problem hiding this comment.
I'm also curious about the two questions in the JIRA ticket. I don't have an instant answer on whether it is optimal to reuse Top-N for the global sort. However, in my impression, it is at least suboptimal and might not be so bad.
Disclaimer: My next response might not be timely because I have to visit a hospital every day.
Indhumathi27
force-pushed
the
disable_topn
branch
from
bc90bb4 to
035e8bf
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests passed
tests pending
labels
…es Better Pruning
Indhumathi27
force-pushed
the
disable_topn
branch
from
035e8bf to
4e3f88b
Compare
|
Member
|
I left some questions under the JIRA ticket to better understand the issue that we are trying to solve here. |
Contributor
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR updates TopNKeyProcessor to skip creating TopNKeyOperator when ReduceSinkDesc.topN is already set by LIMIT pushdown for ORDER BY LIMIT case. This prevents TopNKey from overriding pushdown and ensures the map-side LIMIT optimization is applied correctly.
Why are the changes needed?
Currently, when a query includes ORDER BY + LIMIT, LIMIT pushdown is generated during planning but is effectively overridden by the subsequent TopNKey rewrite. As a result, TopNKey operator receives full input rather than a reduced data set, leading to worse performance (e.g., 16M rows forwarded to reducer instead of a few). In cases where global ordering uses a single reducer, LIMIT pushdown is sufficient and far more efficient. This fix prevents unnecessary TopNKey creation so that pushdown can reduce shuffle and significantly improve execution time.
Test reports:
For query: select * from table order by h limit 100;
Total num of rows: 67764224
with fix:
without fix:
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manual testing + existing testcases