Skip to content

[core] Release every reader in SortMergeReaderWithMinHeap.close() - #9163

Open
PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:fix-sortmergereader-close-leak
Open

[core] Release every reader in SortMergeReaderWithMinHeap.close()#9163
PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:fix-sortmergereader-close-leak

Conversation

@PDGGK

@PDGGK PDGGK commented Aug 11, 2026

Copy link
Copy Markdown

Purpose

Closes #9162.

SortMergeReaderWithMinHeap.close() releases one RecordReader per sorted run being merged, across three loops, and every call was a bare close(). The first one to throw abandoned the rest of its loop and both loops after it.

This is not a two-resource teardown: there is one reader per sorted run, each holds an open data file, and it runs on the compaction path. A single unreadable file therefore strands every descriptor behind it for the remainder of the merge. element.iterator.releaseBatch() had the same exposure — no checked exception, but it can still throw, and when it did the close() on the next line was skipped.

Each step now runs, the first failure propagates, and later ones are attached to it as suppressed.

On the choice of helper. IOUtils.closeAll is the obvious candidate and I did not use it: it declares throws Exception, and both existing callers (FormatTableFileWriter:93, FileChannelManagerImpl:126) widened their own signature to match. This method overrides RecordReader.close() throws IOException and cannot widen. ExceptionUtils.firstOrSuppressed is the right fit — its javadoc gives this exact close-loop as its worked example.

SortMergeReaderWithLoserTree needs no change; it delegates to a single loserTree.close().

Tests

New SortMergeReaderWithMinHeapCloseTest, three cases:

case asserts
an earlier reader fails to close all three readers still closed; the first failure propagates with the later one as suppressed
releaseBatch() throws both readers still closed, and the release failure is what the caller sees
everything closes cleanly no exception — the control

Against the previous close(), the first and third of those fail; the all-succeed control passes on both versions, which is what shows the other two aren't trivially red.

The readers are a small hand-rolled fake rather than mocks, so the test asserts on real call ordering through readBatch() into the heap rather than on stub interactions.

spotless:check, checkstyle:check and apache-rat:check are clean.

Local run caveat. With -Pfast-build, CompactionChangelogFollowUpScannerTest and MergeTreeCompactManagerTest error in setup with Could not instantiate generated class 'RecordComparator' — the codegen module isn't built under that profile. I confirmed it's unrelated by reverting SortMergeReaderWithMinHeap.java to master and deleting the new test: both fail identically. Everything else in the merge-tree and compaction packages passes, including MergeTreeReadersConnectionsLeakTest (20).

close() releases one RecordReader per sorted run being merged, across
three loops -- nextBatchReaders, then the heap, then the polled
elements. Every call was a bare close(), so the first one to throw
abandoned the rest of its loop and both loops after it. Each of those
readers holds an open data file, and this is the compaction path, so a
single unreadable file strands every descriptor behind it for the
remainder of the merge.

Each step now runs and the first failure is the one that propagates,
with later ones attached as suppressed, via ExceptionUtils
.firstOrSuppressed -- the helper whose own javadoc gives this exact
close-loop as its example. IOUtils.closeAll would have been the obvious
choice but does not fit: it declares throws Exception, and both existing
callers widened their own signatures to match, which this override of
RecordReader.close() throws IOException cannot do.

releaseBatch() is collected the same way. It declares no checked
exception but can still throw, and it used to be able to skip the
close() immediately after it.

Three test cases: an earlier reader failing still closes the later ones
and keeps its own exception as primary with the later failure
suppressed; a failing releaseBatch() still closes both readers; and an
all-succeed control that passes against both versions, so the other two
are not trivially red. Against the previous close() the first and third
fail.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] SortMergeReaderWithMinHeap.close() abandons the remaining readers when one close() throws

1 participant