AVRO-4307: [javascript] Destroy BlockDecoder on error to stop the source#3875
Open
iemejia wants to merge 3 commits into
Open
AVRO-4307: [javascript] Destroy BlockDecoder on error to stop the source#3875iemejia wants to merge 3 commits into
iemejia wants to merge 3 commits into
Conversation
BlockDecoder decompresses blocks asynchronously, so many block callbacks can
fail for the same corrupt/oversized input. The failure path emitted 'error' once
per failing block with a bare emit(), which:
- produced an error storm (one 'error' event per block; ~1000 for a modest
file rather than a single error),
- never destroyed the decoder, so its readable side was left dangling, and
- via createFileDecoder's plain .pipe(), left the upstream file ReadStream
reading a file already known to be bad and its descriptor open (only
unpiped, never destroyed).
Route every fatal error through a new _onError() that guards on an _errored flag
and calls destroy(err), so exactly one 'error' is surfaced and the stream is
torn down. Decrement _pending before the error check so accounting stays correct
(previously it was skipped on error). Guard _write/_writeChunk so no further
blocks are processed once errored. Switch createFileDecoder from .pipe() to
stream.pipeline() so the source file stream is destroyed on error and its
descriptor released.
Verified on a 394 KB multi-block file decoded with an always-failing codec:
error events 947 -> 1, bytes read after first error 65536 -> 0, source and
decoder both destroyed. Adds regression tests for single-error emission and for
source teardown via createFileDecoder.
Assisted-by: GitHub Copilot:claude-opus-4.8
There was a problem hiding this comment.
Pull request overview
This PR addresses AVRO-4307 in the JavaScript implementation by ensuring BlockDecoder surfaces a fatal decode/decompression error only once and tears down the decoding stream so upstream sources stop reading (preventing error storms and resource leaks). It also updates createFileDecoder to use stream.pipeline() so the source file stream is destroyed on failure.
Changes:
- Add a guarded
BlockDecoder._onError(err)with an_erroredflag and route fatal header/block errors throughdestroy(err). - Fix
_pendingbookkeeping on block callback errors and stop processing further blocks/writes after the decoder has errored. - Switch
createFileDecoderfrom.pipe()tostream.pipeline()and add tests for single-error behavior and teardown.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| lang/js/lib/files.js | Adds single-error teardown logic to BlockDecoder and uses stream.pipeline() in createFileDecoder to stop/destroy the source on failure. |
| lang/js/test/test_files.js | Adds regression tests for “single error” behavior and createFileDecoder error-path teardown. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
lang/js/lib/files.js:264
invalid sync markercurrently errors viacb(new Error(...))rather than_onError(). That means_erroredis never set for this fatal path, so later async block callbacks (already in-flight) can still reach_onErrorand potentially surface additional errors. Also, if a later write arrives after an earlier destroy, the_erroredguard returns without callingcb. Route the sync-marker failure through_onError()and always invokecb()when bailing due to_errored.
while ((block = tryReadBlock(tap))) {
if (this._errored) {
return; // A prior block already failed and destroyed the stream.
}
if (!this._syncMarker.equals(block.sync)) {
cb(new Error('invalid sync marker'));
return;
}
this._decompress(block.data, this._createBlockCallback());
… destroys the decoder
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Fix a DoS-adjacent defect in the JavaScript
BlockDecoder, resolving AVRO-4307.BlockDecoderdecompresses blocks asynchronously, so many block callbacks can fail for the same bad input (corrupt block, invalid sync marker, unknown codec, or the AVRO-4287 decompression size guard). The failure path emitted'error'once per failing block with a bareemit()and never destroyed the stream. Confirmed empirically on plainmain(independent of the decompression-limit work) with a 394 KB multi-block file decoded through the realcreateFileDecoderpath using a codec that always errors:'error'eventsRoot cause is in
BlockDecoder._createBlockCallback(bare emit, nodestroy, no guard).createFileDecoder's plain.pipe()compounds it: a destination error only unpipes the source, never destroys it, so the file descriptor leaks.Changes:
_erroredflag and a guarded_onError(err)that callsdestroy(err), so exactly one error surfaces and the stream unwinds. Route all fatal errors (block callback, invalid magic, unknown codec, schema parse) through it._pendingbefore the error check (previously skipped on error, which could hang the stream atfinish)._write/_writeChunkso no further blocks are processed once errored.createFileDecoderfrom.pipe()tostream.pipeline()so the source file stream is destroyed and its descriptor released on error.Verifying this change
This change added tests and can be verified as follows:
surfaces a single error for many failing blocks— asserts exactly one'error'event (no storm) and that the decoder is destroyed when every block fails.createFileDecoder tears down the source on error— asserts a single error and decoder teardown on thecreateFileDecoderpath.npm test, 385 + new tests) andjshint(npm run lint) is clean.Documentation