AVRO-4288: [perl] Enforce a maximum decompressed block size by iemejia · Pull Request #3855 · apache/avro

iemejia · 2026-07-11T13:04:08Z

What is the purpose of the change

When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size (a decompression bomb). This enforces a configurable maximum decompressed size while reading each block, mirroring the Java SDK's decompression limit (AVRO-4247). The limit defaults to 200 MiB and can be overridden with the AVRO_MAX_DECOMPRESS_LENGTH environment variable.

The deflate and bzip2 codecs are inflated in chunks and bounded before the full output is materialized, so an over-limit block is rejected without first allocating its entire decompressed size. The zstandard codec is likewise bounded. Exceeding the limit throws Avro::DataFile::Error::DecompressionSize.

The decompressed-size limit is independent of the reader's block_max_size attribute, which is a compressed-byte threshold on the declared size of a compressed block, not how large a block may decompress to. This PR additionally enforces block_max_size for the first time (previously it was declared but never checked): a block whose declared compressed size exceeds it is rejected up front with the new Avro::DataFile::Error::CompressedBlockSize.

This is part of the umbrella issue AVRO-4283.

Verifying this change

This change added tests and can be verified as follows:

Added t/07_datafile_decompress_limit.t covering over-limit rejection for deflate, bzip2, and zstandard, within-limit decoding, and the block_max_size compressed-size guard (CompressedBlockSize). Also bumps the Compress::Zstd requirement to 0.10 (streaming decompressor) in Makefile.PL and guards the zstandard case in t/04_datafile.t.
Run: cd lang/perl && prove -Ilib t/

Documentation

Does this pull request introduce a new feature? (no — hardening/robustness)
If yes, how is the feature documented? (not applicable; the new AVRO_MAX_DECOMPRESS_LENGTH environment variable is documented in code comments)

When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size. Enforce a configurable maximum decompressed size across the deflate, bzip2 and zstandard codecs, mirroring the Java SDK's decompression limit (AVRO-4247): deflate and bzip2 are inflated in chunks and bounded before the full output is materialized. The limit uses the existing (previously unenforced) block_max_size attribute if set, otherwise defaults to 200 MiB and can be overridden with the AVRO_MAX_DECOMPRESS_LENGTH environment variable; exceeding it throws Avro::DataFile::Error::DecompressionSize. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

This PR hardens the Perl Avro DataFileReader against decompression bombs by enforcing a configurable maximum decompressed block size during block decoding, using either the existing block_max_size attribute or the AVRO_MAX_DECOMPRESS_LENGTH environment variable (default 200 MiB).

Changes:

Add bounded, chunked decompression for deflate and bzip2 blocks, with a new Avro::DataFile::Error::DecompressionSize exception on limit exceed.
Enforce a decompressed-size cap for zstandard blocks (currently checked after decompress).
Add a new test file validating over-limit rejection (deflate) and within-limit decoding.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
lang/perl/lib/Avro/DataFileReader.pm	Adds decompression-size limit enforcement and a new DecompressionSize error while reading compressed blocks.
lang/perl/t/07_datafile_decompress_limit.t	Introduces tests for decompression limit behavior (currently focused on deflate).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iemejia · 2026-07-11T15:40:47Z

+    ## The decompressed size of a block is capped to guard against a block with
+    ## a very high compression ratio expanding to far more memory than its
+    ## compressed size.
+    my $limit = $datafile->block_max_size;
+    $limit = _max_decompress_length() unless defined $limit;
+


Fixed in aa239b8 — the reader no longer reuses block_max_size (a writer-side compressed-block flush threshold) for the decompressed-size cap; the limit is the AVRO_MAX_DECOMPRESS_LENGTH env var / DEFAULT_MAX_DECOMPRESS_LENGTH.

iemejia · 2026-07-11T15:40:49Z

        elsif ($codec eq 'zstandard') {
-            do { open $fh, '<', \(decompress(\$block)); $fh };
+            my $uncompressed = decompress(\$block);
+            _check_decompress_length(length($uncompressed), $limit);
+            do { open my $fh, '<', \$uncompressed; $fh };


Fixed in aa239b8 — zstandard now decompresses via Compress::Zstd::Decompressor in chunks, rejecting an over-large block before its full form is materialized.

iemejia · 2026-07-11T15:40:50Z

+    while (($status = $z->read($chunk, 65536)) > 0) {
+        $uncompressed .= $chunk;
+        _check_decompress_length(length($uncompressed), $limit);
+    }
+    if (!defined $status || $status < 0) {
+        croak "Error decompressing block";
+    }


Fixed in aa239b8 — _inflate_bounded sizes each read to the remaining budget (capped at 64 KiB), so the buffer overshoots the limit by at most one byte, and the croak now includes the decompressor's $z->error.

iemejia · 2026-07-11T15:40:51Z

+package Avro::DataFile::Error::DecompressionSize;
+use parent -norequire, 'Error::Simple';


Fixed in aa239b8 — the DecompressionSize error class now uses use parent 'Error::Simple', consistent with the other error classes.

iemejia · 2026-07-11T15:40:52Z

+## A block within the limit still decodes correctly.
+{
+    my $payload = "hello world";
+    my $fh = deflate_file($payload);
+    my $reader = Avro::DataFileReader->new(
+        fh             => $fh,
+        reader_schema  => $schema,
+        block_max_size => 1024 * 1024,
+    );
+    my @all = $reader->all;
+    is_deeply \@all, [$payload], 'deflate block within the limit decodes';
+}
+
+done_testing;


Fixed in aa239b8 — added over-limit tests for bzip2 and zstandard (skipped when the codec module is unavailable).

…; tests - Stop reusing block_max_size (a writer-side compressed-block flush threshold) for the reader's decompressed-size cap; the limit is now the AVRO_MAX_DECOMPRESS_LENGTH environment variable / DEFAULT_MAX_DECOMPRESS_LENGTH. - Decompress zstandard via Compress::Zstd::Decompressor in chunks so an over-large block is rejected before its full form is materialized. - Tighten _inflate_bounded: size each read to the remaining budget (capped at 64 KiB) so the buffer overshoots the limit by at most one byte, and include the decompressor error in the croak message. - Make the DecompressionSize error class consistent (use parent 'Error::Simple'). - Add over-limit tests for bzip2 and zstandard (skipped when unavailable). Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

iemejia · 2026-07-11T17:08:00Z

 use Carp;
-use Compress::Zstd;
+use Compress::Zstd::Decompressor;
 use IO::Uncompress::Bunzip2 qw(bunzip2);


Fixed. Only the OO IO::Uncompress::Bunzip2->new interface is used, so the module is now imported without the bunzip2 function (use IO::Uncompress::Bunzip2 ();). Pushed in 985cabc.

iemejia · 2026-07-11T17:08:01Z

+    ## The decompressed size of a block is capped to guard against a block with
+    ## a very high compression ratio expanding to far more memory than its
+    ## compressed size. The limit is the AVRO_MAX_DECOMPRESS_LENGTH environment
+    ## variable, or DEFAULT_MAX_DECOMPRESS_LENGTH. (Note: block_max_size is a
+    ## writer-side flush threshold measured in compressed bytes and is not reused
+    ## here to avoid conflating the two units.)
+    my $limit = _max_decompress_length();
+


Updated the PR description: the limit is driven by AVRO_MAX_DECOMPRESS_LENGTH (default 200 MiB) and is independent of block_max_size (which bounds buffered compressed input, not decompressed size). The description now also reflects that deflate, bzip2, and zstandard are all covered.

Only the OO IO::Uncompress::Bunzip2->new interface is used; import the module without the bunzip2 function to avoid an unused import. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

iemejia · 2026-07-11T17:39:06Z

+    while ($offset < $length) {
+        my $piece = substr($$block_ref, $offset, 65536);
+        $offset += 65536;
+        my $out = $decompressor->decompress($piece);
+        next unless defined $out;
+        $uncompressed .= $out;
+        _check_decompress_length(length($uncompressed), $limit);
+    }
+    return $uncompressed;


Addressed. Note this module's Compress::Zstd::Decompressor has no error or flush method: decompress() croaks on a corrupt frame and otherwise emits all output while consuming its input. I changed _zstd_decompress_bounded to fail closed on an undefined return (croak) instead of silently skipping it, and documented the drain-on-consume behavior. Pushed in d924bc2.

iemejia · 2026-07-11T17:39:07Z

+## A block within the limit still decodes correctly.
+{
+    my $payload = "hello world";
+    my $fh = codec_file('deflate', $payload);
+    local $ENV{AVRO_MAX_DECOMPRESS_LENGTH} = 1024 * 1024;
+    my $reader = Avro::DataFileReader->new(
+        fh            => $fh,
+        reader_schema => $schema,
+    );
+    my @all = $reader->all;
+    is_deeply \@all, [$payload], 'deflate block within the limit decodes';
+}
+
+done_testing;


Fixed. The within-limit decode test is now a helper exercised for bzip2 and zstandard as well as deflate (skipped when the codec modules are unavailable), guarding the bounded streaming paths against regressions. Pushed in d924bc2.

…zstd round-trips - _zstd_decompress_bounded now treats an undefined decompress() return as a failure and croaks, instead of silently skipping it, so a malformed block cannot masquerade as a short, within-limit result. (The streaming decompressor emits all output while consuming its input and has no separate flush step, noted in a comment.) - The within-limit decode test is now a helper exercised for bzip2 and zstandard as well as deflate (skipped when the codec modules are unavailable), guarding the bounded streaming paths against regressions. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

iemejia · 2026-07-11T17:59:09Z

+        my $budget = $limit - length($uncompressed) + 1;
+        my $to_read = $budget < 65536 ? $budget : 65536;
+        $status = $z->read($chunk, $to_read);
+        last unless defined $status && $status > 0;
+        $uncompressed .= $chunk;
+        _check_decompress_length(length($uncompressed), $limit);


Fixed. _inflate_bounded now uses bytes::length() for both the read budget and the limit check, so the cap is enforced on bytes regardless of any UTF-8 flag, matching Avro::DataFileWriter's block-size accounting. Pushed in f3d866e.

iemejia · 2026-07-11T17:59:10Z

+        $uncompressed .= $out;
+        _check_decompress_length(length($uncompressed), $limit);


Fixed. _zstd_decompress_bounded also uses bytes::length() when enforcing the limit (and for iterating the compressed block). Pushed in f3d866e.

_inflate_bounded and _zstd_decompress_bounded counted the running output with length(), which returns character length. If the scalar were ever UTF-8 upgraded, that could undercount bytes and let a block exceed the configured byte cap. Use bytes::length() for byte-accurate accounting, matching how Avro::DataFileWriter tracks block sizes. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

iemejia · 2026-07-11T18:19:21Z

+    ## The decompressed size of a block is capped to guard against a block with
+    ## a very high compression ratio expanding to far more memory than its
+    ## compressed size. The limit is the AVRO_MAX_DECOMPRESS_LENGTH environment
+    ## variable, or DEFAULT_MAX_DECOMPRESS_LENGTH. (Note: block_max_size is a
+    ## writer-side flush threshold measured in compressed bytes and is not reused
+    ## here to avoid conflating the two units.)
+    my $limit = _max_decompress_length();


Addressed. When the reader is configured with block_max_size, a block whose declared compressed size exceeds it is now rejected before the compressed block is read into memory, guarding against an attacker-controlled block_size allocation. (Without block_max_size configured, the read still relies on the file's actual length; the decompressed-size cap remains the guard against expansion.) Added a test. Pushed in cbba52a.

iemejia · 2026-07-11T18:19:22Z

+            my $z = IO::Uncompress::RawInflate->new(\$block)
+                or croak "Error inflating block: $IO::Uncompress::RawInflate::RawInflateError";
+            my $uncompressed = _inflate_bounded($z, $limit);
+            do { open my $fh, '<', \$uncompressed; $fh };


Fixed. The three in-memory open sites now go through _open_decompressed(), which croaks with $! on failure instead of leaving reader undef. Pushed in cbba52a.

iemejia · 2026-07-11T18:19:23Z

+            my $z = IO::Uncompress::Bunzip2->new(\$block)
+                or croak "Error decompressing bzip2 block: $IO::Uncompress::Bunzip2::Bunzip2Error";
+            my $uncompressed = _inflate_bounded($z, $limit);
+            do { open my $fh, '<', \$uncompressed; $fh };


Fixed via the shared _open_decompressed() helper, which checks open and croaks with $!. Pushed in cbba52a.

iemejia · 2026-07-11T18:19:24Z

        elsif ($codec eq 'zstandard') {
-            do { open $fh, '<', \(decompress(\$block)); $fh };
+            my $uncompressed = _zstd_decompress_bounded(\$block, $limit);
+            do { open my $fh, '<', \$uncompressed; $fh };


Fixed via the shared _open_decompressed() helper, which checks open and croaks with $!. Pushed in cbba52a.

- Guard against an attacker-controlled block_size causing a huge allocation for the compressed block itself: when the reader is configured with block_max_size, reject a block whose declared compressed size exceeds it before reading the block into memory. Added a test. - The in-memory read handles over the decompressed block were opened without checking for failure, leaving reader undef and surfacing unclear errors later. Factor the three open sites into _open_decompressed(), which croaks with $! on failure. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

iemejia · 2026-07-11T18:32:58Z

+    eval { require Compress::Zstd; 1 }
+        or skip 'Compress::Zstd not available', 1;


Fixed. The SKIP guard now requires Compress::Zstd::Decompressor (the exact module the reader uses) instead of Compress::Zstd, so the test is reliably skipped when that dependency is missing. Pushed in ccab12c.

iemejia · 2026-07-11T18:32:59Z

+    eval { require Compress::Zstd; 1 }
+        or skip 'Compress::Zstd not available', 1;


Fixed the same way for the within-limit path: it now checks for Compress::Zstd::Decompressor. Pushed in ccab12c.

The zstandard SKIP guards checked for Compress::Zstd, but the reader depends on Compress::Zstd::Decompressor. Require that submodule in the SKIP conditions so the tests are reliably skipped when the reader-side dependency is missing rather than attempting to run and failing later. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

iemejia · 2026-07-11T18:46:37Z

+use Compress::Zstd::Decompressor;
+use IO::Uncompress::Bunzip2 ();
 use IO::Uncompress::RawInflate ;


Fixed via option (2): Compress::Zstd::Decompressor is now loaded lazily inside the zstandard decompression path rather than with a compile-time use. The reader loads and works for the other codecs even when the Decompressor submodule is unavailable, and reading a zstandard block in that case throws Avro::DataFile::Error::UnsupportedCodec with a clear message. Pushed in 5b1ba03.

The reader hard-required Compress::Zstd::Decompressor at compile time, so an environment with an older Compress::Zstd lacking the Decompressor submodule would fail to load the reader entirely, even when only other codecs are used. Load Compress::Zstd::Decompressor lazily in the zstandard path and throw Avro::DataFile::Error::UnsupportedCodec with a clear message when it is unavailable, so the reader keeps working for the other codecs. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

iemejia · 2026-07-13T01:36:05Z

+    my $want = $datafile->{block_size} + MARKER_SIZE;
+    my $block;
+    my $nread = read $fh, $block, $want;
+    if (!defined $nread) {
+        croak "Error reading from file: $!";
+    }
+    if ($nread != $want) {
+        croak "Short read: expected $want bytes for the block, got $nread (truncated file?)";
+    }


Fixed — the block is now read in bounded 64 KiB chunks until $want or EOF (rather than one read that pre-extends the buffer to the declared block_size), then the exact byte count is verified, so a malformed huge block_size with little data fails after a bounded allocation.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

read($fh, $block, $want) pre-extends the buffer to the attacker-controlled $want (block_size + marker) before a short read is detected, so a malformed file declaring an enormous block_size but carrying little data forces a huge allocation. Read in bounded 64 KiB chunks until $want or EOF, then verify the exact byte count, so a truncated file fails after a bounded allocation. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

iemejia · 2026-07-13T04:41:19Z

+    while (length($block) < $want) {
+        my $need = $want - length($block);
+        $need = $chunk_size if $need > $chunk_size;
+        my $nread = read $fh, my $buf, $need;
+        if (!defined $nread) {
+            croak "Error reading from file: $!";
+        }
+        last if $nread == 0;    # EOF
+        $block .= $buf;
+    }
+    if (length($block) != $want) {
+        croak "Short read: expected $want bytes for the block, got "
+            . length($block) . " (truncated file?)";
+    }


Fixed — the bounded block-read loop and short-read check now use bytes::length, so a UTF-8-flagged scalar can't miscount bytes.

iemejia · 2026-07-13T04:41:21Z

+        my $out = $decompressor->decompress($piece);
+        # The streaming decompressor croaks on a corrupt frame and otherwise
+        # emits all output produced while consuming the input it is given (there
+        # is no separate flush step in this API). Treat an undefined return as a
+        # failure and fail closed, rather than silently skipping it, so a
+        # malformed block cannot masquerade as a short, within-limit result.
+        unless (defined $out) {
+            croak "Error decompressing zstandard block";
+        }
+        # Check the prospective total before growing $uncompressed so a single
+        # large decompressed chunk cannot transiently balloon memory past the
+        # limit (or double peak memory from string reallocation).
+        _check_decompress_length(
+            bytes::length($uncompressed) + bytes::length($out), $limit);
+        $uncompressed .= $out;


Mitigated — Compress::Zstd's streaming decompress() has no max-output parameter, so the smallest guarantee is one call's output. The input is now fed in 16 KiB pieces (down from 64 KiB) to bound each call's transient $out (~one internal block), and the running total is still checked before concatenating into $uncompressed, so peak stays at uncompressed + one bounded piece.

- The bounded block-read loop and short-read check now use bytes::length instead of length, so a UTF-8-flagged scalar (e.g. from an IO layer) cannot miscount the accumulated bytes and mis-handle the marker boundary. - Feed the zstd streaming decompressor 16 KiB input pieces instead of 64 KiB. Compress::Zstd's decompress() has no max-output parameter, so a single call materializes all output from the input it is handed; smaller pieces bound the transient $out (roughly one internal block) so a highly-compressible frame cannot balloon one call's output before the size check runs. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

iemejia · 2026-07-13T05:05:41Z

+    unless (eval { require Compress::Zstd::Decompressor; 1 }) {
+        Avro::DataFile::Error::UnsupportedCodec->throw(
+            "Cannot read zstandard-compressed block: Compress::Zstd::Decompressor is not available: $@"
+        );
+    }


Fixed — the require now runs under local $@ and the error is captured into a lexical ($require_error) before building the message, so the caller's $@ is preserved.

The lazy `require Compress::Zstd::Decompressor` ran inside eval without localizing $@, so a successful probe would clear (or a failed one overwrite) the caller's $@. Localize $@ around the eval and capture the require error into a lexical before building the UnsupportedCodec message. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

github-actions Bot added the Perl label Jul 11, 2026

iemejia requested a review from Copilot July 11, 2026 14:48

Copilot started reviewing on behalf of iemejia July 11, 2026 14:48 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 16:36

Copilot started reviewing on behalf of iemejia July 11, 2026 16:36 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

AVRO-4288: [perl] Drop unused bunzip2 import

985cabc

Only the OO IO::Uncompress::Bunzip2->new interface is used; import the module without the bunzip2 function to avoid an unused import. Assisted-by: GitHub Copilot:claude-opus-4.8

iemejia requested a review from Copilot July 11, 2026 17:14

Copilot started reviewing on behalf of iemejia July 11, 2026 17:14 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 17:49

Copilot started reviewing on behalf of iemejia July 11, 2026 17:49 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 18:06

Copilot started reviewing on behalf of iemejia July 11, 2026 18:06 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 18:26

Copilot started reviewing on behalf of iemejia July 11, 2026 18:27 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 18:40

Copilot started reviewing on behalf of iemejia July 11, 2026 18:40 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 13, 2026 00:44

Copilot started reviewing on behalf of iemejia July 13, 2026 00:45 View session