AVRO-4294: [c++] Bound allocation when decoding length-prefixed values and collections#3859
AVRO-4294: [c++] Bound allocation when decoding length-prefixed values and collections#3859iemejia wants to merge 21 commits into
Conversation
…h-prefixed values and collections A bytes or string value is a length prefix followed by that many bytes, and an array or map block is an element count followed by that many items. A malicious or truncated input can declare a huge length or count with little or no data, causing a correspondingly huge allocation before the shortfall is noticed. - Add remainingBytes() to InputStream (default -1, 'unknown'), overridden by the memory-backed input streams, and exposed via StreamReader. Add bytesRemaining() to the Decoder interface (default -1), overridden by BinaryDecoder and forwarded by the validating and resolving decorators. - decodeString/decodeBytes reject a declared length that exceeds the bytes available before resizing. - GenericReader rejects an array/map block whose element count could not be backed by the bytes remaining, using minBytesPerElement() from the element schema so a zero-byte element (e.g. null) is not falsely rejected. The check is applied only when not resolving, since under resolution the datum (reader) type may be larger than the wire (writer) type and would over-estimate. Mirrors the Java SDK's checks (AVRO-4241). Assisted-by: GitHub Copilot:claude-opus-4.8
There was a problem hiding this comment.
Pull request overview
Hardens the C++ Avro binary decoding path against malicious/truncated inputs that declare large length-prefixed values or collection block counts, by validating declared sizes against the underlying stream’s known remaining bytes before performing large allocations.
Changes:
- Add
remainingBytes()toInputStream/StreamReaderandbytesRemaining()to theDecoderinterface, with forwarding through validating/resolving decoders. - Enforce available-bytes checks in
BinaryDecoder::decodeString/decodeBytesand inGenericReaderwhen resizing arrays/maps (when remaining size is known and schema is not resolving). - Add
AvailableBytesTestsand wire it into the C++ test CMake target.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| lang/c++/test/AvailableBytesTests.cc | New unit tests covering oversized/within-limit lengths and collection counts, including null-element edge case. |
| lang/c++/include/avro/Stream.hh | Introduces InputStream::remainingBytes() and StreamReader::remainingBytes() aggregation. |
| lang/c++/include/avro/Decoder.hh | Introduces Decoder::bytesRemaining() defaulting to unknown (-1). |
| lang/c++/impl/Stream.cc | Implements remainingBytes() for memory-backed input streams. |
| lang/c++/impl/parsing/ValidatingCodec.cc | Forwards bytesRemaining() through validating decoder wrapper. |
| lang/c++/impl/parsing/ResolvingDecoder.cc | Forwards bytesRemaining() through resolving decoder wrapper. |
| lang/c++/impl/Generic.cc | Adds collection block count vs remaining-bytes validation before resizing containers. |
| lang/c++/impl/BinaryDecoder.cc | Adds remaining-bytes enforcement for string/bytes length-prefixed allocations. |
| lang/c++/CMakeLists.txt | Registers the new AvailableBytesTests unit test. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| int64_t remainingBytes() const final { | ||
| // Total capacity across all chunks: full chunks plus the (partial) | ||
| // last chunk, minus what has already been consumed. | ||
| int64_t total = (size_ == 0) | ||
| ? 0 | ||
| : static_cast<int64_t>((size_ - 1) * chunkSize_ + available_); | ||
| return total - static_cast<int64_t>(cur_ * chunkSize_ + curLen_); | ||
| } |
There was a problem hiding this comment.
Fixed in 0f223c5: remainingBytes() now widens each operand to int64_t before multiplying, so (size_-1)*chunkSize_ can no longer overflow size_t before the cast.
| for (size_t m = d.arrayStart(); m != 0; m = d.arrayNext()) { | ||
| ensureCollectionAvailable(d, m, minBytes); | ||
| r.resize(r.size() + m); |
There was a problem hiding this comment.
Fixed in 0f223c5: array decoding now calls ensureCanGrow(r, m), a max_size()-based guard, before r.resize(r.size() + m), so a huge block count cannot overflow size_t or request an impossible allocation (this also covers null-element arrays where the byte check is skipped).
There was a problem hiding this comment.
I am extremely suspicious of the claim that ensureCanGrow can ever fail. Is there some platform I am not considering where max_size limits would be exceeded before RAM is exhausted?
C++ reference implies this won't ever happen, https://en.cppreference.com/cpp/container/vector/max_size
There was a problem hiding this comment.
It is nearly always unreachable now that the structural cap (<= 2147483639) is always applied, so I kept it only as a cheap guard for the edge where max_size() is smaller than that cap (32-bit builds, or very large element types), where resize() would otherwise throw a raw std::length_error instead of a clean Avro Exception. Happy to drop it if you'd prefer to rely on resize()'s own throw.
| for (size_t m = d.mapStart(); m != 0; m = d.mapNext()) { | ||
| ensureCollectionAvailable(d, m, minBytes); | ||
| r.resize(r.size() + m); |
There was a problem hiding this comment.
Fixed in 0f223c5: the same max_size()-based ensureCanGrow(r, m) guard is applied before resizing the map container.
… keep check on deep schemas Review feedback: - MemoryInputStream::remainingBytes() computed (size_-1)*chunkSize_ in size_t before the int64_t cast, which could overflow on large buffers and yield a wrong (possibly negative) remaining count. Widen each operand to int64_t before multiplying. - GenericReader's array/map decoding now guards r.resize(r.size() + m) with a max_size()-based check, so a very large block count cannot overflow size_t or request an impossible allocation (this also covers null-element collections, where the per-element minimum is zero and the byte check is skipped). - minBytesPerElement() returns 1 (not 0) when the depth guard trips, so the collection check stays enabled for a crafted deep/recursive schema. Assisted-by: GitHub Copilot:claude-opus-4.8
| switch (node->type()) { | ||
| case AVRO_NULL: | ||
| return 0; | ||
| case AVRO_FLOAT: | ||
| return 4; | ||
| case AVRO_DOUBLE: | ||
| return 8; | ||
| case AVRO_FIXED: | ||
| return static_cast<int64_t>(node->fixedSize()); | ||
| case AVRO_RECORD: { | ||
| int64_t total = 0; | ||
| for (size_t i = 0; i < node->leaves(); ++i) { | ||
| total += minBytesPerElement(node->leafAt(i), depth + 1); | ||
| } | ||
| return total; | ||
| } | ||
| default: | ||
| // boolean, int, long, bytes, string, enum, union, array, map and | ||
| // symbolic references all encode to at least one byte. | ||
| return 1; | ||
| } |
There was a problem hiding this comment.
Fixed in 711e27b: minBytesPerElement() clamps the fixed size to INT64_MAX and saturates the record field-minima sum, so an overflow can no longer wrap negative and weaken the guard.
| // Map keys are strings (>= 1 byte length prefix) plus the value. | ||
| int64_t minBytes = isResolving ? 0 : (1 + minBytesPerElement(nn, 0)); |
There was a problem hiding this comment.
Fixed in 711e27b: the map's (1 + valueMin) is now saturated (only adds 1 when valueMin < INT64_MAX), so it cannot overflow to a negative minBytes.
Review feedback: minBytesPerElement() could overflow int64_t for a very large fixed size or when summing record field minima, and the map's (1 + valueMin) could wrap. A wrapped negative minimum would disable ensureCollectionAvailable (minBytes <= 0) and bypass the guard. Clamp the fixed size, saturate the record sum, and saturate the map key +1, all to std::numeric_limits<int64_t>::max(). Assisted-by: GitHub Copilot:claude-opus-4.8
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
lang/c++/impl/BinaryDecoder.cc:141
- decodeString writes into std::string storage by const_cast'ing value.c_str(), but the standard specifies the buffer returned by c_str() must not be modified. Since this C++ code is built with C++20, use value.data() (mutable) or &value[0] after resize() to avoid undefined behavior.
if (len > 0) {
in_.readBytes(const_cast<uint8_t *>(
reinterpret_cast<const uint8_t *>(value.c_str())),
len);
}
| #include <cstdint> | ||
| #include <string> | ||
| #include <vector> | ||
|
|
There was a problem hiding this comment.
Fixed in 4d983f8: AvailableBytesTests.cc now includes directly instead of relying on an indirect include.
Review feedback: the test uses std::unique_ptr but relied on an indirect include (e.g. via Boost headers), which is non-portable. Include <memory> directly. Assisted-by: GitHub Copilot:claude-opus-4.8
| if (!node) { | ||
| return 0; | ||
| } | ||
| if (depth > 64) { | ||
| // A cyclic or pathologically deep schema. Return 1 (not 0) so the | ||
| // collection check stays enabled rather than being silently bypassed; | ||
| // a valid recursive value always encodes to at least 1 byte. | ||
| return 1; | ||
| } |
There was a problem hiding this comment.
Good catch. Moved the depth guard inside the AVRO_RECORD case so a zero-byte leaf type (null) nested under deeply recursive records still returns 0, while a cyclic/pathologically deep record returns 1 (keeping the collection check enabled). Fixed in 3f6569d.
…ns 0 Review feedback: the depth cutoff was applied before checking the node type, so a zero-byte leaf type (e.g. null) nested under deeply chained records returned 1 instead of 0, enabling the collection check and potentially rejecting valid data. Move the depth guard into the AVRO_RECORD case; leaf types now return their true minimum (null -> 0) regardless of nesting depth. Assisted-by: GitHub Copilot:claude-opus-4.8
| if (depth > 64) { | ||
| // A cyclic or pathologically deep record. Return 1 (not 0) so the | ||
| // collection check stays enabled rather than being silently | ||
| // bypassed; a valid recursive value always encodes to >= 1 byte. | ||
| // (The depth guard is applied only here, so zero-byte leaf types |
There was a problem hiding this comment.
Good point. Fixed in bf25d41: the depth guard now returns 0 (a valid conservative lower bound) instead of 1. A truly cyclic schema never reaches the guard because a self-reference is an AVRO_SYMBOLIC node handled by the default case (returning 1 without following the link), so recursion always terminates; the guard is purely a stack-overflow safety net for a genuinely deep acyclic record, where returning 1 would over-estimate and could falsely reject a valid array/map of null-leaf records. Added testReadArrayOfDeeplyNestedNullNotFalselyRejected with a ~70-level nested null-leaf record.
…th guard Review feedback: minBytesPerElement() returned 1 when the depth guard tripped, which over-estimates the true lower bound for a genuinely deep but acyclic schema whose leaves are all null (true minimum 0), and could falsely reject a valid array/map of such elements. A truly cyclic schema never reaches the guard: a self-reference is an AVRO_SYMBOLIC node handled by the default case (returning 1 without following the link), so recursion always terminates. The guard is therefore only a stack-overflow safety net for a pathologically deep acyclic record, where returning 0 (always a valid lower bound) avoids the false rejection. Added a test with a ~70-level nested null-leaf record. Assisted-by: GitHub Copilot:claude-opus-4.8
| int64_t remainingBytes() const { | ||
| if (in_ == nullptr) { | ||
| return -1; | ||
| } | ||
| int64_t streamRemaining = in_->remainingBytes(); | ||
| if (streamRemaining < 0) { | ||
| return -1; | ||
| } | ||
| // end_ - next_ is the data already buffered in this reader; it is added | ||
| // to what the underlying stream still has. The addition promotes the | ||
| // pointer difference to int64_t. | ||
| return (end_ - next_) + streamRemaining; | ||
| } |
There was a problem hiding this comment.
Good catch. Fixed in 83b3b71: the buffered-byte count is now computed as (next_ != nullptr) ? (end_ - next_) : 0, so the pointer subtraction is skipped when next_/end_ are both null (right after init()/reset(), before any fill). Added testBytesRemainingRightAfterInit that calls bytesRemaining() immediately after init() on a BinaryDecoder.
…iningBytes Review feedback: after init()/reset() and before any data is buffered, next_ and end_ are both null, and computing end_ - next_ is undefined behavior. A caller triggers this by calling Decoder::bytesRemaining() immediately after init() on a BinaryDecoder. Guard the buffered-byte count so the subtraction is only performed when next_ is non-null. Added a test that calls bytesRemaining() right after init(). Assisted-by: GitHub Copilot:claude-opus-4.8
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
lang/c++/impl/BinaryDecoder.cc:141
- decodeString() writes into std::string via const_cast on value.c_str(). Modifying the buffer returned by c_str() is undefined behavior; use a writable pointer (e.g., &value[0] after resize) instead.
if (len > 0) {
in_.readBytes(const_cast<uint8_t *>(
reinterpret_cast<const uint8_t *>(value.c_str())),
len);
}
Completes the available-bytes protection for collections and supersedes the
separate collection-limit change. Elements whose schema encodes to zero bytes
(null, a zero-length fixed, or a record with only zero-byte fields) consume no
input, so ensureCollectionAvailable cannot bound their count. A tiny payload
declaring a huge array block count of such elements (e.g.
{"type":"array","items":"null"} with a count of 200,000,000) therefore drove a
single huge vector resize and exhausted memory.
ensureCollectionAvailable now enforces the bytes-remaining check plus a
structural cap on all collections (kDefaultMaxCollectionStructural =
Integer.MAX_VALUE - 8), and ensureZeroByteCollectionWithinLimit caps zero-byte
elements at kDefaultMaxCollectionItems = 10,000,000; the true element minimum is
computed independently of the resolving flag (which forces minBytes to 0).
AVRO_MAX_COLLECTION_ITEMS, when set, caps both. BinaryDecoder::skipArray is also
bounded by the structural cap so skipping a huge block cannot loop unboundedly.
Assisted-by: GitHub Copilot:claude-opus-4.8
| static CollectionLimits collectionLimits() { | ||
| const char *env = std::getenv("AVRO_MAX_COLLECTION_ITEMS"); | ||
| if (env != nullptr && *env != '\0') { | ||
| char *end = nullptr; | ||
| long long value = std::strtoll(env, &end, 10); | ||
| if (*end == '\0' && value >= 0) { | ||
| return {static_cast<int64_t>(value), static_cast<int64_t>(value)}; | ||
| } | ||
| } | ||
| return {kDefaultMaxCollectionItems, kDefaultMaxCollectionStructural}; |
There was a problem hiding this comment.
Fixed — collectionLimits() now checks errno after strtoll and ignores an overflowing (ERANGE) value, falling back to the defaults instead of accepting a saturated LLONG_MAX.
There was a problem hiding this comment.
Already handled — collectionLimits() resets errno before strtoll and only accepts the value when errno == 0, so an ERANGE overflow (saturated LLONG_MAX) is ignored and the defaults apply (Generic.cc:122-127).
| #include <cstdint> | ||
| #include <cstdlib> | ||
| #include <limits> | ||
| #include <memory> | ||
|
|
||
| namespace avro { | ||
|
|
||
| using std::make_shared; | ||
|
|
||
| // Structural cap on the number of elements to skip in an array or map (an | ||
| // overflow / defense-in-depth guard). Mirrors the read-path limit in Generic.cc | ||
| // and matches the value used by the other Avro SDKs (Integer.MAX_VALUE - 8). | ||
| static constexpr int64_t kDefaultMaxCollectionStructural = 2147483639; | ||
|
|
||
| // Returns the structural collection cap. It can be overridden by the | ||
| // AVRO_MAX_COLLECTION_ITEMS environment variable (a non-negative integer), | ||
| // matching the read path and the other SDKs so a single knob configures all of | ||
| // them; an invalid or unset value uses the default. | ||
| static int64_t maxCollectionStructural() { | ||
| const char *env = std::getenv("AVRO_MAX_COLLECTION_ITEMS"); | ||
| if (env != nullptr && *env != '\0') { | ||
| char *end = nullptr; | ||
| long long value = std::strtoll(env, &end, 10); | ||
| if (*end == '\0' && value >= 0) { | ||
| return static_cast<int64_t>(value); | ||
| } | ||
| } | ||
| return kDefaultMaxCollectionStructural; | ||
| } |
There was a problem hiding this comment.
Fixed — maxCollectionStructural() applies the same errno/ERANGE check so an overflowing env value can't disable the structural cap.
There was a problem hiding this comment.
Already handled — maxCollectionStructural() has the same errno == 0 guard, so an overflowing AVRO_MAX_COLLECTION_ITEMS falls back to the default structural cap (BinaryDecoder.cc).
collectionLimits() and maxCollectionStructural() parsed the env var with std::strtoll without checking errno, so an overflowing value (e.g. many digits) saturated to LLONG_MAX and was accepted, effectively disabling the structural/ zero-byte caps. Check errno == ERANGE (via <cerrno>) and fall back to the defaults on overflow. Assisted-by: GitHub Copilot:claude-opus-4.8
| void selectBranch(size_t branch) { | ||
| if (branch >= schema()->leaves()) { | ||
| throw Exception("Union branch index out of range: must be less than " + std::to_string(schema()->leaves()) + ", but is " + std::to_string(branch)); | ||
| } |
There was a problem hiding this comment.
This has been my first real experience looking at reviews from AI... and wow, this thing has been so confidently wrong (or at least just noise) in almost every instance. leaves() gives a size_t, this comment makes absolutely no sense. There's absolutely no need to check for negative values against an unsigned type...
There was a problem hiding this comment.
Fixed — decodeUnionIndex now validates the decoded long via decodeIndex(): a negative index is rejected before the size_t cast (so it can't wrap to a huge value), and selectBranch's bound against leaves() still applies.
| // Bytes already buffered in this reader, added to what the underlying | ||
| // stream still has. When next_ and end_ are both null (right after | ||
| // init()/reset(), before any data is buffered), the subtraction is | ||
| // well-defined and yields 0. | ||
| int64_t buffered = end_ - next_; | ||
| return buffered + streamRemaining; |
There was a problem hiding this comment.
As noted previously: this guard was deliberately removed at a maintainer's request to match the rest of Stream.hh. next_ and end_ are always assigned together (both null immediately after init()/reset(), both valid once data is buffered), so the null case yields 0 in practice on every supported platform. Deferring to that maintainer decision; the comment documents the null case.
There was a problem hiding this comment.
Left as-is intentionally: an earlier review round had a maintainer remove an equivalent null-guard here, so re-adding it would conflict with that decision. In practice remainingBytes() is only meaningful after init()/reset() has pointed next_/end_ into a real buffer; right after reset() with both null the value isn't consumed. If the project wants a formal guard I'll defer to a maintainer call rather than reintroduce the reverted change.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
lang/c++/impl/BinaryDecoder.cc:169
- decodeString() writes into the string buffer via value.c_str() + const_cast, which is undefined behavior (c_str() returns a const pointer and is not required to be writable). Since this project requires C++20, std::string::data() returns a mutable pointer and should be used for reading into the resized buffer.
if (len > 0) {
in_.readBytes(const_cast<uint8_t *>(
reinterpret_cast<const uint8_t *>(value.c_str())),
len);
}
| int64_t remainingBytes() const final { | ||
| return static_cast<int64_t>(size_ - curLen_); | ||
| } |
There was a problem hiding this comment.
Fixed — MemoryInputStream2::remainingBytes() now computes (int64_t)size_ - (int64_t)curLen_, so a violated invariant can't underflow an unsigned subtraction.
| /// Returns the number of bytes that remain to be read from the underlying | ||
| /// stream, or a negative value when that count is not known (for example a | ||
| /// streaming source, or a decoder not backed by a byte stream). The default | ||
| /// is "unknown"; byte-stream decoders override it so a length prefix or a | ||
| /// collection block count that exceeds the data actually available can be | ||
| /// rejected before allocating for it. | ||
| virtual int64_t bytesRemaining() const { return -1; } |
There was a problem hiding this comment.
Valid concern. Adding virtual remainingBytes()/bytesRemaining() to InputStream/Decoder changes vtable layout and would break ABI for downstream subclasses across a shared-library boundary. This needs a maintainer decision on the intended compatibility guarantee: (a) accept it for the next minor/major (the C++ SDK has historically bumped SONAME on minor releases), or (b) avoid the vtable change by making these non-virtual with a dynamic_cast to the concrete decoder/stream where the capability exists. I've left the virtual approach in place pending that call rather than pick a compatibility policy unilaterally — happy to switch to the non-virtual + dynamic_cast form if that's preferred.
There was a problem hiding this comment.
Decision: keep the virtual approach and accept the ABI bump for the next release. The C++ shared library's VERSION is derived from AVRO_VERSION (share/VERSION.txt) in CMakeLists.txt (set_target_properties ... VERSION ${AVRO_VERSION}), so the library version already changes with each release and downstream consumers relink against the new version. No separate SOVERSION pin needs bumping. The new virtual remainingBytes()/bytesRemaining() are therefore acceptable here.
| void selectBranch(size_t branch) { | ||
| if (branch >= schema()->leaves()) { | ||
| throw Exception("Union branch index out of range: must be less than " + std::to_string(schema()->leaves()) + ", but is " + std::to_string(branch)); | ||
| } |
There was a problem hiding this comment.
Fixed — decodeIndex() also rejects an index beyond std::numeric_limits<size_t>::max(), so on a 32-bit build a large on-wire union index (e.g. 2^32) can't truncate into a small in-range value; it's rejected before reaching selectBranch.
…g subtraction - decodeUnionIndex/decodeEnum now go through a shared decodeIndex() helper that rejects a negative decoded long (which would wrap to a huge size_t) and a value beyond size_t (which could truncate into a spuriously in-range branch on builds where size_t is narrower than int64_t) before casting. The concrete bound against the enum/union size is still enforced by the caller. - MemoryInputStream2::remainingBytes() subtracts in int64_t so a violated invariant (curLen_ > size_) cannot underflow an unsigned subtraction into a huge positive remaining count. Assisted-by: GitHub Copilot:claude-opus-4.8
| // init()/reset(), before any data is buffered), the subtraction is | ||
| // well-defined and yields 0. | ||
| int64_t buffered = end_ - next_; | ||
| return buffered + streamRemaining; | ||
| } |
There was a problem hiding this comment.
Fixed — reconsidered: you're right that subtracting two null pointers is UB even though it yields 0 in practice. remainingBytes() now computes buffered = (next_ == end_) ? 0 : (end_ - next_), so the both-null (post-reset) case returns 0 via a well-defined pointer equality check and the subtraction only runs on real buffer pointers. Updated the misleading comment too.
…iningBytes end_ and next_ can both be null right after init()/reset(); subtracting two null pointers is undefined behavior (the previous comment wrongly claimed it was well-defined). Guard the equal-pointer case (which covers both-null) and only subtract real buffer pointers. Pointer equality, unlike subtraction, is well-defined for null pointers. Assisted-by: GitHub Copilot:claude-opus-4.8
| // Bytes already buffered in this reader, added to what the underlying | ||
| // stream still has. next_ and end_ can both be null right after | ||
| // init()/reset() (before any data is buffered); subtracting two null | ||
| // pointers is undefined behavior, so treat equal pointers (including the | ||
| // both-null case) as zero buffered and only subtract real buffer | ||
| // pointers. Pointer equality, unlike subtraction, is well-defined here. | ||
| int64_t buffered = (next_ == end_) ? 0 : (end_ - next_); | ||
| return buffered + streamRemaining; |
There was a problem hiding this comment.
Fixed — the guard is now (next_ == nullptr || end_ == nullptr) ? 0 : (end_ - next_), so any null operand (not just the both-null case) is treated as zero buffered and the subtraction only runs when both pointers reference a real buffer.
Harden StreamReader::remainingBytes() further: if either next_ or end_ is null (post-reset or a partial fill), evaluating end_ - next_ is still UB. Guard on either pointer being null and only subtract when both point into a real buffer. Assisted-by: GitHub Copilot:claude-opus-4.8
| void BinaryDecoder::skipFixed(size_t n) { | ||
| in_.skipBytes(n); | ||
| } |
What is the purpose of the change
A
bytesorstringvalue is encoded as a length prefix followed by that many bytes of data, and anarrayormapblock is encoded as an element count followed by that many items. A malicious or truncated input can declare a very large length or count while carrying little or no actual data, which causes a correspondingly large allocation before the shortfall is noticed.This applies the equivalent of the Java SDK fix AVRO-4241 to the C++ SDK and extends it to collections. It has two complementary parts.
1. Validate available bytes before allocating
When the source can report how many bytes remain, a declared length (or a collection block count) that exceeds the bytes actually available is rejected before allocating for it. The collection check uses the minimum on-wire size of the element schema, so a zero-byte element type (such as
null) is never falsely rejected. Sources that cannot report their remaining size are unaffected.Adds
remainingBytes()toInputStream(default -1) overridden by the memory input streams and exposed viaStreamReader, andbytesRemaining()to theDecoderinterface, overridden byBinaryDecoderand forwarded by the validating and resolving decoders.decodeString/decodeBytesandGenericReader's array/map decoding consult it. The available-bytes collection check is skipped for arrays under schema resolution, where the datum (reader) type may be larger than the wire (writer) type; maps keep a minimum of 1 byte (the key), so their bytes-remaining check still applies.2. Cap collection allocation for zero-byte elements
Zero-byte elements (
null, a zero-lengthfixed, or a record with only zero-byte fields) consume no input, so the available-bytes check cannot bound their count: a tiny payload such as{{"type":"array","items":"null"}}declaring a block count of 200,000,000 would otherwise drive an unbounded allocation. In addition to the available-bytes check,ensureCollectionAvailableapplies a structural cap to every collection (kDefaultMaxCollectionStructural=Integer.MAX_VALUE - 8), andensureZeroByteCollectionWithinLimitcaps zero-byte elements atkDefaultMaxCollectionItems= 10,000,000; the true per-element minimum is computed byminBytesPerElement()independently of the resolving flag.BinaryDecoder::skipArrayis also bounded by the structural cap so skipping a huge block cannot loop unboundedly. When set, theAVRO_MAX_COLLECTION_ITEMSenvironment variable caps both limits.This folds in and supersedes the standalone collection-limit change (AVRO-4278, #3849), so this PR is the single complete fix for collection/length-prefixed allocation DoS in the C++ SDK.
This is a sub-task of AVRO-4292 and resolves AVRO-4294.
Verifying this change
This change added tests and can be verified as follows:
lang/c++/test/AvailableBytesTests.cccovering over-limitstring/bytes/array/maprejection, within-limit values, anarray<null>with a huge block count that must be rejected without allocating, andskipArraybounding, plus two INT64_MIN block-count cases added totest/CodecTests.cc.cd lang/c++ && mkdir build && cd build && cmake .. && make && ctestDocumentation