[jira] [Resolved] (AVRO-4300) [java] Bound array/map allocation and skipping when decoding on both the classic and fast readers (zero-byte elements + missing fast-reader available-bytes check)

["Ismaël Mejía (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AVRO-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ismaël Mejía resolved AVRO-4300.
--------------------------------
    Resolution: Fixed

> [java] Bound array/map allocation and skipping when decoding on both the classic and fast readers (zero-byte elements + missing fast-reader available-bytes check)
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: AVRO-4300
>                 URL: https://issues.apache.org/jira/browse/AVRO-4300
>             Project: Apache Avro
>          Issue Type: Sub-task
>          Components: java
>    Affects Versions: 1.11.5, 1.12.1
>            Reporter: Ismaël Mejía
>            Assignee: Ismaël Mejía
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.12.2
>
>          Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> When GenericDatumReader decodes an array it reads the block item count from the stream and pre-allocates the backing store (newArray -> new Object[count]) before decoding any element. Several independent gaps let a tiny payload drive an unbounded allocation or an unbounded skip loop and exhaust the heap:
> 1. Zero-byte elements (classic and fast reader).
> Elements whose schema encodes to zero bytes (null, a zero-length fixed, or a record with only zero-byte fields) consume no input, so ensureAvailableCollectionBytes (AVRO-4241) skips the check for them (minBytesPerElement == 0). The collection-length cap is Integer.MAX_VALUE-8, a JVM array-size ceiling rather than a memory budget. So an array such as {"type":"array","items":"null"} declaring a block count of 200,000,000 is a ~6 byte payload that allocates a 200M-slot array (~1.6 GB).
> 2. The fast reader never had the AVRO-4241 available-bytes guard at all.
> AVRO-4241 only modified the classic GenericDatumReader (and BinaryDecoder / Decoder / ValidatingDecoder); it never touched FastReaderBuilder, which is the default decode path (avro.io.fastread defaults to true). As a result, on the default reader even a non-zero-byte array such as array<long> or array<int> with a huge block count and no data pre-allocated new GenericData.Array<>((int) count) and exhausted the heap.
> 3. The skip path was unbounded.
> BinaryDecoder.skipArray()/skipMap() returned doSkipItems() without applying the collection cap, and GenericDatumReader.skip() looped over that count, so skipping a huge block of zero-byte elements (e.g. a writer array<null> field absent from the reader schema, skipped during projection) could loop unboundedly even though skipping allocates nothing.
> Fix (applied identically on the classic and fast reader paths):
> - Add SystemLimitException.checkMaxCollectionAllocation, a heap-aware cumulative cap for zero-byte elements (default maxMemory()/4/8 elements, overridable via the org.apache.avro.limits.collectionItems.maxAllocation system property, mirroring the existing decompression limit).
> - Expose GenericDatumReader.ensureAvailableCollectionBytes and apply it, together with the zero-byte cap, before allocating each array block in FastReaderBuilder, so the fast and classic readers enforce identical guards.
> - Bound the skip path: BinaryDecoder.skipArray()/skipMap() now apply the structural collection cap (covering the resolving-decoder projection skip on both reader types), and GenericDatumReader.skip() additionally bounds the cumulative count using the heap-aware cap for zero-byte elements and the structural cap otherwise.
> - Maps were already safe against pre-allocation on both paths because each entry carries a string key of at least one byte.
> Verification: a full matrix of array<null|long|int> and map<null|long> at count 200,000,000 under -Xmx256m is rejected on both the fast and classic readers (SystemLimitException for zero-byte elements, EOFException otherwise) instead of OutOfMemoryError; the skip path is bounded on both readers including under schema resolution; legitimate collections within the limit still decode. This is part of AVRO-4292.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)