[jira] [Created] (AVRO-4300) [java] Bound collection allocation for zero-byte elements when decoding arrays and maps

["Ismaël Mejía (Jira)" <ji...@apache.org>]

Ismaël Mejía created AVRO-4300:
----------------------------------

             Summary: [java] Bound collection allocation for zero-byte elements when decoding arrays and maps
                 Key: AVRO-4300
                 URL: https://issues.apache.org/jira/browse/AVRO-4300
             Project: Apache Avro
          Issue Type: Sub-task
          Components: java
    Affects Versions: 1.12.1, 1.11.5
            Reporter: Ismaël Mejía
             Fix For: 1.12.2


When GenericDatumReader decodes an array or map it reads the block item count from the stream and pre-allocates the full backing store up front (newArray/newMap -> new Object[count]) before decoding any element.

The two existing guards do not bound a collection whose elements encode to zero bytes:

- ensureAvailableCollectionBytes (AVRO-4241) only enforces count * minBytesPerElement <= remainingBytes when minBytesPerElement > 0. minBytesPerElement returns 0 for null and for self-referencing records, so the check is skipped for those element types.
- SystemLimitException.checkMaxCollectionLength caps the count at Integer.MAX_VALUE - 8, which is a JVM array-size ceiling rather than a memory budget.

As a result, a small input declaring an array/map of many zero-byte elements (e.g. {"type":"array","items":"null"} with a block count of 200,000,000) drives a single large up-front allocation (new Object[200000000], ~1.6 GB) and exhausts the heap. Confirmed against 1.11.5, 1.12.1 and current main.

Proposed fix: do not pre-size the backing collection to the untrusted block count; clamp the initial capacity and let the collection grow as elements are actually decoded, keeping the cumulative checkMaxCollectionLength cap as the ultimate bound. Add tests and verify the behaviour with a zero-byte-element reproduction. This completes AVRO-4277 by covering the Java SDK's residual gap.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)