[jira] [Created] (AVRO-4323) [Java] Bound DataFileStream block size against available input before allocating the block buffer

["Ismaël Mejía (Jira)" <ji...@apache.org>]

Ismaël Mejía created AVRO-4323:
----------------------------------

             Summary: [Java] Bound DataFileStream block size against available input before allocating the block buffer
                 Key: AVRO-4323
                 URL: https://issues.apache.org/jira/browse/AVRO-4323
             Project: Apache Avro
          Issue Type: Improvement
          Components: java
            Reporter: Ismaël Mejía


When reading an Avro data (container) file, {{DataFileStream}} reads each block's declared size as a long and currently validates it only against the {{Integer}} range before allocating the block {{byte[]}} buffer (in {{DataFileStream.DataBlock}}). For a malformed, corrupted, or truncated file, the declared block size can be much larger than the number of bytes actually present, so the reader eagerly allocates a very large buffer on the first {{hasNext()}}/{{next()}} call before any block byte has been read.

We should make block reading more robust for malformed input:
* validate the declared block size against the number of bytes actually remaining in the input (when that is known, e.g. a seekable/known-length source), and/or
* route the block-size allocation through the existing {{SystemLimits}} configuration already used for bytes/collection sizing elsewhere in the reader,

so that reading a malformed file fails fast with a clear {{IOException}} instead of attempting an oversized allocation.

This can be reproduced with a small hand-crafted file whose block header declares a size near {{Integer.MAX_VALUE}} while carrying no block bytes. A valid file continues to read normally.

Relevant code:
* {{lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java}} — {{hasNextBlock()}} (block-size read/validation) and {{DataBlock}} constructor ({{new byte[blockSize]}}).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)