[jira] [Commented] (AVRO-4323) [Java] Bound DataFileStream block size against available input before allocating the block buffer

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102491#comment-18102491 ] 

ASF subversion and git services commented on AVRO-4323:
-------------------------------------------------------

Commit 23aad5216f874d0b7f3f2f43826cfb336747a970 in avro's branch refs/heads/main from Ismaël Mejía
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=23aad5216f ]

AVRO-4323: [Java] Bound DataFileStream block size against available input before allocating the block buffer (#3919)

* AVRO-4323: [Java] Bound DataFileStream block size against available input

When reading a data (container) file, DataFileStream validated the declared
block size only against the Integer range before allocating the block buffer.
For a malformed, corrupted, or truncated file the declared size can greatly
exceed the bytes actually present, so the reader eagerly allocated a large
buffer before reading any block byte.

Reject a declared block size that exceeds the number of bytes remaining in the
input when that count is known (byte-array- or known-length-stream-backed
decoders), so reading a malformed file fails fast with a clear IOException.
The check is skipped when the remaining count is unknown (-1).

* AVRO-4323: Address review: account for sync marker in block-size guard; close reader in test

Include the trailing sync-marker length (DataFileConstants.SYNC_SIZE) in the
remaining-bytes check so a file truncated right before the sync marker is
rejected before allocating the block buffer, and use try-with-resources in the
regression test.

> [Java] Bound DataFileStream block size against available input before allocating the block buffer
> -------------------------------------------------------------------------------------------------
>
>                 Key: AVRO-4323
>                 URL: https://issues.apache.org/jira/browse/AVRO-4323
>             Project: Apache Avro
>          Issue Type: Improvement
>          Components: java
>            Reporter: Ismaël Mejía
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> When reading an Avro data (container) file, {{DataFileStream}} reads each block's declared size as a long and currently validates it only against the {{Integer}} range before allocating the block {{byte[]}} buffer (in {{DataFileStream.DataBlock}}). For a malformed, corrupted, or truncated file, the declared block size can be much larger than the number of bytes actually present, so the reader eagerly allocates a very large buffer on the first {{hasNext()}}/{{next()}} call before any block byte has been read.
> We should make block reading more robust for malformed input:
> * validate the declared block size against the number of bytes actually remaining in the input (when that is known, e.g. a seekable/known-length source), and/or
> * route the block-size allocation through the existing {{SystemLimits}} configuration already used for bytes/collection sizing elsewhere in the reader,
> so that reading a malformed file fails fast with a clear {{IOException}} instead of attempting an oversized allocation.
> This can be reproduced with a small hand-crafted file whose block header declares a size near {{Integer.MAX_VALUE}} while carrying no block bytes. A valid file continues to read normally.
> Relevant code:
> * {{lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java}} — {{hasNextBlock()}} (block-size read/validation) and {{DataBlock}} constructor ({{new byte[blockSize]}}).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)