You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Ismaël Mejía (Jira)" <ji...@apache.org> on 2026/08/05 20:04:00 UTC

[jira] [Created] (AVRO-4325) [Trevni] Validate column-file header counts and lengths before allocating in the Trevni readers

Ismaël Mejía created AVRO-4325:
----------------------------------

             Summary: [Trevni] Validate column-file header counts and lengths before allocating in the Trevni readers
                 Key: AVRO-4325
                 URL: https://issues.apache.org/jira/browse/AVRO-4325
             Project: Apache Avro
          Issue Type: Improvement
          Components: trevni
            Reporter: Ismaël Mejía


The Trevni columnar reader sizes several allocations directly from values read from the file header/metadata without validating them against the input actually available. For a malformed, corrupted, or truncated Trevni file these counts/lengths may greatly exceed the bytes actually present, leading to oversized eager allocations or an unexpected {{NegativeArraySizeException}}:

* {{ColumnFileReader.readHeader}} allocates {{new ColumnDescriptor[columnCount]}} (and a {{HashMap}} sized to {{columnCount}}) from the 32-bit column count before any column metadata is read.
* {{ColumnDescriptor.ensureBlocksRead}} allocates {{new BlockDescriptor[blockCount]}} from the block count.
* {{InputBuffer.readBytes}}/{{readString}} allocate {{new byte[readInt()]}} from a length prefix.
* {{ColumnValues.startBlock}} computes {{new byte[compressedSize + checksumSize]}}; the addition can overflow to a negative size ({{NegativeArraySizeException}}).

Unlike the Java SDK, Trevni has no shared limit/validation layer. We should:
* validate these counts/lengths against the remaining input (and a sane maximum) before allocating,
* adopt a consistent bound across the Trevni readers ({{ColumnFileReader}}, {{ColumnDescriptor}}, {{InputBuffer}}, {{ColumnValues}}), similar in spirit to the Java SDK's {{SystemLimits}}, and
* guard the {{compressedSize + checksum}} addition against overflow,

so that reading a malformed file fails fast with a clear exception. Valid files continue to read normally.

Relevant code:
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnFileReader.java}} — {{readHeader()}}.
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnDescriptor.java}} — {{ensureBlocksRead()}}.
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/InputBuffer.java}} — {{readBytes()}}/{{readString()}}.
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnValues.java}} — {{startBlock()}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)