You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Ismaël Mejía (Jira)" <ji...@apache.org> on 2026/08/05 20:04:00 UTC
[jira] [Created] (AVRO-4325) [Trevni] Validate column-file header counts and lengths before allocating in the Trevni readers
Ismaël Mejía created AVRO-4325:
----------------------------------
Summary: [Trevni] Validate column-file header counts and lengths before allocating in the Trevni readers
Key: AVRO-4325
URL: https://issues.apache.org/jira/browse/AVRO-4325
Project: Apache Avro
Issue Type: Improvement
Components: trevni
Reporter: Ismaël Mejía
The Trevni columnar reader sizes several allocations directly from values read from the file header/metadata without validating them against the input actually available. For a malformed, corrupted, or truncated Trevni file these counts/lengths may greatly exceed the bytes actually present, leading to oversized eager allocations or an unexpected {{NegativeArraySizeException}}:
* {{ColumnFileReader.readHeader}} allocates {{new ColumnDescriptor[columnCount]}} (and a {{HashMap}} sized to {{columnCount}}) from the 32-bit column count before any column metadata is read.
* {{ColumnDescriptor.ensureBlocksRead}} allocates {{new BlockDescriptor[blockCount]}} from the block count.
* {{InputBuffer.readBytes}}/{{readString}} allocate {{new byte[readInt()]}} from a length prefix.
* {{ColumnValues.startBlock}} computes {{new byte[compressedSize + checksumSize]}}; the addition can overflow to a negative size ({{NegativeArraySizeException}}).
Unlike the Java SDK, Trevni has no shared limit/validation layer. We should:
* validate these counts/lengths against the remaining input (and a sane maximum) before allocating,
* adopt a consistent bound across the Trevni readers ({{ColumnFileReader}}, {{ColumnDescriptor}}, {{InputBuffer}}, {{ColumnValues}}), similar in spirit to the Java SDK's {{SystemLimits}}, and
* guard the {{compressedSize + checksum}} addition against overflow,
so that reading a malformed file fails fast with a clear exception. Valid files continue to read normally.
Relevant code:
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnFileReader.java}} — {{readHeader()}}.
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnDescriptor.java}} — {{ensureBlocksRead()}}.
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/InputBuffer.java}} — {{readBytes()}}/{{readString()}}.
* {{lang/java/trevni/core/src/main/java/org/apache/trevni/ColumnValues.java}} — {{startBlock()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)