[jira] [Updated] (AVRO-4299) [perl] Bound allocation when decoding length-prefixed values and collections

["Ismaël Mejía (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AVRO-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ismaël Mejía updated AVRO-4299:
-------------------------------
    Description: 
A bytes or string value is encoded as a length prefix followed by that many bytes of data, and an array or map block is encoded as an element count followed by that many items. A malicious or truncated input can declare a very large length or count while carrying little or no actual data, causing a large allocation before the shortfall is noticed. When the source can report how many bytes remain, reject a declared length (or a collection block count) that exceeds the bytes actually available before allocating for it. Companion to AVRO-4241 (Java).

_bytes_remaining backs the check (readers that cannot seek to the end yield undef and are skipped); decode_bytes and decode_array/decode_map consult it via _min_bytes_per_element. _ensure_collection_available enforces the per-block limits and skip_block is element-aware. Rejections raise the dedicated Avro::BinaryDecoder::Error::CollectionSize.

Zero-byte elements (null, a zero-length fixed, or a record with only zero-byte fields) consume no input, so the available-bytes check cannot bound their count: a tiny payload such as {"type":"array","items":"null"} declaring a block count of 200,000,000 would otherwise drive an unbounded allocation. In addition to the available-bytes check this therefore caps the cumulative count of zero-byte elements (default 10,000,000), applies a structural cap to every collection (Integer.MAX_VALUE - 8) for readers that cannot report bytes remaining, and bounds the array/map skip paths. When set, the AVRO_MAX_COLLECTION_ITEMS environment variable caps both limits. This supersedes the separate collection-limit sub-task.

  was:A bytes or string value is encoded as a length prefix followed by that many bytes of data. A malicious or truncated input can declare a very large length while carrying little or no actual data, causing a large buffer to be allocated before the shortfall is noticed. When the source can report how many bytes remain, reject a declared length that exceeds the bytes actually available before allocating for it. Companion to AVRO-4241 (Java).

        Summary: [perl] Bound allocation when decoding length-prefixed values and collections  (was: [perl] Validate available bytes before allocating for length-prefixed values)

> [perl] Bound allocation when decoding length-prefixed values and collections
> ----------------------------------------------------------------------------
>
>                 Key: AVRO-4299
>                 URL: https://issues.apache.org/jira/browse/AVRO-4299
>             Project: Apache Avro
>          Issue Type: Sub-task
>          Components: perl
>    Affects Versions: 1.11.5, 1.12.1
>            Reporter: Ismaël Mejía
>            Assignee: Ismaël Mejía
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.13.0, 1.11.6, 1.12.2
>
>          Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> A bytes or string value is encoded as a length prefix followed by that many bytes of data, and an array or map block is encoded as an element count followed by that many items. A malicious or truncated input can declare a very large length or count while carrying little or no actual data, causing a large allocation before the shortfall is noticed. When the source can report how many bytes remain, reject a declared length (or a collection block count) that exceeds the bytes actually available before allocating for it. Companion to AVRO-4241 (Java).
> _bytes_remaining backs the check (readers that cannot seek to the end yield undef and are skipped); decode_bytes and decode_array/decode_map consult it via _min_bytes_per_element. _ensure_collection_available enforces the per-block limits and skip_block is element-aware. Rejections raise the dedicated Avro::BinaryDecoder::Error::CollectionSize.
> Zero-byte elements (null, a zero-length fixed, or a record with only zero-byte fields) consume no input, so the available-bytes check cannot bound their count: a tiny payload such as {"type":"array","items":"null"} declaring a block count of 200,000,000 would otherwise drive an unbounded allocation. In addition to the available-bytes check this therefore caps the cumulative count of zero-byte elements (default 10,000,000), applies a structural cap to every collection (Integer.MAX_VALUE - 8) for readers that cannot report bytes remaining, and bounds the array/map skip paths. When set, the AVRO_MAX_COLLECTION_ITEMS environment variable caps both limits. This supersedes the separate collection-limit sub-task.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)