You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "masumi-ryugo (via GitHub)" <gi...@apache.org> on 2026/05/03 07:45:17 UTC
[PR] fix(parquet): bound thrift list capacity by remaining input bytes [arrow-rs]
masumi-ryugo opened a new pull request, #9883:
URL: https://github.com/apache/arrow-rs/pull/9883
**Builds on #9868.** When that PR is merged this rebases to a single commit; until then the diff includes its commit as ancestor.
## What
Cap the up-front list-allocation in `parquet::parquet_thrift::read_thrift_vec` by the bytes still available to the input protocol. Adds a `bytes_remaining()` method to `ThriftCompactInputProtocol` (default `usize::MAX`; overridden in `ThriftSliceInputProtocol` to return `self.buf.len()`).
```rust
let declared = list_ident.size as usize;
let bound = declared.min(prot.bytes_remaining());
let mut res: Vec<T> = Vec::new();
if res.try_reserve_exact(bound).is_err() { … }
for _ in 0..list_ident.size { // EOF surfaces if header overstated size
let val = T::read_thrift(prot)?;
res.push(val);
}
```
## Why `try_reserve_exact` alone isn't enough
#9868's `try_reserve_exact` is the right defense in principle, but on **Linux with default overcommit** the allocator returns success for the requested size and the OOM only surfaces under tools that observe the malloc size (libFuzzer, ASan). In practice that means production processes will still OOM-kill on these inputs.
A 7-byte cargo-fuzz repro produced by the new `parquet/fuzz/thrift_decode` harness (#5332) demonstrates this:
```
==…== ERROR: libFuzzer: out-of-memory (malloc(6643777440))
…
#11 0x… RawVec<…SchemaElement>::try_reserve_exact alloc/src/raw_vec/mod.rs:384
#12 0x… parquet_thrift::read_thrift_vec::<SchemaElement, ThriftSliceInputProtocol>
parquet_thrift.rs:713
#13 0x… parquet_metadata_from_bytes metadata/thrift/mod.rs:790
```
Input bytes: `[0x26, 0xf6, 0xf6, 0xf6, 0xd4, 0x6b, 0x6c]` — `i32::MAX`-class list size, then `i32::MAX * sizeof(SchemaElement) ≈ 6.6 GiB`.
After this PR the same 7 bytes return `ParquetError::General("cannot allocate Thrift list of … elements")` immediately, peak RSS stays in single-digit MiB.
## Tests
Two new regression tests in `parquet::parquet_thrift::tests`:
- `test_decode_metadata_oom_repro_7byte_does_not_oom` — the 7-byte cargo-fuzz repro
- `test_decode_metadata_oom_repro_padded_does_not_oom` — a 42-byte variant with `0x20` padding (matches the larger `arrow_reader`-harness repro)
The existing `test_decode_metadata_huge_thrift_list_does_not_panic` and `test_read_list_begin_size_above_i32_max_returns_err` from #9868 continue to pass.
## Out-of-scope (follow-ups)
I'll send these as separate PRs once this lands and #5332's fuzz harness is set up to keep them honest:
1. **`parquet/src/schema/types.rs:1404`** — `Vec::with_capacity(n as usize)` over an attacker-controlled `num_children`. Reachable downstream of thrift decoding; produces a separate `capacity overflow` panic on the same input class. A 22-byte fuzz repro is already in hand.
2. **`parquet/src/format.rs`** — ~22 hand-rolled `Vec::with_capacity(list_ident.size as usize)` sites in the legacy thrift codegen path. Identical mitigation pattern (bound by `bytes_remaining()`), but the file is large and noisy enough to deserve its own PR.
---
xref #5332 #9868 #9869 #9874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(parquet): bound thrift list capacity by remaining input bytes [arrow-rs]
Posted by "masumi-ryugo (via GitHub)" <gi...@apache.org>.
masumi-ryugo commented on PR #9883:
URL: https://github.com/apache/arrow-rs/pull/9883#issuecomment-4378766398
Closing this — agreed it's premature in this shape.
The more fundamental issue I should have called out in the PR description: `bytes_remaining` on `ThriftReadInputProtocol` (the `Read`-backed variant) defaults to `usize::MAX`, so the proposed trait method gives zero defense against streaming/async readers — only the sliced case. That's not a complete enough mitigation to justify adding a new public trait surface.
Will revisit the broader allocation-bounding question under #9874 if a shape that covers both the sliced and `Read` paths becomes clear. Thanks for the read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(parquet): bound thrift list capacity by remaining input bytes [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9883:
URL: https://github.com/apache/arrow-rs/pull/9883#discussion_r3183967285
##########
parquet/src/parquet_thrift.rs:
##########
@@ -1106,4 +1166,63 @@ pub(crate) mod tests {
assert_eq!(header.size, 0);
assert_eq!(header.element_type, ElementType::Byte);
}
+
+ /// Regression test for the 10-byte fuzzer repro that panicked inside
+ /// `alloc::raw_vec::handle_error` with "capacity overflow" before this
+ /// fix: the list header for `SchemaElement` decodes to a multi-billion
+ /// `size`, and `Vec::with_capacity(size)` then tripped the
+ /// `size_of::<SchemaElement>() * size` isize-overflow check. After the
+ /// fix the allocation goes through `try_reserve_exact` and returns a
+ /// clean `ParquetError`.
+ #[test]
+ fn test_decode_metadata_huge_thrift_list_does_not_panic() {
+ let bytes: [u8; 10] = [0x28, 0xfc, 0xfc, 0xfc, 0xfc, 0xfc, 0xfc, 0xfc, 0xfc, 0x51];
+ let result = crate::file::metadata::ParquetMetaDataReader::decode_metadata(&bytes);
+ // The bytes are not valid Parquet metadata — we only require that
+ // `decode_metadata` returns an error rather than panicking.
+ assert!(result.is_err(), "expected error, got {result:?}");
+ }
+
+ /// A Thrift list header whose `size` varint decodes above `i32::MAX`
+ /// must be rejected at the protocol layer rather than wrapping into a
+ /// negative `i32` and being smuggled into downstream allocation code.
+ #[test]
+ fn test_read_list_begin_size_above_i32_max_returns_err() {
+ // List header: element_type=8 (Binary), 0xF=follow-up varint.
+ // Varint 80 80 80 80 08 decodes to 0x8000_0000 = i32::MAX + 1.
+ let mut data: Vec<u8> = vec![0xF8];
+ data.extend_from_slice(&[0x80, 0x80, 0x80, 0x80, 0x08]);
+ let mut prot = ThriftSliceInputProtocol::new(&data);
+ let result = prot.read_list_begin();
+ assert!(result.is_err(), "expected error, got {result:?}");
+ }
+
+ /// Regression test for a 7-byte cargo-fuzz repro found by the
+ /// `parquet/fuzz` `thrift_decode` harness on `parquet/main` after the
+ /// `try_reserve_exact` change: the fix above clamps `i32::MAX`-class
+ /// sizes, but on a 64-bit target the allocator is still asked for
+ /// ~6.6 GiB (`i32::MAX * size_of::<SchemaElement>()`) before the
+ /// `try_reserve_exact` Err path can fire, which is enough to OOM the
+ /// process under libFuzzer / ASan. Bounding by `bytes_remaining` keeps
+ /// the up-front allocation proportional to the input size, so on the
+ /// 7-byte input below the allocator is asked for at most a few bytes.
+ #[test]
+ fn test_decode_metadata_oom_repro_7byte_does_not_oom() {
Review Comment:
The added tests does not error on either Mac nor Linux when added to #9868. If they require a special allocator to trigger, then they provide no extra coverage.
##########
parquet/src/parquet_thrift.rs:
##########
@@ -686,8 +727,27 @@ where
R: ThriftCompactInputProtocol<'a>,
T: ReadThrift<'a, R>,
{
+ // `read_list_begin` rejects sizes above `i32::MAX`, but on a 64-bit
+ // target that is still ~2 GiB of `T`s — easily a multi-GB up-front
+ // allocation when `T` is a struct like `SchemaElement`. Cap the
+ // up-front capacity by the bytes still in the input: each Thrift list
+ // element occupies at least one byte on the wire, so a declared size
+ // larger than what the reader has left is always invalid input. The
+ // subsequent `try_reserve_exact` then serves as belt-and-suspenders
+ // for protocol implementations that can't compute a tight bound.
let list_ident = prot.read_list_begin()?;
- let mut res = Vec::with_capacity(list_ident.size as usize);
+ let declared = list_ident.size as usize;
+ let bound = declared.min(prot.bytes_remaining());
Review Comment:
As mentioned in https://github.com/apache/arrow-rs/pull/9868#issuecomment-4373820729, the size of the input stream has little relation to the size of the resultant structures as the former is highly compressed. Also, this code here is simply incorrect...`bound` has nothing to do with the number of elements the vector should ultimately hold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(parquet): bound thrift list capacity by remaining input bytes [arrow-rs]
Posted by "masumi-ryugo (via GitHub)" <gi...@apache.org>.
masumi-ryugo closed pull request #9883: fix(parquet): bound thrift list capacity by remaining input bytes
URL: https://github.com/apache/arrow-rs/pull/9883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org