You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pchintar (via GitHub)" <gi...@apache.org> on 2026/04/23 07:47:56 UTC
[PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
pchintar opened a new pull request, #9797:
URL: https://github.com/apache/arrow-rs/pull/9797
# Which issue does this PR close?
- Closes #9796 .
# Rationale for this change
Currently, `DeltaByteArrayDecoder::get` assumes prefix lengths are always valid and directly slices `previous_value`. Invalid prefix lengths (negative or exceeding previous value length) can cause a panic instead of returning an error.
# What changes are included in this PR?
* Add validation for decoded prefix lengths:
* reject negative values
* reject values exceeding `previous_value.len()`
* Return `Err` instead of panicking on invalid input
* Add a regression test using corrupted encoded data
# Are these changes tested?
Yes.
* Added `test_delta_byte_array_invalid_prefix_len_returns_error`
* Test:
* encodes valid data
* corrupts prefix-length stream
* verifies decoder returns `Err` (previously panicked)
* All the other existing tests pass
# Are there any user-facing changes?
No.
* No API changes
* Only improves error handling for invalid input
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306541776
Hi @etseidl, your benchmark configuration could not be parsed (https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306541572).
**Error:** `invalid configuration: env: invalid type: string "BENCH_FILTER:read.*delta_byte", expected a map at line 2 column 3`
Supported benchmarks:
- Standard: (none)
- Criterion: (any)
Usage:
```
run benchmark <name> # run specific benchmark(s)
run benchmarks # run default suite
run benchmarks <name1> <name2> # run specific benchmarks
```
Per-side configuration (`run benchmark tpch` followed by):
```yaml
env:
SHARED_SETTING: enabled
baseline:
ref: v45.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G
```
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306673323
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306626542)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group delta_byte_array main
----- ---------------- ----
read Binary(100) delta_byte_array 1.00 8.4±0.14ms ? ?/sec 1.08 9.1±0.23ms ? ?/sec
read Binary(20) delta_byte_array 1.01 4.9±0.04ms ? ?/sec 1.00 4.9±0.04ms ? ?/sec
read Fixed(16) delta_byte_array 1.00 3.4±0.06ms ? ?/sec 1.01 3.5±0.11ms ? ?/sec
read Fixed(2) delta_byte_array 1.00 3.2±0.01ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
read String(100) delta_byte_array 1.00 9.3±0.32ms ? ?/sec 1.00 9.3±0.16ms ? ?/sec
read String(20) delta_byte_array 1.01 5.2±0.03ms ? ?/sec 1.00 5.2±0.02ms ? ?/sec
read StringView(100) delta_byte_array 1.00 8.6±0.49ms ? ?/sec 1.07 9.3±0.27ms ? ?/sec
read StringView(20) delta_byte_array 1.00 4.9±0.04ms ? ?/sec 1.00 4.9±0.05ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 85.0s |
| Peak memory | 4.3 GiB |
| Avg memory | 4.2 GiB |
| CPU user | 82.0s |
| CPU sys | 0.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 85.0s |
| Peak memory | 4.3 GiB |
| Avg memory | 4.2 GiB |
| CPU user | 81.0s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306895226
Thank you @etseidl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306541572
run benchmark parquet_round_trip
env:
BENCH_FILTER:read.*delta_byte
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306614077
run benchmark parquet_round_trip
```yaml
env:
BENCH_FILTER:read.*delta_byte
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306614164
Hi @etseidl, your benchmark configuration could not be parsed (https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306614077).
**Error:** `invalid configuration: env: invalid type: string "BENCH_FILTER:read.*delta_byte", expected a map at line 2 column 3`
Supported benchmarks:
- Standard: (none)
- Criterion: (any)
Usage:
```
run benchmark <name> # run specific benchmark(s)
run benchmarks # run default suite
run benchmarks <name1> <name2> # run specific benchmarks
```
Per-side configuration (`run benchmark tpch` followed by):
```yaml
env:
SHARED_SETTING: enabled
baseline:
ref: v45.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G
```
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#issuecomment-4316848366
Thanks @pchintar and @etseidl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on code in PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#discussion_r3132459363
##########
parquet/src/encodings/decoding.rs:
##########
@@ -1134,7 +1134,22 @@ impl<T: DataType> Decoder<T> for DeltaByteArrayDecoder<T> {
let suffix = v[0].data();
// Extract current prefix length, can be 0
- let prefix_len = self.prefix_lengths[self.current_idx] as usize;
+ let prefix_len_i32 = self.prefix_lengths[self.current_idx];
+ if prefix_len_i32 < 0 {
+ return Err(general_err!(
+ "Invalid DELTA_BYTE_ARRAY prefix length {}",
+ prefix_len_i32
+ ));
+ }
+ let prefix_len = prefix_len_i32 as usize;
Review Comment:
Thanks @etseidl for the suggestion — updated to use `usize::try_from(...)` with proper error mapping.
Also added an additional test `test_delta_byte_array_negative_prefix_len_returns_error` to cover the negative prefix length case so that path is now exercised as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306626542
run benchmark parquet_round_trip
```yaml
env:
BENCH_FILTER: read.*delta_byte
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on code in PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#discussion_r3132459363
##########
parquet/src/encodings/decoding.rs:
##########
@@ -1134,7 +1134,22 @@ impl<T: DataType> Decoder<T> for DeltaByteArrayDecoder<T> {
let suffix = v[0].data();
// Extract current prefix length, can be 0
- let prefix_len = self.prefix_lengths[self.current_idx] as usize;
+ let prefix_len_i32 = self.prefix_lengths[self.current_idx];
+ if prefix_len_i32 < 0 {
+ return Err(general_err!(
+ "Invalid DELTA_BYTE_ARRAY prefix length {}",
+ prefix_len_i32
+ ));
+ }
+ let prefix_len = prefix_len_i32 as usize;
Review Comment:
Thanks @etseidl for the suggestion — updated to use `usize::try_from(...)` with proper error mapping.
Also added an additional test to cover the negative prefix length case so that path is now exercised as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306651344
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9797#issuecomment-4306626542)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4306626542-1782-sf5xd 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing delta_byte_array (542824bf1c3aef80cf643aafbe026890a9229571) to b93240a (merge-base) [diff](https://github.com/apache/arrow-rs/compare/b93240a6e3a49866b52eb9ddaf95b513f57fc1f3..542824bf1c3aef80cf643aafbe026890a9229571)
BENCH_NAME=parquet_round_trip
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench parquet_round_trip
BENCH_FILTER=read.*delta_byte
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9797:
URL: https://github.com/apache/arrow-rs/pull/9797#discussion_r3132070010
##########
parquet/src/encodings/decoding.rs:
##########
@@ -1134,7 +1134,22 @@ impl<T: DataType> Decoder<T> for DeltaByteArrayDecoder<T> {
let suffix = v[0].data();
// Extract current prefix length, can be 0
- let prefix_len = self.prefix_lengths[self.current_idx] as usize;
+ let prefix_len_i32 = self.prefix_lengths[self.current_idx];
+ if prefix_len_i32 < 0 {
+ return Err(general_err!(
+ "Invalid DELTA_BYTE_ARRAY prefix length {}",
+ prefix_len_i32
+ ));
+ }
+ let prefix_len = prefix_len_i32 as usize;
Review Comment:
```suggestion
let prefix_len = usize::try_from(self.prefix_lengths[self.current_idx])?;
```
A little less verbose. 😄
I notice the added test doesn't cover this possibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org