You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/09/27 17:27:15 UTC
[PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
adriangb opened a new pull request, #11237:
URL: https://github.com/apache/arrow-rs/pull/11237
# Which issue does this PR close?
- Part of https://github.com/apache/arrow-rs/issues/11234.
# Rationale for this change
`PushBuffers` scans all buffers to find the bytes of a range, and `clear_ranges` compares each buffer with each range to clear. If a caller pushes one buffer per requested range, the decoder does `O(buffers)` work for each lookup, so the work for one row group increases with the square of the number of column chunks. The existing `push_decoder` benchmark shows this:
| Benchmark (`benches/push_decoder.rs`) | `main`, run 1 | `main`, run 2 | This PR |
|---|---:|---:|---:|
| `Nbuf/1000ranges` | 885 ยตs | 517 ยตs | 302 ยตs |
| `Nbuf/10000ranges` | 19.3 ms | 12.9 ms | 3.0 ms |
| `Nbuf/100000ranges` | 574 ms | 602 ms | 35 ms |
| `1buf/*` | | | no reliable change (โ40% to +6% between runs) |
Local laptop, `cargo bench -p parquet --features arrow --bench push_decoder`, 3 s measurement. I ran `main` two times, before and after this PR, because the machine is noisy. The `Nbuf` improvement is clear in both runs. The `1buf` cases push one buffer, so this change does not affect them, and their differences are noise.
The batch granularity in https://github.com/apache/arrow-rs/issues/11234 pushes one buffer per page, which makes the number of buffers much larger.
# What changes are included in this PR?
`PushBuffers` keeps its buffers sorted by the start of their range.
| Operation | Before | After |
|---|---|---|
| `push_range` | append | binary search, then insert (append for ranges pushed in file order) |
| `has_range`, `get_bytes`, `Read` | scan all buffers | binary search, then a short backward scan (overlapping ranges are still allowed, see the diagram on `find`) |
| `clear_ranges` | compare each buffer with each range | sort the ranges once, then binary search |
# Are these changes tested?
Yes. New unit tests in `util/push_buffers.rs`: out-of-order pushes, overlapping pushes (a small buffer inside a larger one), `clear_ranges` with exact matches only, and a randomized test that compares lookups and clears with a linear scan over 200 seeds.
# Are there any user-facing changes?
No.
https://github.com/apache/arrow-rs/pull/11235 and https://github.com/apache/arrow-rs/pull/11236 add `release_ranges` and `retain_ranges` to the same file. The PR that merges last must keep the buffers sorted in those methods.
๐ค Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#discussion_r4218127127
##########
parquet/src/util/push_buffers.rs:
##########
@@ -165,19 +165,32 @@ impl PushBuffers {
/// Returns the index of a buffer that contains all bytes of `start..end`,
/// if any.
fn find(&self, start: u64, end: u64) -> Option<usize> {
- // Ranges can overlap. Thus, the buffer that starts nearest to `start`
- // is not always the buffer that contains `start..end`:
+ // Common case: the buffers do not overlap. Then only the last buffer
Review Comment:
thank yuo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932873694
run benchmark push_decoder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5879527875
@alamb this smaller pr ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5933051521
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932874118)
**Target:** push_decoder
Comparing claude/push-buffers-sorted (b72b195338d1be2da65d30dd6366152cfcfe9a9d) to 007894e (merge-base) [diff](https://github.com/apache/arrow-rs/compare/007894ef0db0eb3b4dc3226d7b490b1104352231..b72b195338d1be2da65d30dd6366152cfcfe9a9d)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
push_decoder/1buf/100000ranges 1.00 46.7ยฑ0.32ms ? ?/sec 1.01 47.0ยฑ0.26ms ? ?/sec
push_decoder/1buf/10000ranges 1.00 4.7ยฑ0.03ms ? ?/sec 1.00 4.7ยฑ0.02ms ? ?/sec
push_decoder/1buf/1000ranges 1.02 468.1ยฑ2.48ยตs ? ?/sec 1.00 460.2ยฑ0.87ยตs ? ?/sec
push_decoder/1buf/500000ranges 1.00 255.8ยฑ2.32ms ? ?/sec 1.01 257.3ยฑ1.99ms ? ?/sec
push_decoder/Nbuf/100000ranges 15.36 893.2ยฑ62.09ms ? ?/sec 1.00 58.1ยฑ0.32ms ? ?/sec
push_decoder/Nbuf/10000ranges 2.36 13.0ยฑ0.02ms ? ?/sec 1.00 5.5ยฑ0.01ms ? ?/sec
push_decoder/Nbuf/1000ranges 1.07 581.5ยฑ2.57ยตs ? ?/sec 1.00 541.4ยฑ1.52ยตs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>push_decoder โ shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-itflr8df-ba3157ff-kwwv
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c5932874118-3001-kpl4d 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench push_decoder
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 195.0s |
| Peak memory | 3.0 GiB |
| Avg memory | 2.3 GiB |
| CPU user | 188.0s |
| CPU sys | 1.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 3.0 GiB |
| Avg memory | 1.8 GiB |
| CPU user | 104.1s |
| CPU sys | 1.0s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5932874118:finish:3970c5eba4b5a080d08384469976a9ab -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#discussion_r4211041578
##########
parquet/src/util/push_buffers.rs:
##########
@@ -275,6 +320,149 @@ mod tests {
assert!(!buffers.has_range(&(10..20)));
}
+ /// The bytes of a fake file: byte `i` is `i % 251`, so any slice of it is
+ /// easy to build and to check.
+ fn file_bytes(range: Range<u64>) -> Bytes {
+ range.map(|i| (i % 251) as u8).collect::<Vec<u8>>().into()
+ }
+
+ fn push(buffers: &mut PushBuffers, range: Range<u64>) {
+ buffers
+ .push_range(range.clone(), file_bytes(range))
+ .unwrap();
+ }
+
+ #[track_caller]
+ fn assert_invariants(buffers: &PushBuffers) {
Review Comment:
Resolved in https://github.com/apache/arrow-rs/pull/11237/commits/53d9331fb2414016e0e8320da74cca84869f7eb7
##########
parquet/src/util/push_buffers.rs:
##########
@@ -132,20 +147,44 @@ impl PushBuffers {
range.end
));
}
- self.ranges.push(range);
- self.buffers.push(buffer);
+ // Insert after all buffers that start at or before `range.start`.
+ // Thus, ranges pushed in file order go at the end.
+ let idx = self.ranges.partition_point(|r| r.start <= range.start);
+ self.max_len = self.max_len.max(expected);
+ self.ranges.insert(idx, range);
+ self.buffers.insert(idx, buffer);
Ok(())
}
/// Returns true if the Buffers contains data for the given range
pub(crate) fn has_range(&self, range: &Range<u64>) -> bool {
- self.ranges
- .iter()
- .any(|r| r.start <= range.start && r.end >= range.end)
+ self.find(range.start, range.end).is_some()
}
- fn iter(&self) -> impl Iterator<Item = (&Range<u64>, &Bytes)> {
- self.ranges.iter().zip(self.buffers.iter())
+ /// Returns the index of a buffer that contains all bytes of `start..end`,
+ /// if any.
+ fn find(&self, start: u64, end: u64) -> Option<usize> {
+ // Ranges can overlap. Thus, the buffer that starts nearest to `start`
+ // is not always the buffer that contains `start..end`:
+ //
+ // buffers: 0..100 โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
+ // 50..60 โโโโโค
+ // 55..58 โโค
+ // find: 55..90 โโโโโโโโโโโโค only 0..100 contains it
+ //
+ // Scan back from the last buffer that starts at or before `start`.
+ // Stop at the first buffer that starts more than `max_len` bytes
+ // before `end`: it and all buffers before it end before `end`. The
Review Comment:
Resolved in https://github.com/apache/arrow-rs/pull/11237/commits/f98e54729620759ceeaa5da2bdb3e8293c0736ee
##########
parquet/src/util/push_buffers.rs:
##########
@@ -41,16 +41,30 @@ use std::ops::Range;
///
/// Thus, the implementation defers to the caller to coalesce subsequent requests
/// if desired.
+///
+/// # Ordering
+///
+/// The buffers are sorted by the start of their range. Thus, lookups use a
+/// binary search, not a scan of all buffers. Pushed ranges can overlap.
#[derive(Debug, Clone, Default)]
pub struct PushBuffers {
/// the virtual "offset" of this buffers (added to any request)
offset: u64,
/// The total length of the file being decoded
file_len: u64,
- /// The ranges of data that are available for decoding (not adjusted for offset)
+ /// The ranges of data that are available for decoding (not adjusted for
+ /// offset), sorted by `start`
ranges: Vec<Range<u64>>,
- /// The buffers of data that can be used to decode the Parquet file
+ /// The buffers of data that can be used to decode the Parquet file, in the
+ /// same order as `ranges`
buffers: Vec<Bytes>,
+ /// The maximum length of the ranges in `ranges`. A buffer that starts
Review Comment:
Resolved in https://github.com/apache/arrow-rs/pull/11237/commits/ae56305f46a6baf5a03bd41be443c8d5fe4749e4
##########
parquet/src/util/push_buffers.rs:
##########
@@ -275,6 +320,149 @@ mod tests {
assert!(!buffers.has_range(&(10..20)));
}
+ /// The bytes of a fake file: byte `i` is `i % 251`, so any slice of it is
+ /// easy to build and to check.
+ fn file_bytes(range: Range<u64>) -> Bytes {
+ range.map(|i| (i % 251) as u8).collect::<Vec<u8>>().into()
+ }
+
+ fn push(buffers: &mut PushBuffers, range: Range<u64>) {
+ buffers
+ .push_range(range.clone(), file_bytes(range))
+ .unwrap();
+ }
+
+ #[track_caller]
+ fn assert_invariants(buffers: &PushBuffers) {
+ assert_eq!(buffers.ranges.len(), buffers.buffers.len());
+ assert!(
+ buffers.ranges.is_sorted_by_key(|r| r.start),
+ "not sorted: {:?}",
+ buffers.ranges
+ );
+ for (range, buffer) in buffers.ranges.iter().zip(&buffers.buffers) {
+ assert_eq!(*buffer, file_bytes(range.clone()));
+ assert!(range.end - range.start <= buffers.max_len);
+ }
+ let max_len = buffers.ranges.iter().map(|r| r.end - r.start).max();
+ assert_eq!(buffers.max_len, max_len.unwrap_or(0));
+ }
+
+ #[test]
+ fn out_of_order_pushes_are_sorted() {
Review Comment:
Resolved in https://github.com/apache/arrow-rs/pull/11237/commits/1fe89c90131d0247cbf044ec4d9d927456fede8a
##########
parquet/src/util/push_buffers.rs:
##########
@@ -275,6 +320,149 @@ mod tests {
assert!(!buffers.has_range(&(10..20)));
}
+ /// The bytes of a fake file: byte `i` is `i % 251`, so any slice of it is
+ /// easy to build and to check.
+ fn file_bytes(range: Range<u64>) -> Bytes {
+ range.map(|i| (i % 251) as u8).collect::<Vec<u8>>().into()
+ }
+
+ fn push(buffers: &mut PushBuffers, range: Range<u64>) {
+ buffers
+ .push_range(range.clone(), file_bytes(range))
+ .unwrap();
+ }
+
+ #[track_caller]
+ fn assert_invariants(buffers: &PushBuffers) {
+ assert_eq!(buffers.ranges.len(), buffers.buffers.len());
+ assert!(
+ buffers.ranges.is_sorted_by_key(|r| r.start),
+ "not sorted: {:?}",
+ buffers.ranges
+ );
+ for (range, buffer) in buffers.ranges.iter().zip(&buffers.buffers) {
+ assert_eq!(*buffer, file_bytes(range.clone()));
+ assert!(range.end - range.start <= buffers.max_len);
+ }
+ let max_len = buffers.ranges.iter().map(|r| r.end - r.start).max();
+ assert_eq!(buffers.max_len, max_len.unwrap_or(0));
+ }
+
+ #[test]
+ fn out_of_order_pushes_are_sorted() {
+ let mut buffers = PushBuffers::new(1000);
+ for range in [50..60, 10..20, 30..40, 0..5, 90..100, 10..15] {
+ push(&mut buffers, range);
+ }
+ assert_invariants(&buffers);
+ assert_eq!(
+ buffers.ranges,
+ vec![0..5, 10..20, 10..15, 30..40, 50..60, 90..100]
+ );
+ assert!(buffers.has_range(&(12..18)));
+ assert!(buffers.has_range(&(30..40)));
+ assert!(!buffers.has_range(&(18..22)));
+ assert!(!buffers.has_range(&(60..61)));
+ assert_eq!(buffers.get_bytes(52, 4).unwrap(), file_bytes(52..56));
+ assert!(matches!(
+ buffers.get_bytes(40, 20),
+ Err(ParquetError::NeedMoreDataRange(r)) if r == (40..60)
+ ));
+ }
+
+ #[test]
+ fn overlapping_pushes_find_the_containing_buffer() {
+ let mut buffers = PushBuffers::new(1000);
+ // Small buffers inside a large one start closer to most offsets.
+ for range in [0..100, 10..20, 50..60, 55..58] {
+ push(&mut buffers, range);
+ }
+ assert_invariants(&buffers);
+ // 55..90 starts in 50..60 and 55..58, but only 0..100 contains it.
+ assert_eq!(buffers.get_bytes(55, 35).unwrap(), file_bytes(55..90));
+ assert!(buffers.has_range(&(0..100)));
+ assert!(buffers.has_range(&(99..100)));
+ assert!(!buffers.has_range(&(99..101)));
+
+ // `Read` finds the same buffer.
+ let mut reader = buffers.get_read(56).unwrap();
+ let mut out = [0u8; 30];
+ std::io::Read::read_exact(&mut reader, &mut out).unwrap();
+ assert_eq!(&out[..], &file_bytes(56..86)[..]);
+ }
+
+ #[test]
+ #[cfg(feature = "arrow")]
+ fn clear_ranges_drops_exact_matches_only() {
+ let mut buffers = PushBuffers::new(1000);
+ for range in [10..20, 0..30, 10..15, 40..50] {
+ push(&mut buffers, range);
+ }
+ buffers.clear_ranges(&[40..50, 10..15, 5..30]);
+ assert_invariants(&buffers);
+ assert_eq!(buffers.ranges, vec![0..30, 10..20]);
+ }
+
+ /// Random pushes, clears and lookups, compared with a list that is
+ /// scanned in full for each lookup.
+ #[test]
+ #[cfg(feature = "arrow")]
+ fn matches_a_linear_scan() {
Review Comment:
Resolved in https://github.com/apache/arrow-rs/pull/11237/commits/a515b5c24285562f77969beff370dac00a1a0ce5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932874580
run benchmark push_decoder
baseline:
ref: 007894ef0db0eb3b4dc3226d7b490b1104352231
changed:
ref: 007894ef0db0eb3b4dc3226d7b490b1104352231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-6058506787
Thanks again @adriangb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932874118
run benchmark push_decoder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5940563482
will check tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#discussion_r4210684965
##########
parquet/src/util/push_buffers.rs:
##########
@@ -132,20 +147,44 @@ impl PushBuffers {
range.end
));
}
- self.ranges.push(range);
- self.buffers.push(buffer);
+ // Insert after all buffers that start at or before `range.start`.
+ // Thus, ranges pushed in file order go at the end.
+ let idx = self.ranges.partition_point(|r| r.start <= range.start);
+ self.max_len = self.max_len.max(expected);
+ self.ranges.insert(idx, range);
+ self.buffers.insert(idx, buffer);
Ok(())
}
/// Returns true if the Buffers contains data for the given range
pub(crate) fn has_range(&self, range: &Range<u64>) -> bool {
- self.ranges
- .iter()
- .any(|r| r.start <= range.start && r.end >= range.end)
+ self.find(range.start, range.end).is_some()
}
- fn iter(&self) -> impl Iterator<Item = (&Range<u64>, &Bytes)> {
- self.ranges.iter().zip(self.buffers.iter())
+ /// Returns the index of a buffer that contains all bytes of `start..end`,
+ /// if any.
+ fn find(&self, start: u64, end: u64) -> Option<usize> {
+ // Ranges can overlap. Thus, the buffer that starts nearest to `start`
+ // is not always the buffer that contains `start..end`:
+ //
+ // buffers: 0..100 โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
+ // 50..60 โโโโโค
+ // 55..58 โโค
+ // find: 55..90 โโโโโโโโโโโโค only 0..100 contains it
+ //
+ // Scan back from the last buffer that starts at or before `start`.
+ // Stop at the first buffer that starts more than `max_len` bytes
+ // before `end`: it and all buffers before it end before `end`. The
Review Comment:
I am confused -- why do we need max_bytes here? Why don't you just look for the first buffer that has at least `end-start` bytes?
##########
parquet/src/util/push_buffers.rs:
##########
@@ -41,16 +41,30 @@ use std::ops::Range;
///
/// Thus, the implementation defers to the caller to coalesce subsequent requests
/// if desired.
+///
+/// # Ordering
+///
+/// The buffers are sorted by the start of their range. Thus, lookups use a
+/// binary search, not a scan of all buffers. Pushed ranges can overlap.
#[derive(Debug, Clone, Default)]
pub struct PushBuffers {
/// the virtual "offset" of this buffers (added to any request)
offset: u64,
/// The total length of the file being decoded
file_len: u64,
- /// The ranges of data that are available for decoding (not adjusted for offset)
+ /// The ranges of data that are available for decoding (not adjusted for
+ /// offset), sorted by `start`
ranges: Vec<Range<u64>>,
- /// The buffers of data that can be used to decode the Parquet file
+ /// The buffers of data that can be used to decode the Parquet file, in the
+ /// same order as `ranges`
buffers: Vec<Bytes>,
+ /// The maximum length of the ranges in `ranges`. A buffer that starts
Review Comment:
I found this comment really hard to understand. COuld you cut this down / add an example of what max_len buys and why it is valuable
Since we know the range of each buffer, why do we need to remember the maximum length across all buffers?
##########
parquet/src/util/push_buffers.rs:
##########
@@ -275,6 +320,149 @@ mod tests {
assert!(!buffers.has_range(&(10..20)));
}
+ /// The bytes of a fake file: byte `i` is `i % 251`, so any slice of it is
+ /// easy to build and to check.
+ fn file_bytes(range: Range<u64>) -> Bytes {
+ range.map(|i| (i % 251) as u8).collect::<Vec<u8>>().into()
+ }
+
+ fn push(buffers: &mut PushBuffers, range: Range<u64>) {
+ buffers
+ .push_range(range.clone(), file_bytes(range))
+ .unwrap();
+ }
+
+ #[track_caller]
+ fn assert_invariants(buffers: &PushBuffers) {
+ assert_eq!(buffers.ranges.len(), buffers.buffers.len());
+ assert!(
+ buffers.ranges.is_sorted_by_key(|r| r.start),
+ "not sorted: {:?}",
+ buffers.ranges
+ );
+ for (range, buffer) in buffers.ranges.iter().zip(&buffers.buffers) {
+ assert_eq!(*buffer, file_bytes(range.clone()));
+ assert!(range.end - range.start <= buffers.max_len);
+ }
+ let max_len = buffers.ranges.iter().map(|r| r.end - r.start).max();
+ assert_eq!(buffers.max_len, max_len.unwrap_or(0));
+ }
+
+ #[test]
+ fn out_of_order_pushes_are_sorted() {
Review Comment:
Is this tst really necessary? Maybe you could change one of the other tests to push the ranges in some different order and then the call to `assert_invaraints` will ensure the data is sorted
##########
parquet/src/util/push_buffers.rs:
##########
@@ -275,6 +320,149 @@ mod tests {
assert!(!buffers.has_range(&(10..20)));
}
+ /// The bytes of a fake file: byte `i` is `i % 251`, so any slice of it is
+ /// easy to build and to check.
+ fn file_bytes(range: Range<u64>) -> Bytes {
+ range.map(|i| (i % 251) as u8).collect::<Vec<u8>>().into()
+ }
+
+ fn push(buffers: &mut PushBuffers, range: Range<u64>) {
+ buffers
+ .push_range(range.clone(), file_bytes(range))
+ .unwrap();
+ }
+
+ #[track_caller]
+ fn assert_invariants(buffers: &PushBuffers) {
Review Comment:
This might be a nice `#test` method on the `PushBuffers` structure itself as it puts the invariant checks closer to the logic that uses them
##########
parquet/src/util/push_buffers.rs:
##########
@@ -132,20 +147,44 @@ impl PushBuffers {
range.end
));
}
- self.ranges.push(range);
- self.buffers.push(buffer);
+ // Insert after all buffers that start at or before `range.start`.
+ // Thus, ranges pushed in file order go at the end.
+ let idx = self.ranges.partition_point(|r| r.start <= range.start);
+ self.max_len = self.max_len.max(expected);
+ self.ranges.insert(idx, range);
+ self.buffers.insert(idx, buffer);
Ok(())
}
/// Returns true if the Buffers contains data for the given range
pub(crate) fn has_range(&self, range: &Range<u64>) -> bool {
- self.ranges
- .iter()
- .any(|r| r.start <= range.start && r.end >= range.end)
+ self.find(range.start, range.end).is_some()
}
- fn iter(&self) -> impl Iterator<Item = (&Range<u64>, &Bytes)> {
- self.ranges.iter().zip(self.buffers.iter())
+ /// Returns the index of a buffer that contains all bytes of `start..end`,
+ /// if any.
+ fn find(&self, start: u64, end: u64) -> Option<usize> {
+ // Ranges can overlap. Thus, the buffer that starts nearest to `start`
+ // is not always the buffer that contains `start..end`:
+ //
+ // buffers: 0..100 โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
+ // 50..60 โโโโโค
+ // 55..58 โโค
+ // find: 55..90 โโโโโโโโโโโโค only 0..100 contains it
+ //
+ // Scan back from the last buffer that starts at or before `start`.
+ // Stop at the first buffer that starts more than `max_len` bytes
+ // before `end`: it and all buffers before it end before `end`. The
Review Comment:
I also found this coment super confusing
##########
parquet/src/util/push_buffers.rs:
##########
@@ -275,6 +320,149 @@ mod tests {
assert!(!buffers.has_range(&(10..20)));
}
+ /// The bytes of a fake file: byte `i` is `i % 251`, so any slice of it is
+ /// easy to build and to check.
+ fn file_bytes(range: Range<u64>) -> Bytes {
+ range.map(|i| (i % 251) as u8).collect::<Vec<u8>>().into()
+ }
+
+ fn push(buffers: &mut PushBuffers, range: Range<u64>) {
+ buffers
+ .push_range(range.clone(), file_bytes(range))
+ .unwrap();
+ }
+
+ #[track_caller]
+ fn assert_invariants(buffers: &PushBuffers) {
+ assert_eq!(buffers.ranges.len(), buffers.buffers.len());
+ assert!(
+ buffers.ranges.is_sorted_by_key(|r| r.start),
+ "not sorted: {:?}",
+ buffers.ranges
+ );
+ for (range, buffer) in buffers.ranges.iter().zip(&buffers.buffers) {
+ assert_eq!(*buffer, file_bytes(range.clone()));
+ assert!(range.end - range.start <= buffers.max_len);
+ }
+ let max_len = buffers.ranges.iter().map(|r| r.end - r.start).max();
+ assert_eq!(buffers.max_len, max_len.unwrap_or(0));
+ }
+
+ #[test]
+ fn out_of_order_pushes_are_sorted() {
+ let mut buffers = PushBuffers::new(1000);
+ for range in [50..60, 10..20, 30..40, 0..5, 90..100, 10..15] {
+ push(&mut buffers, range);
+ }
+ assert_invariants(&buffers);
+ assert_eq!(
+ buffers.ranges,
+ vec![0..5, 10..20, 10..15, 30..40, 50..60, 90..100]
+ );
+ assert!(buffers.has_range(&(12..18)));
+ assert!(buffers.has_range(&(30..40)));
+ assert!(!buffers.has_range(&(18..22)));
+ assert!(!buffers.has_range(&(60..61)));
+ assert_eq!(buffers.get_bytes(52, 4).unwrap(), file_bytes(52..56));
+ assert!(matches!(
+ buffers.get_bytes(40, 20),
+ Err(ParquetError::NeedMoreDataRange(r)) if r == (40..60)
+ ));
+ }
+
+ #[test]
+ fn overlapping_pushes_find_the_containing_buffer() {
+ let mut buffers = PushBuffers::new(1000);
+ // Small buffers inside a large one start closer to most offsets.
+ for range in [0..100, 10..20, 50..60, 55..58] {
+ push(&mut buffers, range);
+ }
+ assert_invariants(&buffers);
+ // 55..90 starts in 50..60 and 55..58, but only 0..100 contains it.
+ assert_eq!(buffers.get_bytes(55, 35).unwrap(), file_bytes(55..90));
+ assert!(buffers.has_range(&(0..100)));
+ assert!(buffers.has_range(&(99..100)));
+ assert!(!buffers.has_range(&(99..101)));
+
+ // `Read` finds the same buffer.
+ let mut reader = buffers.get_read(56).unwrap();
+ let mut out = [0u8; 30];
+ std::io::Read::read_exact(&mut reader, &mut out).unwrap();
+ assert_eq!(&out[..], &file_bytes(56..86)[..]);
+ }
+
+ #[test]
+ #[cfg(feature = "arrow")]
+ fn clear_ranges_drops_exact_matches_only() {
+ let mut buffers = PushBuffers::new(1000);
+ for range in [10..20, 0..30, 10..15, 40..50] {
+ push(&mut buffers, range);
+ }
+ buffers.clear_ranges(&[40..50, 10..15, 5..30]);
+ assert_invariants(&buffers);
+ assert_eq!(buffers.ranges, vec![0..30, 10..20]);
+ }
+
+ /// Random pushes, clears and lookups, compared with a list that is
+ /// scanned in full for each lookup.
+ #[test]
+ #[cfg(feature = "arrow")]
+ fn matches_a_linear_scan() {
Review Comment:
Perhaps using the word `fuzz` in this test name would help people understand it more easily
##########
parquet/src/util/push_buffers.rs:
##########
@@ -132,20 +147,44 @@ impl PushBuffers {
range.end
));
}
- self.ranges.push(range);
- self.buffers.push(buffer);
+ // Insert after all buffers that start at or before `range.start`.
+ // Thus, ranges pushed in file order go at the end.
+ let idx = self.ranges.partition_point(|r| r.start <= range.start);
+ self.max_len = self.max_len.max(expected);
+ self.ranges.insert(idx, range);
+ self.buffers.insert(idx, buffer);
Ok(())
}
/// Returns true if the Buffers contains data for the given range
pub(crate) fn has_range(&self, range: &Range<u64>) -> bool {
- self.ranges
- .iter()
- .any(|r| r.start <= range.start && r.end >= range.end)
+ self.find(range.start, range.end).is_some()
}
- fn iter(&self) -> impl Iterator<Item = (&Range<u64>, &Bytes)> {
- self.ranges.iter().zip(self.buffers.iter())
+ /// Returns the index of a buffer that contains all bytes of `start..end`,
+ /// if any.
+ fn find(&self, start: u64, end: u64) -> Option<usize> {
+ // Ranges can overlap. Thus, the buffer that starts nearest to `start`
+ // is not always the buffer that contains `start..end`:
+ //
+ // buffers: 0..100 โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
+ // 50..60 โโโโโค
+ // 55..58 โโค
+ // find: 55..90 โโโโโโโโโโโโค only 0..100 contains it
+ //
+ // Scan back from the last buffer that starts at or before `start`.
+ // Stop at the first buffer that starts more than `max_len` bytes
+ // before `end`: it and all buffers before it end before `end`. The
Review Comment:
> Thus, the buffer that starts nearest to `start`
> // is not always the buffer that contains `start..end`:
I think the key insight I was missing here is that `max_len` is an optimization for the common case of non overlapping buffers (what the decoder will ask for) to avoid having to check all earlier buffers in the queue for overlap
Maybe the example could show the common case with non overlapping buffers and explain that we want to stop early once we get before the requested range. However, with overlap in general we would have to check all of them (and then add the example of one giant range)
##########
parquet/src/util/push_buffers.rs:
##########
@@ -41,16 +41,30 @@ use std::ops::Range;
///
/// Thus, the implementation defers to the caller to coalesce subsequent requests
/// if desired.
+///
+/// # Ordering
+///
+/// The buffers are sorted by the start of their range. Thus, lookups use a
+/// binary search, not a scan of all buffers. Pushed ranges can overlap.
#[derive(Debug, Clone, Default)]
pub struct PushBuffers {
/// the virtual "offset" of this buffers (added to any request)
offset: u64,
/// The total length of the file being decoded
file_len: u64,
- /// The ranges of data that are available for decoding (not adjusted for offset)
+ /// The ranges of data that are available for decoding (not adjusted for
+ /// offset), sorted by `start`
ranges: Vec<Range<u64>>,
- /// The buffers of data that can be used to decode the Parquet file
+ /// The buffers of data that can be used to decode the Parquet file, in the
+ /// same order as `ranges`
buffers: Vec<Bytes>,
+ /// The maximum length of the ranges in `ranges`. A buffer that starts
Review Comment:
I think the detail I was missing on first read is that this is an optimization for `find` of the common case where there are relatively equal sized non- overlapping buffers
The comment as now sort of describes implementation details of how `find` work ๐คฏ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932876561
๐ค Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932873694)
**Target:** push_decoder
**Sharding factor: 1** (1 workers).
Comparing claude/push-buffers-sorted (b72b195338d1be2da65d30dd6366152cfcfe9a9d) to 007894e (merge-base) [diff](https://github.com/apache/arrow-rs/compare/007894ef0db0eb3b4dc3226d7b490b1104352231..b72b195338d1be2da65d30dd6366152cfcfe9a9d)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5932873694:start:3fc2c5c1105d591876556c8c4fb20259 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932875202
๐ค Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932874118)
**Target:** push_decoder
**Sharding factor: 1** (1 workers).
Comparing claude/push-buffers-sorted (b72b195338d1be2da65d30dd6366152cfcfe9a9d) to 007894e (merge-base) [diff](https://github.com/apache/arrow-rs/compare/007894ef0db0eb3b4dc3226d7b490b1104352231..b72b195338d1be2da65d30dd6366152cfcfe9a9d)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5932874118:start:85f0582ff97b8c2a04afb2bf36ad1982 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932877962
๐ค Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932874580)
**Target:** push_decoder
**Sharding factor: 1** (1 workers).
Comparing 007894ef0db0eb3b4dc3226d7b490b1104352231 (007894ef0db0eb3b4dc3226d7b490b1104352231) to 007894ef0db0eb3b4dc3226d7b490b1104352231 [diff](https://github.com/apache/arrow-rs/compare/007894ef0db0eb3b4dc3226d7b490b1104352231..007894ef0db0eb3b4dc3226d7b490b1104352231)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
baseline:
ref: 007894ef0db0eb3b4dc3226d7b490b1104352231
changed:
ref: 007894ef0db0eb3b4dc3226d7b490b1104352231
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5932874580:start:cb278c6549f693d5db55c1f43589900f -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5933055843
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932873694)
**Target:** push_decoder
Comparing claude/push-buffers-sorted (b72b195338d1be2da65d30dd6366152cfcfe9a9d) to 007894e (merge-base) [diff](https://github.com/apache/arrow-rs/compare/007894ef0db0eb3b4dc3226d7b490b1104352231..b72b195338d1be2da65d30dd6366152cfcfe9a9d)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
push_decoder/1buf/100000ranges 1.00 47.9ยฑ0.41ms ? ?/sec 1.01 48.6ยฑ0.61ms ? ?/sec
push_decoder/1buf/10000ranges 1.00 4.7ยฑ0.02ms ? ?/sec 1.01 4.7ยฑ0.01ms ? ?/sec
push_decoder/1buf/1000ranges 1.02 469.8ยฑ10.00ยตs ? ?/sec 1.00 462.6ยฑ1.26ยตs ? ?/sec
push_decoder/1buf/500000ranges 1.00 264.7ยฑ1.99ms ? ?/sec 1.02 270.0ยฑ4.71ms ? ?/sec
push_decoder/Nbuf/100000ranges 14.55 896.9ยฑ82.47ms ? ?/sec 1.00 61.7ยฑ2.24ms ? ?/sec
push_decoder/Nbuf/10000ranges 2.29 13.1ยฑ0.04ms ? ?/sec 1.00 5.7ยฑ0.14ms ? ?/sec
push_decoder/Nbuf/1000ranges 1.05 579.1ยฑ0.72ยตs ? ?/sec 1.00 549.5ยฑ2.77ยตs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>push_decoder โ shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-itflr8df-56c893bd-jr8m
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c5932873694-3002-nxgvt 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench push_decoder
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 195.0s |
| Peak memory | 3.0 GiB |
| Avg memory | 2.3 GiB |
| CPU user | 187.9s |
| CPU sys | 1.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 3.0 GiB |
| Avg memory | 1.9 GiB |
| CPU user | 104.2s |
| CPU sys | 1.0s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5932873694:finish:10df3c5ee5a7a3b0df0056633c9c056e -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): keep PushBuffers sorted by offset [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11237:
URL: https://github.com/apache/arrow-rs/pull/11237#issuecomment-5933084587
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11237#issuecomment-5932874580)
**Target:** push_decoder
Comparing 007894ef0db0eb3b4dc3226d7b490b1104352231 (007894ef0db0eb3b4dc3226d7b490b1104352231) to 007894ef0db0eb3b4dc3226d7b490b1104352231 [diff](https://github.com/apache/arrow-rs/compare/007894ef0db0eb3b4dc3226d7b490b1104352231..007894ef0db0eb3b4dc3226d7b490b1104352231)
<details><summary>Run configuration</summary>
```yaml
run benchmark push_decoder
baseline:
ref: 007894ef0db0eb3b4dc3226d7b490b1104352231
changed:
ref: 007894ef0db0eb3b4dc3226d7b490b1104352231
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
push_decoder/1buf/100000ranges 1.00 48.8ยฑ2.74ms ? ?/sec 1.06 51.7ยฑ5.00ms ? ?/sec
push_decoder/1buf/10000ranges 1.00 4.7ยฑ0.01ms ? ?/sec 1.00 4.7ยฑ0.02ms ? ?/sec
push_decoder/1buf/1000ranges 1.00 467.2ยฑ1.33ยตs ? ?/sec 1.02 474.8ยฑ6.65ยตs ? ?/sec
push_decoder/1buf/500000ranges 1.01 264.4ยฑ7.68ms ? ?/sec 1.00 260.5ยฑ5.55ms ? ?/sec
push_decoder/Nbuf/100000ranges 1.00 874.3ยฑ14.97ms ? ?/sec 1.03 896.9ยฑ62.89ms ? ?/sec
push_decoder/Nbuf/10000ranges 1.00 13.1ยฑ0.04ms ? ?/sec 1.02 13.3ยฑ0.17ms ? ?/sec
push_decoder/Nbuf/1000ranges 1.00 579.0ยฑ0.73ยตs ? ?/sec 1.01 583.4ยฑ1.42ยตs ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>push_decoder โ shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-io4gwr6s-c27e50db-hvdh
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c5932874580-3003-hk849 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench push_decoder
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 195.0s |
| Peak memory | 3.0 GiB |
| Avg memory | 2.3 GiB |
| CPU user | 186.4s |
| CPU sys | 1.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 195.0s |
| Peak memory | 3.0 GiB |
| Avg memory | 2.2 GiB |
| CPU user | 188.2s |
| CPU sys | 1.0s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:5932874580:finish:ec7a4f3249e1901aaa7e738e23a98298 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org