You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "haohuaijin (via GitHub)" <gi...@apache.org> on 2026/06/14 07:33:56 UTC
[PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
haohuaijin opened a new pull request, #10141:
URL: https://github.com/apache/arrow-rs/pull/10141
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
-->
- Closes #10140
# Rationale for this change
`RowSelection` currently stores selections as `Vec<RowSelector>` (16 bytes per selector). This is compact for long runs, but expensive for scattered matches. With ~35% isolated single-row hits, it uses about 11.2 bytes per input row. A `BooleanBuffer` uses 1 bit per input row, about 90x less memory.
The reader can also choose the `Mask` strategy, which converts selectors back into a bitmap. When the caller already had a bitmap, this conversion round-trip is unnecessary.
# What changes are included in this PR?
`RowSelection` can now be backed by either `Vec<RowSelector>` or `BooleanBuffer`. New public construction:
```rust
pub fn RowSelection::from_boolean_buffer(mask: BooleanBuffer) -> Self;
impl From<BooleanBuffer> for RowSelection;
```
Methods that can work directly on the bitmap now do so:
- `iter()` streams via `BitSliceIterator`
- `row_count` / `skipped_row_count` use `count_set_bits`
- `selects_any` uses `set_indices().next()`
- `trim` preserves mask backing via `BooleanBuffer::slice`
- `intersection` / `union` on `Mask`+`Mask` use `BitAnd` / `BitOr`
- `split_off` on a mask uses `BooleanBuffer::slice` (`O(1)`, both halves stay mask-backed)
- `limit` slices at the selected-row boundary via `find_nth_set_bit_position`, staying mask-backed
- `offset` finds the first selected row to keep via `find_nth_set_bit_position` and rebuilds only the mask buffer, avoiding selector materialization
- `and_then` applies the inner selection over the mask's set positions, returning a mask-backed result
- `FromIterator<RowSelection>` concatenates `BooleanBuffer`s when every input is mask-backed
Mixed inputs, and existing selector-backed inputs, still use the existing selector helpers. Existing callers keep the same behavior.
The reader (`ReadPlanBuilder::build`) passes a mask-backed selection straight to `RowSelectionCursor::new_mask_from_buffer`, so it skips rebuilding the bitmap from selectors.
# Are these changes tested?
Yes. This PR extends the existing `RowSelection` unit tests with coverage for:
- constructing from `BooleanBuffer`, including empty and all-unset masks
- `From<BooleanBuffer>`
- preserving mask backing across clone, `split_off`, `limit`, `offset`, `and_then`, and all-mask `FromIterator<RowSelection>`
- falling back to selector backing for mixed-backed concatenation
- equality between equivalent selector-backed and mask-backed selections
- mask-backed `intersection` / `union`, including uneven-length inputs
- fuzz-style equivalence between mask-backed selections and the existing `from_filters` selector path
# Are there any user-facing changes?
**Yes — one source-breaking change suitable for the next major release**: `RowSelection::iter()` now yields `RowSelector` (`Item = RowSelector`) instead of `&RowSelector`. This is needed because a mask-backed selection does not have a `Vec<RowSelector>` to borrow from.
`RowSelector` is `Copy` (16 bytes), so most call sites are source-compatible:
```rust
selection.iter().map(|s| s.row_count).sum::<usize>() // unchanged
selection.iter().filter(|s| !s.skip).count() // unchanged
selection.iter().any(|s| !s.skip) // unchanged
```
Call sites that need updating:
```rust
// before
let v: Vec<RowSelector> = selection.iter().cloned().collect();
let v: Vec<&RowSelector> = selection.iter().collect();
// after
let v: Vec<RowSelector> = selection.iter().collect();
```
Within arrow-rs, the required updates are limited to parquet's own `RowSelection` code and tests. The other crates do not use `RowSelection::iter()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4810052243
FYI, today I ran `arrow_reader_clickbench` 3 times in #10217, which is a no-op PR (one-char README change), so it's basically main vs main. The run-to-run noise is still large:
- sync/Q13 shows the branch ~14–24% "slower" in all 3 runs, with identical code.
- Run 2 was much noisier: async/Q21 was +34%, while the other runs had it within ±2%.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787719886
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787709681)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4787709681-651-d6b6g 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (0ffbf26319f13b931c40678b0480c6a3ea9cbcaa) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..0ffbf26319f13b931c40678b0480c6a3ea9cbcaa)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_clickbench
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783400488
run benchmark arrow_reader_clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783522864
run benchmark arrow_reader_row_filter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4817713621
I split mask-backed row selection logic into `selection/boolean.rs` to keep `selection/mod.rs(selection.rs`) simpler and more focused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3608539782
##########
parquet/src/arrow/arrow_reader/selection/boolean.rs:
##########
@@ -0,0 +1,962 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{RowSelection, RowSelectionInner, RowSelector};
+use crate::errors::ParquetError;
+use arrow_array::BooleanArray;
+use arrow_buffer::bit_iterator::BitSliceIterator;
+use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder};
+use std::cmp::Ordering;
+use std::sync::OnceLock;
+
+/// Mask-backed [`RowSelection`] storage.
+///
+/// `selectors` is only populated if callers use the borrowed [`RowSelection::iter`]
+/// compatibility API. Internal paths that can stream or consume the bitmap avoid
+/// this cache.
+#[derive(Debug)]
+pub(crate) struct MaskSelection {
+ mask: BooleanBuffer,
+ selectors: OnceLock<Vec<RowSelector>>,
+}
+
+impl MaskSelection {
+ pub(super) fn new(mask: BooleanBuffer) -> Self {
+ Self {
+ mask,
+ selectors: OnceLock::new(),
+ }
+ }
+
+ pub(crate) fn mask(&self) -> &BooleanBuffer {
+ &self.mask
+ }
+
+ pub(crate) fn into_mask(self) -> BooleanBuffer {
+ let Self { mask, .. } = self;
+ mask
+ }
+
+ pub(super) fn selectors(&self) -> &[RowSelector] {
+ self.selectors
+ .get_or_init(|| mask_to_selectors(&self.mask))
+ .as_slice()
+ }
+}
+
+impl Clone for MaskSelection {
+ fn clone(&self) -> Self {
+ Self::new(self.mask.clone())
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`], yielding owned [`RowSelector`]s
+/// without allocation.
+///
+/// Useful as a zero-cost alternative to [`RowSelection::iter`] for mask-backed
+/// selections, via [`RowSelection::as_mask`]:
+///
+/// ```ignore
+/// if let Some(mask) = selection.as_mask() {
+/// for run in MaskRunIter::new(mask) { ... }
+/// }
+/// ```
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ /// Create a streaming RLE iterator over a [`BooleanBuffer`].
+ pub fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
+ if let Some(p) = self.pending.take() {
+ return Some(p);
+ }
+ if self.finished {
+ return None;
+ }
+ match self.slices.next() {
+ Some((start, end)) => {
+ let select = RowSelector::select(end - start);
+ if start > self.cursor {
+ let skip = RowSelector::skip(start - self.cursor);
+ self.pending = Some(select);
+ self.cursor = end;
+ Some(skip)
+ } else {
+ self.cursor = end;
+ Some(select)
+ }
+ }
+ None => {
+ self.finished = true;
+ if self.cursor < self.total {
+ let skip = RowSelector::skip(self.total - self.cursor);
+ self.cursor = self.total;
+ Some(skip)
+ } else {
+ None
+ }
+ }
+ }
+ }
+}
+
+/// Cursor for iterating a mask-backed [`RowSelection`]
+///
+/// This is best for dense selections where there are many small skips
+/// or selections. For example, selecting every other row.
+#[derive(Debug)]
+pub struct MaskCursor {
+ pub(super) mask: BooleanBuffer,
+ /// Current absolute offset into the selection
+ pub(super) position: usize,
+}
+
+impl MaskCursor {
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ self.position >= self.mask.len()
+ }
+
+ /// Advance through the mask representation, producing the next chunk summary
+ pub fn next_mask_chunk(&mut self, batch_size: usize) -> Option<MaskChunk> {
+ let (initial_skip, chunk_rows, selected_rows, mask_start, end_position) = {
+ let mask = &self.mask;
+
+ if self.position >= mask.len() {
+ return None;
+ }
+
+ let start_position = self.position;
+ let mut cursor = start_position;
+ let mut initial_skip = 0;
+
+ while cursor < mask.len() && !mask.value(cursor) {
+ initial_skip += 1;
+ cursor += 1;
+ }
+
+ let mask_start = cursor;
+ let mut chunk_rows = 0;
+ let mut selected_rows = 0;
+
+ // Advance until enough rows have been selected to satisfy the batch size,
+ // or until the mask is exhausted. This mirrors the behaviour of the legacy
+ // `RowSelector` queue-based iteration.
+ while cursor < mask.len() && selected_rows < batch_size {
+ chunk_rows += 1;
+ if mask.value(cursor) {
+ selected_rows += 1;
+ }
+ cursor += 1;
+ }
+
+ (initial_skip, chunk_rows, selected_rows, mask_start, cursor)
+ };
+
+ self.position = end_position;
+
+ Some(MaskChunk {
+ initial_skip,
+ chunk_rows,
+ selected_rows,
+ mask_start,
+ })
+ }
+
+ /// Materialise the boolean values for a mask-backed chunk
+ pub fn mask_values_for(&self, chunk: &MaskChunk) -> Result<BooleanArray, ParquetError> {
+ if chunk.mask_start.saturating_add(chunk.chunk_rows) > self.mask.len() {
+ return Err(ParquetError::General(
+ "Internal Error: MaskChunk exceeds mask length".to_string(),
+ ));
+ }
+ Ok(BooleanArray::from(
+ self.mask.slice(chunk.mask_start, chunk.chunk_rows),
+ ))
+ }
+}
+
+/// Result of computing the next chunk to read when using a [`MaskCursor`]
+#[derive(Debug)]
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+/// Materialize a [`BooleanBuffer`] into its RLE form.
+pub(crate) fn mask_to_selectors(mask: &BooleanBuffer) -> Vec<RowSelector> {
+ let total_rows = mask.len();
+ if total_rows == 0 {
+ return Vec::new();
+ }
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut last_end = 0;
+ for (start, end) in mask.set_slices() {
+ if start > last_end {
+ selectors.push(RowSelector::skip(start - last_end));
+ }
+ selectors.push(RowSelector::select(end - start));
+ last_end = end;
+ }
+ if last_end != total_rows {
+ selectors.push(RowSelector::skip(total_rows - last_end));
+ }
+ selectors
+}
+
+pub(super) fn mask_run_count(mask: &BooleanBuffer) -> usize {
+ let total_rows = mask.len();
+ if total_rows == 0 {
+ return 0;
+ }
+
+ let mut run_count = 0;
+ let mut last_end = 0;
+ for (start, end) in mask.set_slices() {
+ if start > last_end {
+ run_count += 1;
+ }
+ if end > start {
+ run_count += 1;
+ }
+ last_end = end;
+ }
+ if last_end < total_rows {
+ run_count += 1;
+ }
+
+ run_count
+}
+
+/// Bitwise AND of two mask-backed selections. Longer side's tail passes through.
+pub(super) fn intersect_masks(l: &BooleanBuffer, r: &BooleanBuffer) -> BooleanBuffer {
+ if l.len() == r.len() {
+ return l & r;
+ }
+ let common = l.len().min(r.len());
+ let head = &l.slice(0, common) & &r.slice(0, common);
+ let (longer, longer_len) = if l.len() > r.len() {
+ (l, l.len())
+ } else {
+ (r, r.len())
+ };
+ let tail = longer.slice(common, longer_len - common);
+ let mut builder = BooleanBufferBuilder::new(longer_len);
+ builder.append_buffer(&head);
+ builder.append_buffer(&tail);
+ builder.finish()
+}
+
+/// Bitwise OR of two mask-backed selections. Longer side's tail passes through.
+pub(super) fn union_masks(l: &BooleanBuffer, r: &BooleanBuffer) -> BooleanBuffer {
+ if l.len() == r.len() {
+ return l | r;
+ }
+ let common = l.len().min(r.len());
+ let head = &l.slice(0, common) | &r.slice(0, common);
+ let (longer, longer_len) = if l.len() > r.len() {
+ (l, l.len())
+ } else {
+ (r, r.len())
+ };
+ let tail = longer.slice(common, longer_len - common);
+ let mut builder = BooleanBufferBuilder::new(longer_len);
+ builder.append_buffer(&head);
+ builder.append_buffer(&tail);
+ builder.finish()
+}
+
+/// Applies `other` to the selected rows of `mask`, preserving the original row domain.
+pub(super) fn and_then_mask(mask: &BooleanBuffer, other: &RowSelection) -> BooleanBuffer {
+ match &other.inner {
+ RowSelectionInner::Mask(other_mask) => and_then_masks(mask, other_mask.mask()),
+ RowSelectionInner::Selectors(selectors) => {
+ and_then_mask_from_selectors(mask, selectors.iter().copied())
+ }
+ }
+}
+
+fn and_then_mask_from_selectors<I>(mask: &BooleanBuffer, other: I) -> BooleanBuffer
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut builder = BooleanBufferBuilder::new(mask.len());
+ let mut other_iter = other.into_iter();
+ let mut current = other_iter.next();
+ let mut cursor = 0usize;
+
+ // Iterate only over the set positions in `mask`; the gaps of unset bits
+ // are filled in bulk with `append_n` instead of bit-by-bit.
+ for set_idx in mask.set_indices() {
+ if set_idx > cursor {
+ builder.append_n(set_idx - cursor, false);
+ }
+ cursor = set_idx + 1;
+
+ while current.as_ref().is_some_and(|s| s.row_count == 0) {
+ current = other_iter.next();
+ }
+ let selector = current
+ .as_mut()
+ .expect("selection contains less than the number of selected rows");
+ let selected = !selector.skip;
+ selector.row_count -= 1;
+ builder.append(selected);
+ }
+ if cursor < mask.len() {
+ builder.append_n(mask.len() - cursor, false);
+ }
+
+ if current.is_some_and(|s| s.row_count != 0) || other_iter.any(|s| s.row_count != 0) {
+ panic!("selection exceeds the number of selected rows");
+ }
+
+ builder.finish()
+}
+
+fn and_then_masks(mask: &BooleanBuffer, other: &BooleanBuffer) -> BooleanBuffer {
+ let selected_count = mask.count_set_bits();
+ match other.len().cmp(&selected_count) {
+ Ordering::Less => panic!("selection contains less than the number of selected rows"),
+ Ordering::Greater => panic!("selection exceeds the number of selected rows"),
+ Ordering::Equal => {}
+ }
+
+ let other_true_count = other.count_set_bits();
+ if other_true_count == 0 {
+ return BooleanBuffer::new_unset(mask.len());
+ }
+ if other_true_count == selected_count {
+ return mask.clone();
+ }
+
+ let mut builder = BooleanBufferBuilder::new(mask.len());
+ let mut outer_set_indices = mask.set_indices();
+ let mut next_selected_ordinal = 0usize;
+ let mut cursor = 0usize;
+
+ for selected_ordinal in other.set_indices() {
+ let skip = selected_ordinal - next_selected_ordinal;
+ let set_idx = outer_set_indices
+ .nth(skip)
+ .expect("validated other length matches selected row count");
+ if set_idx > cursor {
+ builder.append_n(set_idx - cursor, false);
+ }
+ builder.append(true);
+ cursor = set_idx + 1;
+ next_selected_ordinal = selected_ordinal + 1;
+ }
+
+ if cursor < mask.len() {
+ builder.append_n(mask.len() - cursor, false);
+ }
+
+ builder.finish()
+}
+
+/// Split a mask into `(head, tail)` at `row_count`, preserving an empty mask tail
+/// when the split point is past the end.
+pub(super) fn split_off_mask(
+ mask: BooleanBuffer,
+ row_count: usize,
+) -> (BooleanBuffer, BooleanBuffer) {
+ let total = mask.len();
+ if row_count >= total {
+ return (mask, BooleanBuffer::new_unset(0));
+ }
+
+ let head = mask.slice(0, row_count);
+ let tail = mask.slice(row_count, total - row_count);
+ (head, tail)
+}
+
+/// Trims trailing unset bits from a mask-backed selection.
+pub(super) fn trim_mask(mask: &BooleanBuffer) -> Option<BooleanBuffer> {
Review Comment:
Could we avoid walking every selected bit here?
`trim_mask` first computes a full popcount and then calls
`find_nth_set_bit_position(0, popcount)`. That helper is implemented using
`set_indices().nth(n - 1)`, and `BitIndexIterator` does not override `nth`.
Therefore a dense mask enumerates every selected row just to find its final set
bit.
`ReadPlanBuilder::build` calls `trim()` for every selection, so an all-set or
near-dense 1B-row bitmap can execute up to 1B iterator steps before decoding
starts, even when there is nothing to trim.
Could we at least fast-path a set final bit, and use a reverse word scan for the
remaining case?
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -422,96 +662,151 @@ impl RowSelection {
///
/// returned: NYYYYYNNYYNYN
pub fn union(&self, other: &Self) -> Self {
- union_row_selections(&self.selectors, &other.selectors)
+ match &self.inner {
+ RowSelectionInner::Mask(l) => match &other.inner {
+ RowSelectionInner::Mask(r) => {
+ Self::from_boolean_buffer(union_masks(l.mask(), r.mask()))
+ }
+ RowSelectionInner::Selectors(r) => {
+ let l = mask_to_selectors(l.mask());
+ union_row_selections(&l, r)
+ }
+ },
+ RowSelectionInner::Selectors(l) => match &other.inner {
+ RowSelectionInner::Mask(r) => {
+ let r = mask_to_selectors(r.mask());
+ union_row_selections(l, &r)
+ }
+ RowSelectionInner::Selectors(r) => union_row_selections(l, r),
+ },
+ }
}
/// Returns `true` if this [`RowSelection`] selects any rows
pub fn selects_any(&self) -> bool {
- self.selectors.iter().any(|x| !x.skip)
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => s.iter().any(|x| !x.skip),
+ RowSelectionInner::Mask(m) => m.mask().set_indices().next().is_some(),
+ }
}
/// Trims this [`RowSelection`] removing any trailing skips
pub(crate) fn trim(mut self) -> Self {
- while self.selectors.last().map(|x| x.skip).unwrap_or(false) {
- self.selectors.pop();
+ if let RowSelectionInner::Mask(m) = &self.inner {
+ if let Some(mask) = trim_mask(m.mask()) {
+ return Self::from_boolean_buffer(mask);
+ }
+ return self;
+ }
+ let selectors = self.selectors_mut();
+ while selectors.last().map(|x| x.skip).unwrap_or(false) {
+ selectors.pop();
}
self
}
/// Applies an offset to this [`RowSelection`], skipping the first `offset` selected rows
- pub(crate) fn offset(mut self, offset: usize) -> Self {
+ pub(crate) fn offset(self, offset: usize) -> Self {
if offset == 0 {
return self;
}
+ let mut selectors = match self.inner {
+ RowSelectionInner::Mask(mask) => {
+ return Self::from_boolean_buffer(offset_mask((*mask).into_mask(), offset));
+ }
+ RowSelectionInner::Selectors(selectors) => selectors,
+ };
let mut selected_count = 0;
let mut skipped_count = 0;
// Find the index where the selector exceeds the row count
- let find = self
- .selectors
- .iter()
- .position(|selector| match selector.skip {
- true => {
- skipped_count += selector.row_count;
- false
- }
- false => {
- selected_count += selector.row_count;
- selected_count > offset
- }
- });
+ let find = selectors.iter().position(|selector| match selector.skip {
+ true => {
+ skipped_count += selector.row_count;
+ false
+ }
+ false => {
+ selected_count += selector.row_count;
+ selected_count > offset
+ }
+ });
let split_idx = match find {
Some(idx) => idx,
None => {
- self.selectors.clear();
- return self;
+ selectors.clear();
+ return Self::from_selectors(selectors);
}
};
- let mut selectors = Vec::with_capacity(self.selectors.len() - split_idx + 1);
- selectors.push(RowSelector::skip(skipped_count + offset));
- selectors.push(RowSelector::select(selected_count - offset));
- selectors.extend_from_slice(&self.selectors[split_idx + 1..]);
+ let mut new_selectors = Vec::with_capacity(selectors.len() - split_idx + 1);
+ new_selectors.push(RowSelector::skip(skipped_count + offset));
+ new_selectors.push(RowSelector::select(selected_count - offset));
+ new_selectors.extend_from_slice(&selectors[split_idx + 1..]);
- Self { selectors }
+ Self::from_selectors(new_selectors)
}
/// Limit this [`RowSelection`] to only select `limit` rows
- pub(crate) fn limit(mut self, mut limit: usize) -> Self {
+ pub(crate) fn limit(self, mut limit: usize) -> Self {
+ let mut selectors = match self.inner {
+ RowSelectionInner::Mask(mask) => {
+ return Self::from_boolean_buffer(limit_mask((*mask).into_mask(), limit));
+ }
+ RowSelectionInner::Selectors(selectors) => selectors,
+ };
if limit == 0 {
- self.selectors.clear();
+ selectors.clear();
}
- for (idx, selection) in self.selectors.iter_mut().enumerate() {
+ for (idx, selection) in selectors.iter_mut().enumerate() {
if !selection.skip {
if selection.row_count >= limit {
selection.row_count = limit;
- self.selectors.truncate(idx + 1);
+ selectors.truncate(idx + 1);
break;
} else {
limit -= selection.row_count;
}
}
}
- self
+ Self::from_selectors(selectors)
}
- /// Returns an iterator over the [`RowSelector`]s for this
- /// [`RowSelection`].
- pub fn iter(&self) -> impl Iterator<Item = &RowSelector> {
- self.selectors.iter()
+ /// Returns a borrowed iterator yielding the [`RowSelector`]s for this selection.
+ ///
+ /// Mask-backed selections materialize a `Vec<RowSelector>` cache on first
+ /// call (one allocation, `O(set_slices)` work) so the iterator can hand out
+ /// `&RowSelector`; the cache is not copied on clone. For single-pass walks
+ /// over mask-backed selections, prefer streaming directly via
+ /// [`Self::as_mask`] + [`MaskRunIter::new`] — that path is allocation-free
+ /// and avoids populating the cache.
+ pub fn iter(&self) -> RowSelectionIter<'_> {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => RowSelectionIter(s.iter()),
+ RowSelectionInner::Mask(m) => RowSelectionIter(m.selectors().iter()),
+ }
}
/// Returns the number of selected rows
pub fn row_count(&self) -> usize {
Review Comment:
`row_count()` is a full bitmap popcount for mask-backed selections. The
row-group frontier calls it on the entire shrinking tail before every row group,
then again on the split head. A caller using `peek_next_row_group` repeats the
same traversal on a clone.
This makes the mask path O(number_of_row_groups * bitmap_size). For example, a
1B-row bitmap is about 125 MB; with 100 equal row groups, the shrinking-tail
checks alone rescan about 6.3 GB of bitmap data.
Could `MaskSelection` lazily cache the selected count and propagate head/tail
counts through `split_off`, or could the frontier explicitly track the remaining
selected count?
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -412,7 +637,22 @@ impl RowSelection {
///
/// returned: NNNNNNNNYYNYN
pub fn intersection(&self, other: &Self) -> Self {
- intersect_row_selections(&self.selectors, &other.selectors)
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Mask(l), RowSelectionInner::Mask(r)) => {
+ Self::from_boolean_buffer(intersect_masks(l.mask(), r.mask()))
+ }
+ (RowSelectionInner::Selectors(l), RowSelectionInner::Selectors(r)) => {
+ intersect_row_selections(l, r)
+ }
+ (RowSelectionInner::Selectors(l), RowSelectionInner::Mask(r)) => {
Review Comment:
Could the mixed-representation path avoid eagerly converting the entire mask
with `mask_to_selectors`? The same issue also applies to the mixed `union` arms
below.
A natural target use case is an externally supplied fragmented bitmap composed
with selector-backed page-index pruning. DataFusion's
`ParquetAccessPlan::scan_selection` performs exactly this kind of
`intersection`. An alternating 3M-row mask can therefore materialize up to 3M
`RowSelector`s before reading begins, reintroducing the memory cost this PR is
intended to avoid.
Could this either stream the mixed operation, or choose which side to convert
based on the two representations' shapes and preserve mask backing when that is
cheaper? A mixed bitmap + page-selection benchmark would also protect this
case.
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -135,12 +149,297 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(Box<MaskSelection>),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.mask().len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a.mask() == b.mask(),
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.mask().len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.mask().set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Borrowed iterator over the [`RowSelector`]s of a [`RowSelection`].
+#[derive(Debug)]
+pub struct RowSelectionIter<'a>(std::slice::Iter<'a, RowSelector>);
+
+impl<'a> Iterator for RowSelectionIter<'a> {
+ type Item = &'a RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
+ self.0.next()
+ }
+}
+
+#[inline]
+fn scan_ranges_from_selectors<I>(selectors: I, page_locations: &[PageLocation]) -> Vec<Range<u64>>
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut ranges: Vec<Range<u64>> = vec![];
+ let mut row_offset = 0;
+
+ let mut pages = page_locations.iter().peekable();
+ let mut selectors = selectors.into_iter();
+ let mut current_selector = selectors.next();
+ let mut current_page = pages.next();
+
+ let mut current_page_included = false;
+
+ while let Some((selector, page)) = current_selector.as_mut().zip(current_page) {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ current_page_included = true;
+ }
+
+ if let Some(next_page) = pages.peek() {
+ if row_offset + selector.row_count > next_page.first_row_index as usize {
+ let remaining_in_page = next_page.first_row_index as usize - row_offset;
+ selector.row_count -= remaining_in_page;
+ row_offset += remaining_in_page;
+ current_page = pages.next();
+ current_page_included = false;
+
+ continue;
+ } else {
+ if row_offset + selector.row_count == next_page.first_row_index as usize {
+ current_page = pages.next();
+ current_page_included = false;
+ }
+ row_offset += selector.row_count;
+ current_selector = selectors.next();
+ }
+ } else {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ }
+ current_selector = selectors.next()
+ }
+ }
+
+ ranges
+}
+
+#[inline]
+fn expand_to_batch_boundaries_from_selectors<I>(
+ selectors: I,
+ batch_size: usize,
+ total_rows: usize,
+) -> RowSelection
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut expanded_ranges = Vec::new();
+ let mut row_offset = 0;
+
+ for selector in selectors {
+ if selector.skip {
+ row_offset += selector.row_count;
+ } else {
+ let start = row_offset;
+ let end = row_offset + selector.row_count;
+
+ // Expand start to batch boundary
+ let expanded_start = (start / batch_size) * batch_size;
+ // Expand end to batch boundary
+ let expanded_end = end.div_ceil(batch_size) * batch_size;
+ let expanded_end = expanded_end.min(total_rows);
+
+ expanded_ranges.push(expanded_start..expanded_end);
+ row_offset += selector.row_count;
+ }
+ }
+
+ // Sort ranges by start position
+ expanded_ranges.sort_by_key(|range| range.start);
+
+ // Merge overlapping or consecutive ranges
+ let mut merged_ranges: Vec<Range<usize>> = Vec::new();
+ for range in expanded_ranges {
+ if let Some(last) = merged_ranges.last_mut() {
+ if range.start <= last.end {
+ // Overlapping or consecutive - merge them
+ last.end = last.end.max(range.end);
+ } else {
+ // No overlap - add new range
+ merged_ranges.push(range);
+ }
+ } else {
+ // First range
+ merged_ranges.push(range);
+ }
+ }
+
+ RowSelection::from_consecutive_ranges(merged_ranges.into_iter(), total_rows)
}
impl RowSelection {
+ fn from_selectors(selectors: Vec<RowSelector>) -> Self {
+ Self {
+ inner: RowSelectionInner::Selectors(selectors),
+ }
+ }
+
+ /// Create a [`RowSelection`] from a packed [`BooleanBuffer`].
+ ///
+ /// Each set bit selects a row, each unset bit skips one. Unlike
+ /// [`Self::from_filters`], the bitmap is kept as-is rather than
+ /// eagerly run-length-encoded. [`Self::iter`] materializes and caches the
+ /// RLE form on first use; use [`MaskRunIter`] to stream the RLE form
+ /// directly from the bitmap.
+ pub fn from_boolean_buffer(mask: BooleanBuffer) -> Self {
+ Self {
+ inner: RowSelectionInner::Mask(Box::new(MaskSelection::new(mask))),
+ }
+ }
+
+ /// Returns the underlying mask if this selection is mask-backed.
+ ///
+ /// Public so that engines composing selections (e.g. DataFusion's
+ /// `ParquetAccessPlan::into_overall_row_selection`) can concatenate
+ /// mask-backed selections without materialising the RLE form.
+ pub fn as_mask(&self) -> Option<&BooleanBuffer> {
+ match &self.inner {
+ RowSelectionInner::Mask(m) => Some(m.mask()),
+ _ => None,
+ }
+ }
+
+ /// Consume the selection and return its internal storage.
+ pub(crate) fn into_inner(self) -> RowSelectionInner {
+ self.inner
+ }
+
+ /// Choose the automatic materialisation strategy without converting between
+ /// selector and mask backing.
+ #[inline]
+ pub(crate) fn auto_selection_strategy(&self, threshold: usize) -> RowSelectionStrategy {
+ let (total_rows, effective_count) = match &self.inner {
+ RowSelectionInner::Selectors(selectors) => {
+ selectors.iter().fold((0usize, 0usize), |(rows, count), s| {
+ if s.row_count > 0 {
+ (rows + s.row_count, count + 1)
+ } else {
+ (rows, count)
+ }
+ })
+ }
+ RowSelectionInner::Mask(mask) => {
+ let mask = mask.mask();
+ let total_rows = mask.len();
+ (total_rows, mask_run_count(mask))
+ }
+ };
+
+ if effective_count == 0 {
+ return RowSelectionStrategy::Mask;
+ }
+
+ if total_rows < effective_count.saturating_mul(threshold) {
+ RowSelectionStrategy::Mask
+ } else {
+ RowSelectionStrategy::Selectors
+ }
+ }
+
+ #[cfg(test)]
+ fn selectors(&self) -> Vec<RowSelector> {
+ self.iter().copied().collect()
+ }
+
+ fn into_selectors_vec(self) -> Vec<RowSelector> {
Review Comment:
Could we consume the lazy selector cache here instead of always materializing
the mask again?
`RowSelection::iter()` populates `MaskSelection::selectors`, but
`into_selectors_vec()` ignores that cache and calls `mask_to_selectors()` a
second time. Current DataFusion does exactly this sequence in
`ParquetAccessPlan::into_overall_row_selection`: it first calls `iter()` for
validation and then consumes the same selection with `selection.into()`.
For an alternating 3M-row mask, each selector Vec has a 64 MiB capacity. This
path therefore performs a second full scan and can temporarily retain about
128 MiB of selector storage.
Could `MaskSelection` provide an owning `into_selectors()` that uses
`OnceLock::into_inner()` when initialized, and share it with the cursor
conversion path?
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -135,12 +149,297 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(Box<MaskSelection>),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.mask().len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a.mask() == b.mask(),
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.mask().len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.mask().set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Borrowed iterator over the [`RowSelector`]s of a [`RowSelection`].
+#[derive(Debug)]
+pub struct RowSelectionIter<'a>(std::slice::Iter<'a, RowSelector>);
+
+impl<'a> Iterator for RowSelectionIter<'a> {
+ type Item = &'a RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
Review Comment:
Could `RowSelectionIter` preserve the underlying slice iterator's optimized
methods?
The previous `iter()` implementation delegated directly to `slice::Iter`, so
its exact `size_hint` and specialized `nth` / `count` implementations were used.
This wrapper only forwards `next()`, which falls back to `(0, None)` for
`size_hint`; consequently, existing selector-backed code such as
`selection.iter().copied().collect::<Vec<_>>()` now repeatedly grows and copies
the Vec, while `count()` and `nth()` become element-by-element walks.
Since both backing types expose a slice iterator after any lazy materialization,
could this delegate `size_hint`, `nth`, `count`, and `last`, and ideally
implement `ExactSizeIterator` (plus `DoubleEndedIterator` / `FusedIterator`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3610572627
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -422,96 +662,151 @@ impl RowSelection {
///
/// returned: NYYYYYNNYYNYN
pub fn union(&self, other: &Self) -> Self {
- union_row_selections(&self.selectors, &other.selectors)
+ match &self.inner {
+ RowSelectionInner::Mask(l) => match &other.inner {
+ RowSelectionInner::Mask(r) => {
+ Self::from_boolean_buffer(union_masks(l.mask(), r.mask()))
+ }
+ RowSelectionInner::Selectors(r) => {
+ let l = mask_to_selectors(l.mask());
+ union_row_selections(&l, r)
+ }
+ },
+ RowSelectionInner::Selectors(l) => match &other.inner {
+ RowSelectionInner::Mask(r) => {
+ let r = mask_to_selectors(r.mask());
+ union_row_selections(l, &r)
+ }
+ RowSelectionInner::Selectors(r) => union_row_selections(l, r),
+ },
+ }
}
/// Returns `true` if this [`RowSelection`] selects any rows
pub fn selects_any(&self) -> bool {
- self.selectors.iter().any(|x| !x.skip)
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => s.iter().any(|x| !x.skip),
+ RowSelectionInner::Mask(m) => m.mask().set_indices().next().is_some(),
+ }
}
/// Trims this [`RowSelection`] removing any trailing skips
pub(crate) fn trim(mut self) -> Self {
- while self.selectors.last().map(|x| x.skip).unwrap_or(false) {
- self.selectors.pop();
+ if let RowSelectionInner::Mask(m) = &self.inner {
+ if let Some(mask) = trim_mask(m.mask()) {
+ return Self::from_boolean_buffer(mask);
+ }
+ return self;
+ }
+ let selectors = self.selectors_mut();
+ while selectors.last().map(|x| x.skip).unwrap_or(false) {
+ selectors.pop();
}
self
}
/// Applies an offset to this [`RowSelection`], skipping the first `offset` selected rows
- pub(crate) fn offset(mut self, offset: usize) -> Self {
+ pub(crate) fn offset(self, offset: usize) -> Self {
if offset == 0 {
return self;
}
+ let mut selectors = match self.inner {
+ RowSelectionInner::Mask(mask) => {
+ return Self::from_boolean_buffer(offset_mask((*mask).into_mask(), offset));
+ }
+ RowSelectionInner::Selectors(selectors) => selectors,
+ };
let mut selected_count = 0;
let mut skipped_count = 0;
// Find the index where the selector exceeds the row count
- let find = self
- .selectors
- .iter()
- .position(|selector| match selector.skip {
- true => {
- skipped_count += selector.row_count;
- false
- }
- false => {
- selected_count += selector.row_count;
- selected_count > offset
- }
- });
+ let find = selectors.iter().position(|selector| match selector.skip {
+ true => {
+ skipped_count += selector.row_count;
+ false
+ }
+ false => {
+ selected_count += selector.row_count;
+ selected_count > offset
+ }
+ });
let split_idx = match find {
Some(idx) => idx,
None => {
- self.selectors.clear();
- return self;
+ selectors.clear();
+ return Self::from_selectors(selectors);
}
};
- let mut selectors = Vec::with_capacity(self.selectors.len() - split_idx + 1);
- selectors.push(RowSelector::skip(skipped_count + offset));
- selectors.push(RowSelector::select(selected_count - offset));
- selectors.extend_from_slice(&self.selectors[split_idx + 1..]);
+ let mut new_selectors = Vec::with_capacity(selectors.len() - split_idx + 1);
+ new_selectors.push(RowSelector::skip(skipped_count + offset));
+ new_selectors.push(RowSelector::select(selected_count - offset));
+ new_selectors.extend_from_slice(&selectors[split_idx + 1..]);
- Self { selectors }
+ Self::from_selectors(new_selectors)
}
/// Limit this [`RowSelection`] to only select `limit` rows
- pub(crate) fn limit(mut self, mut limit: usize) -> Self {
+ pub(crate) fn limit(self, mut limit: usize) -> Self {
+ let mut selectors = match self.inner {
+ RowSelectionInner::Mask(mask) => {
+ return Self::from_boolean_buffer(limit_mask((*mask).into_mask(), limit));
+ }
+ RowSelectionInner::Selectors(selectors) => selectors,
+ };
if limit == 0 {
- self.selectors.clear();
+ selectors.clear();
}
- for (idx, selection) in self.selectors.iter_mut().enumerate() {
+ for (idx, selection) in selectors.iter_mut().enumerate() {
if !selection.skip {
if selection.row_count >= limit {
selection.row_count = limit;
- self.selectors.truncate(idx + 1);
+ selectors.truncate(idx + 1);
break;
} else {
limit -= selection.row_count;
}
}
}
- self
+ Self::from_selectors(selectors)
}
- /// Returns an iterator over the [`RowSelector`]s for this
- /// [`RowSelection`].
- pub fn iter(&self) -> impl Iterator<Item = &RowSelector> {
- self.selectors.iter()
+ /// Returns a borrowed iterator yielding the [`RowSelector`]s for this selection.
+ ///
+ /// Mask-backed selections materialize a `Vec<RowSelector>` cache on first
+ /// call (one allocation, `O(set_slices)` work) so the iterator can hand out
+ /// `&RowSelector`; the cache is not copied on clone. For single-pass walks
+ /// over mask-backed selections, prefer streaming directly via
+ /// [`Self::as_mask`] + [`MaskRunIter::new`] — that path is allocation-free
+ /// and avoids populating the cache.
+ pub fn iter(&self) -> RowSelectionIter<'_> {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => RowSelectionIter(s.iter()),
+ RowSelectionInner::Mask(m) => RowSelectionIter(m.selectors().iter()),
+ }
}
/// Returns the number of selected rows
pub fn row_count(&self) -> usize {
Review Comment:
[496d7cb](https://github.com/apache/arrow-rs/pull/10141/commits/496d7cb62a2aff8064ac35c2bcdc5d682e78d291)
##########
parquet/src/arrow/arrow_reader/selection/boolean.rs:
##########
@@ -0,0 +1,962 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{RowSelection, RowSelectionInner, RowSelector};
+use crate::errors::ParquetError;
+use arrow_array::BooleanArray;
+use arrow_buffer::bit_iterator::BitSliceIterator;
+use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder};
+use std::cmp::Ordering;
+use std::sync::OnceLock;
+
+/// Mask-backed [`RowSelection`] storage.
+///
+/// `selectors` is only populated if callers use the borrowed [`RowSelection::iter`]
+/// compatibility API. Internal paths that can stream or consume the bitmap avoid
+/// this cache.
+#[derive(Debug)]
+pub(crate) struct MaskSelection {
+ mask: BooleanBuffer,
+ selectors: OnceLock<Vec<RowSelector>>,
+}
+
+impl MaskSelection {
+ pub(super) fn new(mask: BooleanBuffer) -> Self {
+ Self {
+ mask,
+ selectors: OnceLock::new(),
+ }
+ }
+
+ pub(crate) fn mask(&self) -> &BooleanBuffer {
+ &self.mask
+ }
+
+ pub(crate) fn into_mask(self) -> BooleanBuffer {
+ let Self { mask, .. } = self;
+ mask
+ }
+
+ pub(super) fn selectors(&self) -> &[RowSelector] {
+ self.selectors
+ .get_or_init(|| mask_to_selectors(&self.mask))
+ .as_slice()
+ }
+}
+
+impl Clone for MaskSelection {
+ fn clone(&self) -> Self {
+ Self::new(self.mask.clone())
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`], yielding owned [`RowSelector`]s
+/// without allocation.
+///
+/// Useful as a zero-cost alternative to [`RowSelection::iter`] for mask-backed
+/// selections, via [`RowSelection::as_mask`]:
+///
+/// ```ignore
+/// if let Some(mask) = selection.as_mask() {
+/// for run in MaskRunIter::new(mask) { ... }
+/// }
+/// ```
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ /// Create a streaming RLE iterator over a [`BooleanBuffer`].
+ pub fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
+ if let Some(p) = self.pending.take() {
+ return Some(p);
+ }
+ if self.finished {
+ return None;
+ }
+ match self.slices.next() {
+ Some((start, end)) => {
+ let select = RowSelector::select(end - start);
+ if start > self.cursor {
+ let skip = RowSelector::skip(start - self.cursor);
+ self.pending = Some(select);
+ self.cursor = end;
+ Some(skip)
+ } else {
+ self.cursor = end;
+ Some(select)
+ }
+ }
+ None => {
+ self.finished = true;
+ if self.cursor < self.total {
+ let skip = RowSelector::skip(self.total - self.cursor);
+ self.cursor = self.total;
+ Some(skip)
+ } else {
+ None
+ }
+ }
+ }
+ }
+}
+
+/// Cursor for iterating a mask-backed [`RowSelection`]
+///
+/// This is best for dense selections where there are many small skips
+/// or selections. For example, selecting every other row.
+#[derive(Debug)]
+pub struct MaskCursor {
+ pub(super) mask: BooleanBuffer,
+ /// Current absolute offset into the selection
+ pub(super) position: usize,
+}
+
+impl MaskCursor {
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ self.position >= self.mask.len()
+ }
+
+ /// Advance through the mask representation, producing the next chunk summary
+ pub fn next_mask_chunk(&mut self, batch_size: usize) -> Option<MaskChunk> {
+ let (initial_skip, chunk_rows, selected_rows, mask_start, end_position) = {
+ let mask = &self.mask;
+
+ if self.position >= mask.len() {
+ return None;
+ }
+
+ let start_position = self.position;
+ let mut cursor = start_position;
+ let mut initial_skip = 0;
+
+ while cursor < mask.len() && !mask.value(cursor) {
+ initial_skip += 1;
+ cursor += 1;
+ }
+
+ let mask_start = cursor;
+ let mut chunk_rows = 0;
+ let mut selected_rows = 0;
+
+ // Advance until enough rows have been selected to satisfy the batch size,
+ // or until the mask is exhausted. This mirrors the behaviour of the legacy
+ // `RowSelector` queue-based iteration.
+ while cursor < mask.len() && selected_rows < batch_size {
+ chunk_rows += 1;
+ if mask.value(cursor) {
+ selected_rows += 1;
+ }
+ cursor += 1;
+ }
+
+ (initial_skip, chunk_rows, selected_rows, mask_start, cursor)
+ };
+
+ self.position = end_position;
+
+ Some(MaskChunk {
+ initial_skip,
+ chunk_rows,
+ selected_rows,
+ mask_start,
+ })
+ }
+
+ /// Materialise the boolean values for a mask-backed chunk
+ pub fn mask_values_for(&self, chunk: &MaskChunk) -> Result<BooleanArray, ParquetError> {
+ if chunk.mask_start.saturating_add(chunk.chunk_rows) > self.mask.len() {
+ return Err(ParquetError::General(
+ "Internal Error: MaskChunk exceeds mask length".to_string(),
+ ));
+ }
+ Ok(BooleanArray::from(
+ self.mask.slice(chunk.mask_start, chunk.chunk_rows),
+ ))
+ }
+}
+
+/// Result of computing the next chunk to read when using a [`MaskCursor`]
+#[derive(Debug)]
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+/// Materialize a [`BooleanBuffer`] into its RLE form.
+pub(crate) fn mask_to_selectors(mask: &BooleanBuffer) -> Vec<RowSelector> {
+ let total_rows = mask.len();
+ if total_rows == 0 {
+ return Vec::new();
+ }
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut last_end = 0;
+ for (start, end) in mask.set_slices() {
+ if start > last_end {
+ selectors.push(RowSelector::skip(start - last_end));
+ }
+ selectors.push(RowSelector::select(end - start));
+ last_end = end;
+ }
+ if last_end != total_rows {
+ selectors.push(RowSelector::skip(total_rows - last_end));
+ }
+ selectors
+}
+
+pub(super) fn mask_run_count(mask: &BooleanBuffer) -> usize {
+ let total_rows = mask.len();
+ if total_rows == 0 {
+ return 0;
+ }
+
+ let mut run_count = 0;
+ let mut last_end = 0;
+ for (start, end) in mask.set_slices() {
+ if start > last_end {
+ run_count += 1;
+ }
+ if end > start {
+ run_count += 1;
+ }
+ last_end = end;
+ }
+ if last_end < total_rows {
+ run_count += 1;
+ }
+
+ run_count
+}
+
+/// Bitwise AND of two mask-backed selections. Longer side's tail passes through.
+pub(super) fn intersect_masks(l: &BooleanBuffer, r: &BooleanBuffer) -> BooleanBuffer {
+ if l.len() == r.len() {
+ return l & r;
+ }
+ let common = l.len().min(r.len());
+ let head = &l.slice(0, common) & &r.slice(0, common);
+ let (longer, longer_len) = if l.len() > r.len() {
+ (l, l.len())
+ } else {
+ (r, r.len())
+ };
+ let tail = longer.slice(common, longer_len - common);
+ let mut builder = BooleanBufferBuilder::new(longer_len);
+ builder.append_buffer(&head);
+ builder.append_buffer(&tail);
+ builder.finish()
+}
+
+/// Bitwise OR of two mask-backed selections. Longer side's tail passes through.
+pub(super) fn union_masks(l: &BooleanBuffer, r: &BooleanBuffer) -> BooleanBuffer {
+ if l.len() == r.len() {
+ return l | r;
+ }
+ let common = l.len().min(r.len());
+ let head = &l.slice(0, common) | &r.slice(0, common);
+ let (longer, longer_len) = if l.len() > r.len() {
+ (l, l.len())
+ } else {
+ (r, r.len())
+ };
+ let tail = longer.slice(common, longer_len - common);
+ let mut builder = BooleanBufferBuilder::new(longer_len);
+ builder.append_buffer(&head);
+ builder.append_buffer(&tail);
+ builder.finish()
+}
+
+/// Applies `other` to the selected rows of `mask`, preserving the original row domain.
+pub(super) fn and_then_mask(mask: &BooleanBuffer, other: &RowSelection) -> BooleanBuffer {
+ match &other.inner {
+ RowSelectionInner::Mask(other_mask) => and_then_masks(mask, other_mask.mask()),
+ RowSelectionInner::Selectors(selectors) => {
+ and_then_mask_from_selectors(mask, selectors.iter().copied())
+ }
+ }
+}
+
+fn and_then_mask_from_selectors<I>(mask: &BooleanBuffer, other: I) -> BooleanBuffer
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut builder = BooleanBufferBuilder::new(mask.len());
+ let mut other_iter = other.into_iter();
+ let mut current = other_iter.next();
+ let mut cursor = 0usize;
+
+ // Iterate only over the set positions in `mask`; the gaps of unset bits
+ // are filled in bulk with `append_n` instead of bit-by-bit.
+ for set_idx in mask.set_indices() {
+ if set_idx > cursor {
+ builder.append_n(set_idx - cursor, false);
+ }
+ cursor = set_idx + 1;
+
+ while current.as_ref().is_some_and(|s| s.row_count == 0) {
+ current = other_iter.next();
+ }
+ let selector = current
+ .as_mut()
+ .expect("selection contains less than the number of selected rows");
+ let selected = !selector.skip;
+ selector.row_count -= 1;
+ builder.append(selected);
+ }
+ if cursor < mask.len() {
+ builder.append_n(mask.len() - cursor, false);
+ }
+
+ if current.is_some_and(|s| s.row_count != 0) || other_iter.any(|s| s.row_count != 0) {
+ panic!("selection exceeds the number of selected rows");
+ }
+
+ builder.finish()
+}
+
+fn and_then_masks(mask: &BooleanBuffer, other: &BooleanBuffer) -> BooleanBuffer {
+ let selected_count = mask.count_set_bits();
+ match other.len().cmp(&selected_count) {
+ Ordering::Less => panic!("selection contains less than the number of selected rows"),
+ Ordering::Greater => panic!("selection exceeds the number of selected rows"),
+ Ordering::Equal => {}
+ }
+
+ let other_true_count = other.count_set_bits();
+ if other_true_count == 0 {
+ return BooleanBuffer::new_unset(mask.len());
+ }
+ if other_true_count == selected_count {
+ return mask.clone();
+ }
+
+ let mut builder = BooleanBufferBuilder::new(mask.len());
+ let mut outer_set_indices = mask.set_indices();
+ let mut next_selected_ordinal = 0usize;
+ let mut cursor = 0usize;
+
+ for selected_ordinal in other.set_indices() {
+ let skip = selected_ordinal - next_selected_ordinal;
+ let set_idx = outer_set_indices
+ .nth(skip)
+ .expect("validated other length matches selected row count");
+ if set_idx > cursor {
+ builder.append_n(set_idx - cursor, false);
+ }
+ builder.append(true);
+ cursor = set_idx + 1;
+ next_selected_ordinal = selected_ordinal + 1;
+ }
+
+ if cursor < mask.len() {
+ builder.append_n(mask.len() - cursor, false);
+ }
+
+ builder.finish()
+}
+
+/// Split a mask into `(head, tail)` at `row_count`, preserving an empty mask tail
+/// when the split point is past the end.
+pub(super) fn split_off_mask(
+ mask: BooleanBuffer,
+ row_count: usize,
+) -> (BooleanBuffer, BooleanBuffer) {
+ let total = mask.len();
+ if row_count >= total {
+ return (mask, BooleanBuffer::new_unset(0));
+ }
+
+ let head = mask.slice(0, row_count);
+ let tail = mask.slice(row_count, total - row_count);
+ (head, tail)
+}
+
+/// Trims trailing unset bits from a mask-backed selection.
+pub(super) fn trim_mask(mask: &BooleanBuffer) -> Option<BooleanBuffer> {
Review Comment:
solve in [496d7cb](https://github.com/apache/arrow-rs/pull/10141/commits/496d7cb62a2aff8064ac35c2bcdc5d682e78d291)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5074709175
Let;s go go go go!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3465724642
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -135,12 +143,439 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(BooleanBuffer),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a == b,
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Iterator over the [`RowSelector`]s of a [`RowSelection`].
+///
+/// Yields owned [`RowSelector`] values; mask-backed selections stream the
+/// run-length form via [`BitSliceIterator`] without allocation.
+#[derive(Debug)]
+pub enum RowSelectionIter<'a> {
+ Selectors(std::iter::Copied<std::slice::Iter<'a, RowSelector>>),
+ Mask(MaskRunIter<'a>),
+}
+
+impl Iterator for RowSelectionIter<'_> {
+ type Item = RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<RowSelector> {
+ match self {
+ Self::Selectors(it) => it.next(),
+ Self::Mask(it) => it.next(),
+ }
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`].
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
Review Comment:
I updated `RowSelection::iter()` to keep returning `Iterator<Item = &RowSelector>`.
- For selector-backed selections this still borrows from the internal `Vec<RowSelector>`.
- For mask-backed selections I added a lazy selector cache, because the returned `&RowSelector` values need stable storage inside `RowSelection`; otherwise they would point into a temporary `Vec`.
The internal hot paths still avoid this cache and use the bitmap or `MaskRunIter` directly.
the struct is below
```rust
pub struct RowSelection {
inner: RowSelectionInner,
}
pub(crate) enum RowSelectionInner {
Selectors(Vec<RowSelector>),
Mask(Box<MaskSelection>),
}
pub(crate) struct MaskSelection {
mask: BooleanBuffer,
selectors: OnceLock<Vec<RowSelector>>,
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3465724642
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -135,12 +143,439 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(BooleanBuffer),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a == b,
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Iterator over the [`RowSelector`]s of a [`RowSelection`].
+///
+/// Yields owned [`RowSelector`] values; mask-backed selections stream the
+/// run-length form via [`BitSliceIterator`] without allocation.
+#[derive(Debug)]
+pub enum RowSelectionIter<'a> {
+ Selectors(std::iter::Copied<std::slice::Iter<'a, RowSelector>>),
+ Mask(MaskRunIter<'a>),
+}
+
+impl Iterator for RowSelectionIter<'_> {
+ type Item = RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<RowSelector> {
+ match self {
+ Self::Selectors(it) => it.next(),
+ Self::Mask(it) => it.next(),
+ }
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`].
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
Review Comment:
I updated `RowSelection::iter()` to keep returning `Iterator<Item = &RowSelector>`.
- For selector-backed selections this still borrows from the internal `Vec<RowSelector>`.
- For mask-backed selections I added a lazy selector cache, because the returned `&RowSelector` values need stable storage inside `RowSelection`; otherwise they would point into a temporary `Vec`.
The internal hot paths still avoid this cache and use the bitmap or `MaskRunIter` directly.
the struct is below
```rust
pub struct RowSelection {
inner: RowSelectionInner,
}
pub(crate) enum RowSelectionInner {
Selectors(Vec<RowSelector>),
Mask(Box<MaskSelection>),
}
pub(crate) struct MaskSelection {
mask: BooleanBuffer,
selectors: OnceLock<Vec<RowSelector>>,
}
```
I am not fully sure this is the best approach, so suggestions and feedback are welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788450139
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788216344)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_clickbench/async/Q1 1.00 1092.4±5.01µs ? ?/sec 1.01 1098.1±6.39µs ? ?/sec
arrow_reader_clickbench/async/Q10 1.00 6.6±0.03ms ? ?/sec 1.00 6.5±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.00 7.5±0.05ms ? ?/sec 1.00 7.5±0.04ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.00 14.5±0.06ms ? ?/sec 1.00 14.5±0.07ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.00 17.2±0.12ms ? ?/sec 1.00 17.1±0.10ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.00 15.9±0.09ms ? ?/sec 1.00 15.9±0.07ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.00 3.1±0.03ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.10 92.7±1.70ms ? ?/sec 1.00 84.4±0.28ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.00 103.7±4.77ms ? ?/sec 1.02 105.4±6.65ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.00 138.7±6.97ms ? ?/sec 1.00 138.7±1.64ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.00 255.1±1.50ms ? ?/sec 1.02 259.8±10.85ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.00 19.5±0.18ms ? ?/sec 1.00 19.6±0.17ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.01 58.9±0.54ms ? ?/sec 1.00 58.5±0.42ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.00 59.5±0.46ms ? ?/sec 1.00 59.3±0.45ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.00 18.7±0.09ms ? ?/sec 1.00 18.6±0.11ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.00 15.5±0.25ms ? ?/sec 1.00 15.5±0.17ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.00 5.5±0.03ms ? ?/sec 1.00 5.5±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.03 13.9±0.31ms ? ?/sec 1.00 13.6±0.23ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.02 25.2±0.62ms ? ?/sec 1.00 24.7±0.41ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.00 5.8±0.05ms ? ?/sec 1.00 5.8±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.00 5.0±0.03ms ? ?/sec 1.01 5.1±0.04ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.00 3.5±0.02ms ? ?/sec 1.01 3.6±0.05ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q1 1.01 1074.9±7.08µs ? ?/sec 1.00 1064.0±4.25µs ? ?/sec
arrow_reader_clickbench/async_object_store/Q10 1.00 6.4±0.04ms ? ?/sec 1.01 6.5±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q11 1.00 7.4±0.04ms ? ?/sec 1.00 7.4±0.05ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q12 1.00 14.4±0.07ms ? ?/sec 1.00 14.5±0.08ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q13 1.00 17.1±0.14ms ? ?/sec 1.00 17.1±0.12ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q14 1.00 15.9±0.09ms ? ?/sec 1.00 15.9±0.10ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q19 1.00 3.0±0.03ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q20 1.01 72.7±0.64ms ? ?/sec 1.00 72.0±0.51ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q21 1.01 81.6±0.66ms ? ?/sec 1.00 81.0±0.52ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q22 1.02 100.7±0.77ms ? ?/sec 1.00 98.6±0.53ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q23 1.07 244.4±2.29ms ? ?/sec 1.00 229.0±0.43ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q24 1.00 19.1±0.11ms ? ?/sec 1.00 19.1±0.14ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q27 1.02 58.2±0.72ms ? ?/sec 1.00 57.1±0.45ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q28 1.00 58.3±0.71ms ? ?/sec 1.00 58.2±0.60ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q30 1.01 18.4±0.11ms ? ?/sec 1.00 18.3±0.13ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q36 1.01 15.0±0.26ms ? ?/sec 1.00 14.9±0.18ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q37 1.01 5.4±0.03ms ? ?/sec 1.00 5.4±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q38 1.02 13.2±0.25ms ? ?/sec 1.00 13.0±0.18ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q39 1.01 23.7±0.37ms ? ?/sec 1.00 23.5±0.26ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q40 1.01 5.6±0.05ms ? ?/sec 1.00 5.5±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q41 1.01 4.9±0.03ms ? ?/sec 1.00 4.8±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q42 1.01 3.4±0.02ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.00 890.2±1.34µs ? ?/sec 1.00 890.4±1.23µs ? ?/sec
arrow_reader_clickbench/sync/Q10 1.00 5.1±0.01ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.00 6.0±0.02ms ? ?/sec 1.00 6.0±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.00 21.7±0.05ms ? ?/sec 1.00 21.8±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.17 28.7±0.11ms ? ?/sec 1.00 24.6±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.00 23.1±0.07ms ? ?/sec 1.01 23.2±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.01 125.7±0.17ms ? ?/sec 1.00 124.8±0.36ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.01 100.0±0.13ms ? ?/sec 1.00 98.8±0.20ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.01 147.9±0.27ms ? ?/sec 1.00 146.7±0.36ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.00 307.0±8.33ms ? ?/sec 1.00 308.4±10.12ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.00 27.0±0.07ms ? ?/sec 1.00 26.9±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.00 110.1±0.12ms ? ?/sec 1.00 109.7±0.16ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.00 108.0±0.11ms ? ?/sec 1.00 108.0±0.11ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.00 18.7±0.06ms ? ?/sec 1.00 18.6±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.01 22.2±0.10ms ? ?/sec 1.00 22.0±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.00 6.8±0.05ms ? ?/sec 1.00 6.8±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.00 11.5±0.04ms ? ?/sec 1.00 11.4±0.04ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.00 20.6±0.07ms ? ?/sec 1.01 20.8±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.01 5.2±0.01ms ? ?/sec 1.00 5.2±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.00 5.6±0.03ms ? ?/sec 1.00 5.6±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.00 4.3±0.01ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 795.2s |
| Peak memory | 288.4 MiB |
| Avg memory | 135.8 MiB |
| CPU user | 704.5s |
| CPU sys | 85.5s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 790.2s |
| Peak memory | 286.1 MiB |
| Avg memory | 129.6 MiB |
| CPU user | 697.6s |
| CPU sys | 90.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3465724642
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -135,12 +143,439 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(BooleanBuffer),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a == b,
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Iterator over the [`RowSelector`]s of a [`RowSelection`].
+///
+/// Yields owned [`RowSelector`] values; mask-backed selections stream the
+/// run-length form via [`BitSliceIterator`] without allocation.
+#[derive(Debug)]
+pub enum RowSelectionIter<'a> {
+ Selectors(std::iter::Copied<std::slice::Iter<'a, RowSelector>>),
+ Mask(MaskRunIter<'a>),
+}
+
+impl Iterator for RowSelectionIter<'_> {
+ type Item = RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<RowSelector> {
+ match self {
+ Self::Selectors(it) => it.next(),
+ Self::Mask(it) => it.next(),
+ }
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`].
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
Review Comment:
I updated `RowSelection::iter()` to keep returning `Iterator<Item = &RowSelector>`.
- For selector-backed selections this still borrows from the internal `Vec<RowSelector>`.
- For mask-backed selections I added a lazy selector cache, because the returned `&RowSelector` values need stable storage inside `RowSelection`; otherwise they would point into a temporary `Vec`.
The internal hot paths still avoid this cache and use the `BooleanBuffer` or `MaskRunIter` directly.
the struct is below
```rust
pub struct RowSelection {
inner: RowSelectionInner,
}
pub(crate) enum RowSelectionInner {
Selectors(Vec<RowSelector>),
Mask(Box<MaskSelection>),
}
pub(crate) struct MaskSelection {
mask: BooleanBuffer,
selectors: OnceLock<Vec<RowSelector>>,
}
```
I am not fully sure this is the best approach, so suggestions and feedback are welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783425974
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783400488)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4783400488-637-cprmq 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (7b102f41c4e92c03788a991dfd7d59be4fd67b21) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..7b102f41c4e92c03788a991dfd7d59be4fd67b21)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_clickbench
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5011741531
I reran one representative point from `row_selection_cursor` on the current
head (`8c217d8`) against the merge-base (`ee30b61`), using five alternating
current/base runs on a pinned CPU.
For `uniform50/L32`:
- forced `Selectors`: +7.67% paired geometric mean
- forced `Mask`: +2.80%
All five pairs moved in the same direction and Criterion produced no warnings.
However, I want to be precise about the interpretation: this benchmark forces
the two policies in order to map their crossover; it does not exercise `Auto`.
Therefore these numbers are a signal that the underlying forced paths became
slower at L32, not evidence that the Auto decision or threshold is wrong. A
single point is also not enough to establish a broad regression.
Could we rerun the complete L4–L40 curve in `row_selection_cursor` on the current head and compare:
1. the Selector/Mask crossover against the merge-base;
2. the strategy selected by Auto at each point; and
3. Auto's runtime against the faster forced strategy?
Since this PR adds a second input representation, it would be particularly
useful to run the matrix for both selector-backed and mask-backed
`RowSelection`s. That would show whether Auto should use a different threshold
or cost model when choosing a strategy requires first converting the backing
representation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5236486859
> I think we are very close to being able to implement something like this
>
> * https://dl.acm.org/doi/10.1145/3589323
>
> We probably already do it for unnested (leaf) nodes -- but need a bit more cleverness for structured nodes
@alamb this comment sent me down a rabbit hole — I went and implemented the paper on top of this `Mask` infrastructure and measured it end to end. It didn't pay off, but the structured-node question you raised turned out to have a concrete answer, and two other things blocked me that I don't think are obvious from the outside.
**On "a bit more cleverness for structured nodes":** the constraint is an ordering one. Admission has to be decided for the *entire* reader subtree **before any child produces output**. My first cut walked children in order and only discovered a mixed projection after an earlier child had already written compact, already-filtered data — at which point you're holding one filtered column and one unfiltered column destined for the same `RecordBatch`, and it cannot be unwound. It aborted a real ClickBench query (a `BYTE_ARRAY` column projected alongside an eligible integer one).
The fix was a side-effect-free predicate answered for the whole subtree up front, before the first byte is decoded, plus a contract that a leaf may never decline mid-stream for anything it could have known statically. The interesting part is the corollary: anything a leaf *cannot* know statically has to be handled internally rather than by declining — a column chunk can switch from `RLE_DICTIONARY` to `PLAIN` partway through, and if a leaf declines there and the caller reads the short return as "chunk exhausted", you get the correct row count with values sourced from the wrong row group. That one only showed up under a full-value content check.
**Two things blocked me that weren't the structured-node problem.** Both look identical to "the workload isn't eligible", so a coverage counter alone can't tell them apart:
1. `CachedArrayReader` wraps any column read by both a filter and the projection. It can't forward an admission predicate it doesn't know about, so it reports ineligible, and under an all-or-nothing subtree rule that one wrapper disqualifies the entire scan. Disabling the predicate cache moved TPC-H coverage from 10/22 to 14/22 queries and 3.9x the rows. It also explains exactly which ClickBench queries survived: q30/q31, whose filter column isn't in their output projection.
2. `ReadPlanBuilder::with_predicate_options` builds its reader via `ParquetRecordBatchReader::new`. Predicate *N* there reads under the selection accumulated from predicates *1..N-1* — which is the mechanism the paper is actually about. I wired the output path and not that constructor, so on TPC-H Q6 (the paper's own headline query) my implementation touched **under 2%** of the maskable rows: 711 filter-chain calls over ~6M rows, against 106 output-phase calls over 114K.
(To be clear, neither is an upstream bug — the admission predicate is an addition on my branch, not an existing API. They're both "here's where a new `ArrayReader` capability silently loses coverage".)
**Outcome, honestly:** the leaf kernel is fast — 2.6x–4.6x against stock `get_batch_with_dict` + `arrow::compute::filter` on captured low-survival pages — but I never established a query-level benefit. My two fully-covered queries ran at 100% coverage over 65M rows each and showed no reproducible direction across two rounds. And for sizing: the paper's 3.1x on Q6 is against a 2023 C++ baseline, while modern arrow-rs's full-decode path has already absorbed most of that headroom, so the residual against today's `main` should be expected to be much smaller.
Write-up with the data and commit pins: https://github.com/hhhizzz/arrow-rs/blob/exp/v21-rle-selected-fill-20260807/experiments/shape-aware-selected-decoding/README.md — I've also filed the longer version on #7456 since the predicate-cache interaction lands squarely on that epic's second slow case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5035510795
Thanks for the updates. I re-ran the `row_selection_cursor` benchmark on the latest head with three alternating rounds against the main baseline containing #10288.
The new early-exit run-count logic substantially reduces mask-backed Auto overhead (for example, L4 drops from +24.5% to +2.8%), and the measured crossover remains between L28 and L32, so the current threshold of 32 still looks appropriate. The residual selector variation is codegen/layout-sensitive rather than additional cursor work.
The sparse-page integration and added coverage also look good.
LGTM from both correctness and performance perspectives.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3643101412
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -135,12 +149,297 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(Box<MaskSelection>),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.mask().len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a.mask() == b.mask(),
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.mask().len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.mask().set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Borrowed iterator over the [`RowSelector`]s of a [`RowSelection`].
+#[derive(Debug)]
+pub struct RowSelectionIter<'a>(std::slice::Iter<'a, RowSelector>);
+
+impl<'a> Iterator for RowSelectionIter<'a> {
+ type Item = &'a RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
Review Comment:
apply in [86610b5](https://github.com/apache/arrow-rs/pull/10141/commits/86610b5bbb3a2a91c7cae070eddbb57fff24e015)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787519934
Thanks for your reviews @alamb , i revert the breaking api change, and fix some performance issue, i retrigger the benchmark, let see the result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4789987723
weird, in my local(M2 Max), i can't reproduce the result for `arrow_reader_clickbench` benchmark
```shell
git checkout export-mask
cargo bench --bench arrow_reader_clickbench --features="arrow,async,object_store,experimental,test_common" -- --save-baseline 10141
git checkout main
cargo bench --bench arrow_reader_clickbench --features="arrow,async,object_store,experimental,test_common" -- --save-baseline main
critcmp main 10141
```
<img width="1378" height="1488" alt="image" src="https://github.com/user-attachments/assets/333ff3d9-12ed-4e24-99d3-e341f3323a3e" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4791423784
I tried one more change in #10209: inline the selector-backed bodies of `scan_ranges` and `expand_to_batch_boundaries` directly into the `RowSelection` methods, so the selector path keeps the same code shape as `main`.
I ran `arrow_reader_clickbench` three times on that branch. The results are not stable across runs:
| case | Run 1 | Run 2 | Run 3 |
|---|---|---|---|
| `async/Q20` | **+25.8%** | −12.5% | −14.5% |
| `async/Q21` | **+27.2%** | −16.7% | −19.1% |
| `async/Q22` | **+16.4%** | −16.3% | −7.8% |
| `sync/Q13` | +0.8% | **+16.6%** | **+16.7%** |
| `async_object_store/Q23` | −1.9% | −9.7% | −4.2% |
Same commit, same merge-base, but results swing 30–46 percentage points between runs. The `sync/Q13` regression only shows up in 2 of 3 runs, and the big `async/Q20/Q21/Q22` improvements only show up in 2 of 3 runs.
Filtering to cases consistent across all three runs (spread ≤ 3pp, same sign), the remaining regressions are all small (≤ 2.2%):
```
async_object_store/Q36 +2.2%
async_object_store/Q38 +1.5%
sync/Q12 +1.5%
async_object_store/Q20 +1.2%
async_object_store/Q27 +1.1%
sync/Q36 +0.9%
async/Q14 +0.6%
```
No stable improvement across all three runs either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4732734563
I did a quick TPC-DS follow-up on this PR to check whether the bitmap-backed
`RowSelection` change shows any broad regression in DataFusion. This is **not**
a formal 10-round benchmark result; it is a 3-round/debug signal plus targeted
rechecks.
Setup:
- TPC-DS SF10, all 99 queries
- 3 full rounds, 1 iteration per query
- CPUSET `0-23`
- Parquet pushdown filters / reorder filters / pruning all enabled
- Arrow baseline: PR base `be664614ab`
- Arrow current: this PR head `7f40e67a`
- DataFusion base: `ddc157d10e8`, with only Cargo patch wiring changed between the two sides
Full 3-round result was effectively neutral:
```text
main total: 175786.07 ms
current total: 175010.49 ms
geomean current/main: 0.993443
runner-reported current speedup: 0.660%
failures: 0
```
The round-level direction was not stable:
```text
round 1: +2.18% current speedup
round 2: +6.44% current speedup
round 3: -6.22% current speedup
```
Some queries initially looked slower in the full run, so I rechecked the largest
apparent regressions in isolation with 5 rounds each:
| query | isolated geomean current/main | bucket | range |
|---:|---:|---|---:|
| q39 | 1.014365 | neutral | 0.991326-1.057961 |
| q23 | 1.045529 | neutral | 0.980200-1.171810 |
| q28 | 0.991207 | neutral | 0.953509-1.051072 |
| q47 | 1.011234 | neutral | 0.989303-1.030131 |
| q87 | 0.989210 | neutral | 0.957960-1.017851 |
I also ran `dfbench tpcds --debug` for those queries and compared
`DataSourceExec` metrics. I did not see evidence of row-pushdown getting worse
on the PR side. For example, summed task metrics showed:
| query/table | processing main -> current | scanning main -> current | row_pushdown main -> current |
|---|---:|---:|---:|
| q39 `inventory` | 752.4 -> 764.4 ms | 7770.0 -> 7660.0 ms | 74.9 -> 74.5 ms |
| q23 `store_sales` | 23310.0 -> 22520.0 ms | 87870.0 -> 84350.0 ms | 13157.2 -> 12881.9 ms |
| q28 `store_sales` | 15470.0 -> 14710.0 ms | 81720.0 -> 78300.0 ms | 247.2 -> 243.8 ms |
Those metrics are summed across tasks, not wall-clock time, but they are useful
for attribution. The main takeaway is that the large per-query slowdowns from
the 3-round full run did not reproduce in isolated runs and do not appear to
come from row-pushdown evaluation.
My interpretation is that standard TPC-DS is mostly exercising the existing
selector-backed `RowSelection::from_filters` path, not the new "caller already
has a `BooleanBuffer`" path. So near-neutral TPC-DS is expected and should not
block this PR by itself.
A few suggestions:
1. Keep the PR focused on first-class bitmap-backed `RowSelection`; this is the
right shape for external indexes / FTS / runtime filters that already produce
a bitmap.
2. Add or keep direct benchmarks for the existing selector-backed operations
(`from_filters`, `and_then`, `intersection`, `split_off`, `limit`, `offset`)
because the PR changes the common representation and iterator surface even
when callers do not use the new bitmap constructor.
3. Add an end-to-end benchmark where the access plan starts as a `BooleanBuffer`,
with both fragmented and clustered masks. Standard TPC-DS is not a strong
workload for that new path.
4. For `Auto` with mask-backed input, consider documenting the current "preserve
mask" behavior or adding a shape gate. The raw representation break-even is
roughly `selector_count > row_count / 128`; fragmented masks strongly favor
bitmap, but clustered masks can still be better as selectors.
Overall, I see the TPC-DS signal as neutral, while the synthetic
fragmented-selection benchmark still strongly supports the API direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788508610
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788477163)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4788477163-653-wzxgs 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (31216822d8c2acae4f3e8d7143488947a5f42712) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..31216822d8c2acae4f3e8d7143488947a5f42712)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_clickbench
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788246859
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788216344)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4788216344-652-npbdx 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (31216822d8c2acae4f3e8d7143488947a5f42712) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..31216822d8c2acae4f3e8d7143488947a5f42712)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_clickbench
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787701295
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787465734)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_clickbench/async/Q1 1.00 1082.9±4.02µs ? ?/sec 1.00 1083.7±8.52µs ? ?/sec
arrow_reader_clickbench/async/Q10 1.00 6.5±0.05ms ? ?/sec 1.00 6.4±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.01 7.5±0.07ms ? ?/sec 1.00 7.4±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.01 14.4±0.08ms ? ?/sec 1.00 14.3±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.01 17.0±0.14ms ? ?/sec 1.00 16.8±0.07ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.01 15.9±0.10ms ? ?/sec 1.00 15.8±0.09ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.00 3.0±0.02ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.09 91.6±0.89ms ? ?/sec 1.00 83.7±10.72ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.00 87.9±10.09ms ? ?/sec 1.22 107.1±0.51ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.00 123.5±1.86ms ? ?/sec 1.10 136.3±4.29ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.04 253.0±1.97ms ? ?/sec 1.00 242.6±0.66ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.03 19.5±0.17ms ? ?/sec 1.00 19.1±0.08ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.04 59.0±0.54ms ? ?/sec 1.00 56.9±0.31ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.04 59.3±0.56ms ? ?/sec 1.00 57.1±0.28ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.01 18.6±0.10ms ? ?/sec 1.00 18.4±0.07ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.05 15.6±0.32ms ? ?/sec 1.00 14.9±0.09ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.00 5.5±0.01ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.05 13.8±0.27ms ? ?/sec 1.00 13.2±0.10ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.07 24.9±0.52ms ? ?/sec 1.00 23.4±0.14ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.01 5.7±0.05ms ? ?/sec 1.00 5.7±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.01 5.0±0.03ms ? ?/sec 1.00 5.0±0.04ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.00 3.5±0.02ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q1 1.00 1068.9±4.11µs ? ?/sec 1.00 1065.9±4.37µs ? ?/sec
arrow_reader_clickbench/async_object_store/Q10 1.00 6.3±0.05ms ? ?/sec 1.00 6.4±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q11 1.00 7.3±0.05ms ? ?/sec 1.00 7.3±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q12 1.00 14.3±0.09ms ? ?/sec 1.00 14.3±0.06ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q13 1.01 16.9±0.15ms ? ?/sec 1.00 16.8±0.08ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q14 1.01 15.8±0.13ms ? ?/sec 1.00 15.7±0.07ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q19 1.01 3.0±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q20 1.03 72.8±0.68ms ? ?/sec 1.00 71.0±0.14ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q21 1.03 81.9±0.66ms ? ?/sec 1.00 79.6±0.17ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q22 1.04 100.6±0.77ms ? ?/sec 1.00 96.8±0.15ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q23 1.00 230.2±0.70ms ? ?/sec 1.02 234.2±0.66ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q24 1.02 19.3±0.16ms ? ?/sec 1.00 18.8±0.10ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q27 1.03 58.0±0.74ms ? ?/sec 1.00 56.0±0.16ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q28 1.03 58.1±0.80ms ? ?/sec 1.00 56.5±0.16ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q30 1.02 18.3±0.14ms ? ?/sec 1.00 18.0±0.07ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q36 1.03 14.8±0.30ms ? ?/sec 1.00 14.4±0.15ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q37 1.01 5.4±0.02ms ? ?/sec 1.00 5.3±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q38 1.04 13.1±0.25ms ? ?/sec 1.00 12.6±0.12ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q39 1.08 24.2±0.66ms ? ?/sec 1.00 22.4±0.13ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q40 1.03 5.5±0.04ms ? ?/sec 1.00 5.4±0.05ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q41 1.01 4.8±0.02ms ? ?/sec 1.00 4.8±0.02ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q42 1.01 3.4±0.01ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.00 891.2±0.98µs ? ?/sec 1.00 890.6±1.72µs ? ?/sec
arrow_reader_clickbench/sync/Q10 1.00 5.1±0.02ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.00 6.0±0.02ms ? ?/sec 1.00 6.0±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.00 21.6±0.10ms ? ?/sec 1.00 21.7±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.16 28.4±0.24ms ? ?/sec 1.00 24.5±0.13ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.00 23.0±0.12ms ? ?/sec 1.00 23.0±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.03 124.4±0.27ms ? ?/sec 1.00 121.3±0.24ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.03 99.3±0.20ms ? ?/sec 1.00 96.4±0.22ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.02 146.5±0.41ms ? ?/sec 1.00 143.4±0.32ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.01 302.2±7.89ms ? ?/sec 1.00 298.7±10.01ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.00 26.7±0.15ms ? ?/sec 1.00 26.7±0.10ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.02 109.5±0.23ms ? ?/sec 1.00 106.9±0.20ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.03 107.5±0.21ms ? ?/sec 1.00 104.3±0.18ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.00 18.6±0.10ms ? ?/sec 1.00 18.7±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.00 21.9±0.10ms ? ?/sec 1.01 22.1±0.11ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.00 6.7±0.01ms ? ?/sec 1.01 6.8±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.00 11.3±0.07ms ? ?/sec 1.01 11.4±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.00 20.5±0.11ms ? ?/sec 1.00 20.4±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.00 5.1±0.02ms ? ?/sec 1.01 5.2±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.00 5.5±0.03ms ? ?/sec 1.00 5.6±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.00 4.2±0.02ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 790.2s |
| Peak memory | 286.3 MiB |
| Avg memory | 130.7 MiB |
| CPU user | 709.8s |
| CPU sys | 74.2s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 780.2s |
| Peak memory | 286.1 MiB |
| Avg memory | 133.7 MiB |
| CPU user | 695.3s |
| CPU sys | 82.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4789267756
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4789243487)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4789243487-654-fw46z 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (ee2df31354821271252272b819cc827a9f2c74ac) to 7fb7751 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/7fb775196456bbc7892cecec2d07c655608f7bb5..ee2df31354821271252272b819cc827a9f2c74ac)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_clickbench
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3641496098
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -132,12 +146,324 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
Review Comment:
🧑🍳 👌
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -124,6 +131,13 @@ impl RowSelector {
/// RowSelection::from_consecutive_ranges(ranges.into_iter(), 20);
/// let actual: Vec<RowSelector> = selection.into();
/// assert_eq!(actual, expected);
+///
+/// // or directly from a packed bitmap, when the upstream producer already
+/// // has one. The bitmap is kept as-is rather than run-length-encoded.
+/// use arrow_buffer::BooleanBuffer;
+/// let mask = BooleanBuffer::from(vec![true, false, true, true]);
+/// let selection = RowSelection::from_boolean_buffer(mask);
+/// assert_eq!(selection.row_count(), 3);
/// ```
///
/// A [`RowSelection`] maintains the following invariants:
Review Comment:
The note on invariants I think only apply to RLE/Selector backed one -- it would be good to update this documentation
##########
parquet/src/arrow/arrow_reader/read_plan.rs:
##########
@@ -167,26 +167,7 @@ impl ReadPlanBuilder {
None => return RowSelectionStrategy::Selectors,
};
- // total_rows: total number of rows selected / skipped
- // effective_count: number of non-empty selectors
- let (total_rows, effective_count) =
- selection.iter().fold((0usize, 0usize), |(rows, count), s| {
- if s.row_count > 0 {
- (rows + s.row_count, count + 1)
- } else {
- (rows, count)
- }
- });
-
- if effective_count == 0 {
- return RowSelectionStrategy::Mask;
- }
-
- if total_rows < effective_count.saturating_mul(threshold) {
- RowSelectionStrategy::Mask
- } else {
- RowSelectionStrategy::Selectors
- }
+ selection.auto_selection_strategy(threshold)
Review Comment:
Well that is certainly nicer - I like how it is now much more encapsulated
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -124,6 +131,13 @@ impl RowSelector {
/// RowSelection::from_consecutive_ranges(ranges.into_iter(), 20);
/// let actual: Vec<RowSelector> = selection.into();
/// assert_eq!(actual, expected);
+///
Review Comment:
And then we could add a note somewhere like
```rust
///
/// Depending on the pattern or rows to be selected, [`RowSelection`] has
/// either a bitmap or an RLE ([`RowSelector`]) based implementation.
```
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -124,6 +131,13 @@ impl RowSelector {
/// RowSelection::from_consecutive_ranges(ranges.into_iter(), 20);
/// let actual: Vec<RowSelector> = selection.into();
/// assert_eq!(actual, expected);
+///
Review Comment:
It might be worth updating the descriptio of this struct (we can do it in a follow on PR too)
The current one talks about skipping/selecting
```rust
/// [`RowSelection`] allows selecting or skipping a provided number of rows
/// when scanning the parquet file.
///
/// This is applied prior to reading column data, and can therefore
/// be used to skip IO to fetch data into memory
```
Maybe we could focus on the selection aspect part
```rust
/// [`RowSelection`] represents selecting a subset of rows
/// when scanning a parquet file.
///
/// This is applied prior to reading column data, and can therefore
/// be used to skip IO to fetch data into memory.
```
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -132,12 +146,324 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(Box<MaskSelection>),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.mask().len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a.mask() == b.mask(),
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.mask().len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.mask().set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Borrowed iterator over the [`RowSelector`]s of a [`RowSelection`].
+#[derive(Debug)]
+pub struct RowSelectionIter<'a>(std::slice::Iter<'a, RowSelector>);
+
+impl<'a> Iterator for RowSelectionIter<'a> {
+ type Item = &'a RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
+ self.0.next()
+ }
+}
+
+#[inline]
+fn scan_ranges_from_selectors<I>(selectors: I, page_locations: &[PageLocation]) -> Vec<Range<u64>>
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut ranges: Vec<Range<u64>> = vec![];
+ let mut row_offset = 0;
+
+ let mut pages = page_locations.iter().peekable();
+ let mut selectors = selectors.into_iter();
+ let mut current_selector = selectors.next();
+ let mut current_page = pages.next();
+
+ let mut current_page_included = false;
+
+ while let Some((selector, page)) = current_selector.as_mut().zip(current_page) {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ current_page_included = true;
+ }
+
+ if let Some(next_page) = pages.peek() {
+ if row_offset + selector.row_count > next_page.first_row_index as usize {
+ let remaining_in_page = next_page.first_row_index as usize - row_offset;
+ selector.row_count -= remaining_in_page;
+ row_offset += remaining_in_page;
+ current_page = pages.next();
+ current_page_included = false;
+
+ continue;
+ } else {
+ if row_offset + selector.row_count == next_page.first_row_index as usize {
+ current_page = pages.next();
+ current_page_included = false;
+ }
+ row_offset += selector.row_count;
+ current_selector = selectors.next();
+ }
+ } else {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ }
+ current_selector = selectors.next()
+ }
+ }
+
+ ranges
+}
+
+#[inline]
+fn expand_to_batch_boundaries_from_selectors<I>(
+ selectors: I,
+ batch_size: usize,
+ total_rows: usize,
+) -> RowSelection
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut expanded_ranges = Vec::new();
+ let mut row_offset = 0;
+
+ for selector in selectors {
+ if selector.skip {
+ row_offset += selector.row_count;
+ } else {
+ let start = row_offset;
+ let end = row_offset + selector.row_count;
+
+ // Expand start to batch boundary
+ let expanded_start = (start / batch_size) * batch_size;
+ // Expand end to batch boundary
+ let expanded_end = end.div_ceil(batch_size) * batch_size;
+ let expanded_end = expanded_end.min(total_rows);
+
+ expanded_ranges.push(expanded_start..expanded_end);
+ row_offset += selector.row_count;
+ }
+ }
+
+ // Sort ranges by start position
+ expanded_ranges.sort_by_key(|range| range.start);
+
+ // Merge overlapping or consecutive ranges
+ let mut merged_ranges: Vec<Range<usize>> = Vec::new();
+ for range in expanded_ranges {
+ if let Some(last) = merged_ranges.last_mut() {
+ if range.start <= last.end {
+ // Overlapping or consecutive - merge them
+ last.end = last.end.max(range.end);
+ } else {
+ // No overlap - add new range
+ merged_ranges.push(range);
+ }
+ } else {
+ // First range
+ merged_ranges.push(range);
+ }
+ }
+
+ RowSelection::from_consecutive_ranges(merged_ranges.into_iter(), total_rows)
}
impl RowSelection {
+ fn from_selectors(selectors: Vec<RowSelector>) -> Self {
Review Comment:
Should this be `pub` for parity with the other constructors like `from_boolean_buffer`? I think there is a `From<Vec<RowSelector>>` impl, but it is strange that the named method is not also pub
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -132,12 +146,324 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(Box<MaskSelection>),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.mask().len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a.mask() == b.mask(),
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.mask().len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.mask().set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Borrowed iterator over the [`RowSelector`]s of a [`RowSelection`].
+#[derive(Debug)]
+pub struct RowSelectionIter<'a>(std::slice::Iter<'a, RowSelector>);
+
+impl<'a> Iterator for RowSelectionIter<'a> {
+ type Item = &'a RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
+ self.0.next()
+ }
+}
+
+#[inline]
+fn scan_ranges_from_selectors<I>(selectors: I, page_locations: &[PageLocation]) -> Vec<Range<u64>>
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut ranges: Vec<Range<u64>> = vec![];
+ let mut row_offset = 0;
+
+ let mut pages = page_locations.iter().peekable();
+ let mut selectors = selectors.into_iter();
+ let mut current_selector = selectors.next();
+ let mut current_page = pages.next();
+
+ let mut current_page_included = false;
+
+ while let Some((selector, page)) = current_selector.as_mut().zip(current_page) {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ current_page_included = true;
+ }
+
+ if let Some(next_page) = pages.peek() {
+ if row_offset + selector.row_count > next_page.first_row_index as usize {
+ let remaining_in_page = next_page.first_row_index as usize - row_offset;
+ selector.row_count -= remaining_in_page;
+ row_offset += remaining_in_page;
+ current_page = pages.next();
+ current_page_included = false;
+
+ continue;
+ } else {
+ if row_offset + selector.row_count == next_page.first_row_index as usize {
+ current_page = pages.next();
+ current_page_included = false;
+ }
+ row_offset += selector.row_count;
+ current_selector = selectors.next();
+ }
+ } else {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ }
+ current_selector = selectors.next()
+ }
+ }
+
+ ranges
+}
+
+#[inline]
+fn expand_to_batch_boundaries_from_selectors<I>(
+ selectors: I,
+ batch_size: usize,
+ total_rows: usize,
+) -> RowSelection
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut expanded_ranges = Vec::new();
+ let mut row_offset = 0;
+
+ for selector in selectors {
+ if selector.skip {
+ row_offset += selector.row_count;
+ } else {
+ let start = row_offset;
+ let end = row_offset + selector.row_count;
+
+ // Expand start to batch boundary
+ let expanded_start = (start / batch_size) * batch_size;
+ // Expand end to batch boundary
+ let expanded_end = end.div_ceil(batch_size) * batch_size;
+ let expanded_end = expanded_end.min(total_rows);
+
+ expanded_ranges.push(expanded_start..expanded_end);
+ row_offset += selector.row_count;
+ }
+ }
+
+ // Sort ranges by start position
+ expanded_ranges.sort_by_key(|range| range.start);
+
+ // Merge overlapping or consecutive ranges
+ let mut merged_ranges: Vec<Range<usize>> = Vec::new();
+ for range in expanded_ranges {
+ if let Some(last) = merged_ranges.last_mut() {
+ if range.start <= last.end {
+ // Overlapping or consecutive - merge them
+ last.end = last.end.max(range.end);
+ } else {
+ // No overlap - add new range
+ merged_ranges.push(range);
+ }
+ } else {
+ // First range
+ merged_ranges.push(range);
+ }
+ }
+
+ RowSelection::from_consecutive_ranges(merged_ranges.into_iter(), total_rows)
}
impl RowSelection {
+ fn from_selectors(selectors: Vec<RowSelector>) -> Self {
Review Comment:
Claude says "no because this does no validation" -- so maybe a small comment saying that would help
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -135,12 +149,297 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(Box<MaskSelection>),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.mask().len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a.mask() == b.mask(),
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.mask().len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.mask().set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Borrowed iterator over the [`RowSelector`]s of a [`RowSelection`].
+#[derive(Debug)]
+pub struct RowSelectionIter<'a>(std::slice::Iter<'a, RowSelector>);
+
+impl<'a> Iterator for RowSelectionIter<'a> {
+ type Item = &'a RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
Review Comment:
This also came up with claude code in my review
I think it would basically look like
```rust
impl<'a> Iterator for RowSelectionIter<'a> {
type Item = &'a RowSelector;
#[inline]
fn next(&mut self) -> Option<Self::Item> {
self.0.next()
}
#[inline]
fn size_hint(&self) -> (usize, Option<usize>) {
self.0.size_hint()
}
}
impl ExactSizeIterator for RowSelectionIter<'_> {}
// once it returns None, it will continue returning None
impl std::iter::FusedIterator for RowSelectionIter<'_> {}
```
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -24,7 +24,9 @@ use arrow_schema::{ArrowError, DataType as ArrowType, FieldRef, Schema, SchemaRe
use arrow_select::filter::filter_record_batch;
pub use filter::{ArrowPredicate, ArrowPredicateFn, RowFilter};
use selection::MaskCursor;
-pub use selection::{RowSelection, RowSelectionCursor, RowSelectionPolicy, RowSelector};
+pub use selection::{
+ MaskRunIter, RowSelection, RowSelectionCursor, RowSelectionPolicy, RowSelector,
+};
Review Comment:
Since it is pub I (and claude) suggest also adding `RowSelectionIter` here
```suggestion
pub use selection::{
MaskRunIter, RowSelection, RowSelectionCursor, RowSelectionIter, RowSelectionPolicy,
RowSelector,
};
```
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -132,12 +146,324 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(Box<MaskSelection>),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.mask().len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a.mask() == b.mask(),
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.mask().len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.mask().set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Borrowed iterator over the [`RowSelector`]s of a [`RowSelection`].
+#[derive(Debug)]
+pub struct RowSelectionIter<'a>(std::slice::Iter<'a, RowSelector>);
+
+impl<'a> Iterator for RowSelectionIter<'a> {
+ type Item = &'a RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
+ self.0.next()
+ }
+}
+
+#[inline]
+fn scan_ranges_from_selectors<I>(selectors: I, page_locations: &[PageLocation]) -> Vec<Range<u64>>
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut ranges: Vec<Range<u64>> = vec![];
+ let mut row_offset = 0;
+
+ let mut pages = page_locations.iter().peekable();
+ let mut selectors = selectors.into_iter();
+ let mut current_selector = selectors.next();
+ let mut current_page = pages.next();
+
+ let mut current_page_included = false;
+
+ while let Some((selector, page)) = current_selector.as_mut().zip(current_page) {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ current_page_included = true;
+ }
+
+ if let Some(next_page) = pages.peek() {
+ if row_offset + selector.row_count > next_page.first_row_index as usize {
+ let remaining_in_page = next_page.first_row_index as usize - row_offset;
+ selector.row_count -= remaining_in_page;
+ row_offset += remaining_in_page;
+ current_page = pages.next();
+ current_page_included = false;
+
+ continue;
+ } else {
+ if row_offset + selector.row_count == next_page.first_row_index as usize {
+ current_page = pages.next();
+ current_page_included = false;
+ }
+ row_offset += selector.row_count;
+ current_selector = selectors.next();
+ }
+ } else {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ }
+ current_selector = selectors.next()
+ }
+ }
+
+ ranges
+}
+
+#[inline]
+fn expand_to_batch_boundaries_from_selectors<I>(
+ selectors: I,
+ batch_size: usize,
+ total_rows: usize,
+) -> RowSelection
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut expanded_ranges = Vec::new();
+ let mut row_offset = 0;
+
+ for selector in selectors {
+ if selector.skip {
+ row_offset += selector.row_count;
+ } else {
+ let start = row_offset;
+ let end = row_offset + selector.row_count;
+
+ // Expand start to batch boundary
+ let expanded_start = (start / batch_size) * batch_size;
+ // Expand end to batch boundary
+ let expanded_end = end.div_ceil(batch_size) * batch_size;
+ let expanded_end = expanded_end.min(total_rows);
+
+ expanded_ranges.push(expanded_start..expanded_end);
+ row_offset += selector.row_count;
+ }
+ }
+
+ // Sort ranges by start position
+ expanded_ranges.sort_by_key(|range| range.start);
+
+ // Merge overlapping or consecutive ranges
+ let mut merged_ranges: Vec<Range<usize>> = Vec::new();
+ for range in expanded_ranges {
+ if let Some(last) = merged_ranges.last_mut() {
+ if range.start <= last.end {
+ // Overlapping or consecutive - merge them
+ last.end = last.end.max(range.end);
+ } else {
+ // No overlap - add new range
+ merged_ranges.push(range);
+ }
+ } else {
+ // First range
+ merged_ranges.push(range);
+ }
+ }
+
+ RowSelection::from_consecutive_ranges(merged_ranges.into_iter(), total_rows)
}
impl RowSelection {
+ fn from_selectors(selectors: Vec<RowSelector>) -> Self {
+ Self {
+ inner: RowSelectionInner::Selectors(selectors),
+ }
+ }
+
+ /// Create a [`RowSelection`] from a packed [`BooleanBuffer`].
+ ///
+ /// Each set bit selects a row, each unset bit skips one. Unlike
+ /// [`Self::from_filters`], the bitmap is kept as-is rather than
+ /// eagerly run-length-encoded. [`Self::iter`] materializes and caches the
+ /// RLE form on first use; use [`MaskRunIter`] to stream the RLE form
+ /// directly from the bitmap.
+ pub fn from_boolean_buffer(mask: BooleanBuffer) -> Self {
+ Self {
+ inner: RowSelectionInner::Mask(Box::new(MaskSelection::new(mask))),
+ }
+ }
+
+ fn from_mask_selection(mask: MaskSelection) -> Self {
+ Self {
+ inner: RowSelectionInner::Mask(Box::new(mask)),
+ }
+ }
+
+ /// Returns the underlying mask if this selection is mask-backed.
+ ///
+ /// Public so that engines composing selections (e.g. DataFusion's
+ /// `ParquetAccessPlan::into_overall_row_selection`) can concatenate
Review Comment:
if we are going to refer to this, it probably would make sense to make it hyperlink rather than a name
##########
parquet/src/arrow/arrow_reader/selection/boolean.rs:
##########
@@ -0,0 +1,1308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{LoadedRowRanges, RowSelection, RowSelectionInner, RowSelector};
+use crate::errors::ParquetError;
+use arrow_array::BooleanArray;
+use arrow_buffer::bit_iterator::BitSliceIterator;
+use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder, Buffer};
+use std::cmp::Ordering;
+use std::sync::{Arc, OnceLock};
+
+/// Mask-backed [`RowSelection`] storage.
+///
+/// `selectors` is only populated if callers use the borrowed [`RowSelection::iter`]
+/// compatibility API. Internal paths that can stream or consume the bitmap avoid
+/// this cache.
+///
+/// `count` caches the popcount; `RowSelection::split_off` propagates it to
+/// both halves so repeated `row_count()` calls do not rescan the bitmap.
+#[derive(Debug)]
+pub(crate) struct MaskSelection {
+ mask: BooleanBuffer,
+ selectors: OnceLock<Vec<RowSelector>>,
+ count: OnceLock<usize>,
+}
+
+impl MaskSelection {
+ pub(super) fn new(mask: BooleanBuffer) -> Self {
+ Self {
+ mask,
+ selectors: OnceLock::new(),
+ count: OnceLock::new(),
+ }
+ }
+
+ /// Create a selection whose selected-row count is already known.
+ pub(super) fn with_count(mask: BooleanBuffer, count: usize) -> Self {
+ debug_assert!(count <= mask.len());
+ let cell = OnceLock::new();
+ let _ = cell.set(count);
+ Self {
+ mask,
+ selectors: OnceLock::new(),
+ count: cell,
+ }
+ }
+
+ pub(crate) fn mask(&self) -> &BooleanBuffer {
+ &self.mask
+ }
+
+ pub(crate) fn into_mask(self) -> BooleanBuffer {
+ let Self { mask, .. } = self;
+ mask
+ }
+
+ /// Number of selected rows, computed once and cached.
+ pub(super) fn count(&self) -> usize {
+ *self.count.get_or_init(|| self.mask.count_set_bits())
+ }
+
+ /// The cached selected-row count, if it has been computed.
+ pub(super) fn cached_count(&self) -> Option<usize> {
+ self.count.get().copied()
+ }
+
+ pub(super) fn selectors(&self) -> &[RowSelector] {
+ self.selectors
+ .get_or_init(|| mask_to_selectors(&self.mask))
+ .as_slice()
+ }
+}
+
+impl Clone for MaskSelection {
+ fn clone(&self) -> Self {
+ // Drop the selector cache but keep the cheap count cache.
+ Self {
+ mask: self.mask.clone(),
+ selectors: OnceLock::new(),
+ count: self.count.clone(),
+ }
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`], yielding owned [`RowSelector`]s
+/// without allocation.
+///
+/// Useful as a zero-cost alternative to [`RowSelection::iter`] for mask-backed
+/// selections, via [`RowSelection::as_mask`]:
+///
+/// ```ignore
+/// if let Some(mask) = selection.as_mask() {
+/// for run in MaskRunIter::new(mask) { ... }
+/// }
+/// ```
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ /// Create a streaming RLE iterator over a [`BooleanBuffer`].
+ pub fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
+ if let Some(p) = self.pending.take() {
+ return Some(p);
+ }
+ if self.finished {
+ return None;
+ }
+ match self.slices.next() {
+ Some((start, end)) => {
+ let select = RowSelector::select(end - start);
+ if start > self.cursor {
+ let skip = RowSelector::skip(start - self.cursor);
+ self.pending = Some(select);
+ self.cursor = end;
+ Some(skip)
+ } else {
+ self.cursor = end;
+ Some(select)
+ }
+ }
+ None => {
+ self.finished = true;
+ if self.cursor < self.total {
+ let skip = RowSelector::skip(self.total - self.cursor);
+ self.cursor = self.total;
+ Some(skip)
+ } else {
+ None
+ }
+ }
+ }
+ }
+}
+
+/// Cursor for iterating a mask-backed [`RowSelection`]
+///
+/// This is best for dense selections where there are many small skips
+/// or selections. For example, selecting every other row.
+///
+/// When page pruning produces sparse column data, `loaded_row_ranges` limits
+/// each decoded chunk to rows whose pages are loaded for every projected leaf.
+/// For example, two projected columns can have different page boundaries:
+///
+/// ```text
+/// Row ranges: [0, 4) [4, 6) [6, 8) [8, 10) [10, 12)
+/// Selection mask: 1000 00 00 00 01
+/// Column A pages: loaded | missing [4, 8) | loaded [8, 12)
+/// Column B pages: loaded [0, 6) | missing [6, 10) | loaded
+/// LoadedRowRanges: [0, 4) [10, 12)
+/// ```
+///
+/// The first chunk decodes `[0, 4)` with mask `1000`. The next chunk skips to
+/// row 11 and decodes `[11, 12)` with mask `1`. The loaded ranges are decode
+/// boundaries, not output batch boundaries: [`ParquetRecordBatchReader`]
+/// accumulates both chunks and applies the combined mask `10001` once.
+///
+/// [`ParquetRecordBatchReader`]: crate::arrow::arrow_reader::ParquetRecordBatchReader
+#[derive(Debug)]
+pub struct MaskCursor {
+ pub(super) mask: BooleanBuffer,
+ /// Current absolute offset into the selection
+ pub(super) position: usize,
+ /// Row ranges whose backing pages are loaded for every projected column.
+ pub(super) loaded_row_ranges: Option<Arc<LoadedRowRanges>>,
+}
+
+impl MaskCursor {
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ self.position >= self.mask.len()
+ }
+
+ /// Advance through the mask representation, producing the next chunk summary
+ pub fn next_mask_chunk(&mut self, batch_size: usize) -> Option<MaskChunk> {
+ if self.is_empty() {
+ return None;
+ }
+
+ Some(self.next_mask_chunk_non_empty(batch_size))
+ }
+
+ /// Produces the next chunk for a non-empty, trailing-skip-free mask.
+ fn next_mask_chunk_non_empty(&mut self, batch_size: usize) -> MaskChunk {
+ debug_assert!(!self.is_empty());
+
+ let (initial_skip, chunk_rows, selected_rows, mask_start, end_position) = {
+ let mask = &self.mask;
+ let start_position = self.position;
+ let mut cursor = start_position;
+ let mut initial_skip = 0;
+
+ while cursor < mask.len() && !mask.value(cursor) {
+ initial_skip += 1;
+ cursor += 1;
+ }
+ debug_assert!(
+ cursor < mask.len(),
+ "ReadPlan must remove trailing skips from Mask selections"
+ );
+
+ let mask_start = cursor;
+ let mut chunk_rows = 0;
+ let mut selected_rows = 0;
+
+ // Advance until enough rows have been selected to satisfy the batch size,
+ // or until the mask is exhausted. This mirrors the behaviour of the legacy
+ // `RowSelector` queue-based iteration.
+ while cursor < mask.len() && selected_rows < batch_size {
+ chunk_rows += 1;
+ if mask.value(cursor) {
+ selected_rows += 1;
+ }
+ cursor += 1;
+ }
+
+ (initial_skip, chunk_rows, selected_rows, mask_start, cursor)
+ };
+
+ self.position = end_position;
+
+ MaskChunk {
+ initial_skip,
+ chunk_rows,
+ selected_rows,
+ mask_start,
+ }
+ }
+
+ /// Returns the next non-empty mask chunk without crossing an unloaded row range.
+ ///
+ /// The [`ReadPlan`](crate::arrow::arrow_reader::ReadPlan) removes trailing
+ /// skips before constructing this cursor. Callers therefore only invoke
+ /// this method for a non-empty mask that has another selected row.
+ pub(crate) fn next_chunk(&mut self, batch_size: usize) -> Result<MaskChunk, ParquetError> {
+ debug_assert!(batch_size > 0);
+ debug_assert!(!self.is_empty());
+
+ if self.loaded_row_ranges.is_none() {
+ return Ok(self.next_mask_chunk_non_empty(batch_size));
+ }
+
+ let start_position = self.position;
+ let mut cursor = start_position;
+ while cursor < self.mask.len() && !self.mask.value(cursor) {
+ cursor += 1;
+ }
+
+ debug_assert!(
+ cursor < self.mask.len(),
+ "ReadPlan must remove trailing skips from Mask selections"
+ );
+
+ let loaded_range_end = self
+ .loaded_row_ranges
+ .as_ref()
+ .and_then(|ranges| ranges.end_containing(cursor))
+ .ok_or_else(|| {
+ ParquetError::General(format!(
+ "Internal Error: selected row {cursor} has no loaded page range"
+ ))
+ })?;
+
+ let mask_start = cursor;
+ let mut selected_rows = 0;
+ while cursor < loaded_range_end && cursor < self.mask.len() && selected_rows < batch_size {
+ if self.mask.value(cursor) {
+ selected_rows += 1;
+ }
+ cursor += 1;
+ }
+
+ self.position = cursor;
+ Ok(MaskChunk {
+ initial_skip: mask_start - start_position,
+ chunk_rows: cursor - mask_start,
+ selected_rows,
+ mask_start,
+ })
+ }
+
+ /// Materialise the boolean values for a mask-backed chunk
+ pub fn mask_values_for(&self, chunk: &MaskChunk) -> Result<BooleanArray, ParquetError> {
+ if chunk.mask_start.saturating_add(chunk.chunk_rows) > self.mask.len() {
+ return Err(ParquetError::General(
+ "Internal Error: MaskChunk exceeds mask length".to_string(),
+ ));
+ }
+ Ok(BooleanArray::from(
+ self.mask.slice(chunk.mask_start, chunk.chunk_rows),
+ ))
+ }
+}
+
+/// Result of computing the next chunk to read when using a [`MaskCursor`]
+#[derive(Debug)]
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+/// Materialize a [`BooleanBuffer`] into its RLE form.
+pub(crate) fn mask_to_selectors(mask: &BooleanBuffer) -> Vec<RowSelector> {
+ let total_rows = mask.len();
+ if total_rows == 0 {
+ return Vec::new();
+ }
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut last_end = 0;
+ for (start, end) in mask.set_slices() {
+ if start > last_end {
+ selectors.push(RowSelector::skip(start - last_end));
+ }
+ selectors.push(RowSelector::select(end - start));
+ last_end = end;
+ }
+ if last_end != total_rows {
+ selectors.push(RowSelector::skip(total_rows - last_end));
+ }
+ selectors
+}
+
+/// Returns whether `mask` contains at least `min_runs` alternating set/unset runs.
+///
+/// Stops as soon as the requested number of runs is found, avoiding a full scan
+/// when callers only need to know whether a boundary has been crossed.
+pub(super) fn mask_has_at_least_runs(mask: &BooleanBuffer, min_runs: usize) -> bool {
+ if min_runs == 0 {
+ return true;
+ }
+
+ let total_rows = mask.len();
+ if total_rows == 0 {
+ return false;
+ }
+
+ let mut run_count = 0;
+ let mut last_end = 0;
+ for (start, end) in mask.set_slices() {
+ run_count += usize::from(start > last_end) + 1;
+ if run_count >= min_runs {
+ return true;
+ }
+ last_end = end;
+ }
+
+ run_count + usize::from(last_end < total_rows) >= min_runs
+}
+
+/// Bitwise AND of two mask-backed selections. Longer side's tail passes through.
+pub(super) fn intersect_masks(l: &BooleanBuffer, r: &BooleanBuffer) -> BooleanBuffer {
+ if l.len() == r.len() {
+ return l & r;
+ }
+ let common = l.len().min(r.len());
+ let head = &l.slice(0, common) & &r.slice(0, common);
+ let (longer, longer_len) = if l.len() > r.len() {
+ (l, l.len())
+ } else {
+ (r, r.len())
+ };
+ let tail = longer.slice(common, longer_len - common);
+ let mut builder = BooleanBufferBuilder::new(longer_len);
+ builder.append_buffer(&head);
+ builder.append_buffer(&tail);
+ builder.finish()
+}
+
+/// Bitwise OR of two mask-backed selections. Longer side's tail passes through.
+pub(super) fn union_masks(l: &BooleanBuffer, r: &BooleanBuffer) -> BooleanBuffer {
+ if l.len() == r.len() {
+ return l | r;
+ }
+ let common = l.len().min(r.len());
Review Comment:
you might be able to do this more efficiently by copying the longer one and then using `&=` -- maybe in some follow on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3643179273
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -412,7 +637,22 @@ impl RowSelection {
///
/// returned: NNNNNNNNYYNYN
pub fn intersection(&self, other: &Self) -> Self {
- intersect_row_selections(&self.selectors, &other.selectors)
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Mask(l), RowSelectionInner::Mask(r)) => {
+ Self::from_boolean_buffer(intersect_masks(l.mask(), r.mask()))
+ }
+ (RowSelectionInner::Selectors(l), RowSelectionInner::Selectors(r)) => {
+ intersect_row_selections(l, r)
+ }
+ (RowSelectionInner::Selectors(l), RowSelectionInner::Mask(r)) => {
Review Comment:
track in https://github.com/apache/arrow-rs/issues/10423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4809413116
Will try and review this one more carefully today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787487324
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787465734)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4787465734-649-7hk2v 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (0ffbf26319f13b931c40678b0480c6a3ea9cbcaa) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..0ffbf26319f13b931c40678b0480c6a3ea9cbcaa)
BENCH_NAME=arrow_reader_row_filter
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_row_filter
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788712784
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788477163)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_clickbench/async/Q1 1.02 1097.5±6.11µs ? ?/sec 1.00 1079.9±4.52µs ? ?/sec
arrow_reader_clickbench/async/Q10 1.00 6.6±0.06ms ? ?/sec 1.00 6.5±0.07ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.00 7.5±0.04ms ? ?/sec 1.01 7.6±0.07ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.01 14.5±0.08ms ? ?/sec 1.00 14.4±0.10ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.01 17.2±0.13ms ? ?/sec 1.00 17.1±0.09ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.01 16.0±0.10ms ? ?/sec 1.00 15.9±0.08ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.02 3.1±0.03ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.10 91.9±0.92ms ? ?/sec 1.00 83.7±0.42ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.00 102.8±4.58ms ? ?/sec 1.04 106.7±0.55ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.00 124.2±0.83ms ? ?/sec 1.11 138.2±1.22ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.02 255.1±4.05ms ? ?/sec 1.00 250.7±1.24ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.01 19.6±0.15ms ? ?/sec 1.00 19.5±0.17ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.00 59.0±0.44ms ? ?/sec 1.00 58.8±0.43ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.01 59.0±0.76ms ? ?/sec 1.00 58.8±0.42ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.00 18.6±0.10ms ? ?/sec 1.00 18.6±0.12ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.00 15.6±0.28ms ? ?/sec 1.00 15.7±0.32ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.00 5.4±0.02ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.01 13.9±0.34ms ? ?/sec 1.00 13.8±0.26ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.02 25.1±0.46ms ? ?/sec 1.00 24.6±0.38ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.00 5.7±0.05ms ? ?/sec 1.00 5.7±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.01 5.0±0.06ms ? ?/sec 1.00 5.0±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.01 3.5±0.02ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q1 1.01 1064.1±3.48µs ? ?/sec 1.00 1049.2±4.70µs ? ?/sec
arrow_reader_clickbench/async_object_store/Q10 1.00 6.4±0.03ms ? ?/sec 1.00 6.4±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q11 1.00 7.4±0.03ms ? ?/sec 1.00 7.4±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q12 1.00 14.4±0.08ms ? ?/sec 1.00 14.4±0.08ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q13 1.00 17.0±0.13ms ? ?/sec 1.00 17.1±0.15ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q14 1.01 16.0±0.07ms ? ?/sec 1.00 15.8±0.10ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q19 1.01 3.0±0.03ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q20 1.00 72.9±0.66ms ? ?/sec 1.00 72.6±0.59ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q21 1.00 81.7±0.74ms ? ?/sec 1.00 81.5±0.60ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q22 1.02 100.7±0.79ms ? ?/sec 1.00 98.8±0.52ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q23 1.07 242.5±0.51ms ? ?/sec 1.00 226.4±3.64ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q24 1.01 19.3±0.10ms ? ?/sec 1.00 19.2±0.17ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q27 1.00 57.8±0.70ms ? ?/sec 1.00 57.9±0.64ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q28 1.01 58.5±0.76ms ? ?/sec 1.00 58.1±0.55ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q30 1.00 18.4±0.11ms ? ?/sec 1.00 18.4±0.16ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q36 1.00 15.1±0.27ms ? ?/sec 1.00 15.1±0.28ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q37 1.01 5.4±0.01ms ? ?/sec 1.00 5.3±0.02ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q38 1.00 13.2±0.21ms ? ?/sec 1.00 13.2±0.23ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q39 1.01 23.8±0.52ms ? ?/sec 1.00 23.5±0.38ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q40 1.00 5.5±0.05ms ? ?/sec 1.00 5.5±0.06ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q41 1.00 4.8±0.02ms ? ?/sec 1.00 4.8±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q42 1.00 3.4±0.01ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.00 891.2±1.41µs ? ?/sec 1.00 888.6±1.61µs ? ?/sec
arrow_reader_clickbench/sync/Q10 1.00 5.1±0.01ms ? ?/sec 1.00 5.1±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.00 6.0±0.02ms ? ?/sec 1.00 6.0±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.00 21.7±0.05ms ? ?/sec 1.00 21.8±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.16 28.5±0.12ms ? ?/sec 1.00 24.6±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.00 23.1±0.07ms ? ?/sec 1.00 23.2±0.09ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.01 125.2±0.17ms ? ?/sec 1.00 124.6±0.23ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.01 99.7±0.14ms ? ?/sec 1.00 99.2±0.15ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.02 147.4±0.34ms ? ?/sec 1.00 144.7±0.28ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.00 304.4±8.00ms ? ?/sec 1.01 307.1±9.65ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.00 26.9±0.09ms ? ?/sec 1.01 27.1±0.13ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.00 109.8±0.11ms ? ?/sec 1.01 110.7±0.13ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.00 107.7±0.13ms ? ?/sec 1.00 107.5±0.16ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.00 18.7±0.06ms ? ?/sec 1.00 18.7±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.00 22.1±0.12ms ? ?/sec 1.01 22.2±0.09ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.00 6.7±0.01ms ? ?/sec 1.00 6.8±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.00 11.4±0.04ms ? ?/sec 1.01 11.5±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.00 20.7±0.06ms ? ?/sec 1.01 20.8±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.00 5.2±0.02ms ? ?/sec 1.00 5.2±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.00 5.6±0.02ms ? ?/sec 1.00 5.6±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.00 4.2±0.02ms ? ?/sec 1.01 4.3±0.02ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 795.2s |
| Peak memory | 286.4 MiB |
| Avg memory | 134.0 MiB |
| CPU user | 702.5s |
| CPU sys | 85.3s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 790.2s |
| Peak memory | 286.1 MiB |
| Avg memory | 130.9 MiB |
| CPU user | 699.8s |
| CPU sys | 87.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783293361
run benchmark arrow_reader arrow_reader_row_filter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5018786357
> The mask-backed Auto decision cost from fully enumerating mask_run_count, on the other hand, remains independently reproducible within the same binary and is still worth optimizing.
optimized this part in [3f89ed0](https://github.com/apache/arrow-rs/pull/10141/commits/3f89ed018856554434dd482b9f6157899427519a)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5040681307
Thanks for your reviews @hhhizzz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5102276302
https://github.com/apache/arrow-rs/pull/10450 pr is ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5066534191
Thanks for your review @alamb
i filed follow-up issues to track some performance improve
- #10422 — reuse the lazy selector cache in `into_selectors_vec` (review finding 4)
- #10423 — avoid eager mask→RLE expansion in mixed `intersection`/`union` (review finding 6)
- #10424 — split up `selection/mod.rs`
- #10425 — in-place bitwise ops for `intersect_masks`/`union_masks`
The `RowSelectionIter` slice-iterator specializations (review finding 5) are addressed in this PR directly in [86610b5](https://github.com/apache/arrow-rs/pull/10141/commits/86610b5bbb3a2a91c7cae070eddbb57fff24e015) and [bdf2726](https://github.com/apache/arrow-rs/pull/10141/commits/bdf2726c21930d5264b49c751c9b5c8f0fc569c4)
cc @hhhizzz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3643176544
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -135,12 +149,297 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(Box<MaskSelection>),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.mask().len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a.mask() == b.mask(),
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.mask().len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.mask().set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Borrowed iterator over the [`RowSelector`]s of a [`RowSelection`].
+#[derive(Debug)]
+pub struct RowSelectionIter<'a>(std::slice::Iter<'a, RowSelector>);
+
+impl<'a> Iterator for RowSelectionIter<'a> {
+ type Item = &'a RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
+ self.0.next()
+ }
+}
+
+#[inline]
+fn scan_ranges_from_selectors<I>(selectors: I, page_locations: &[PageLocation]) -> Vec<Range<u64>>
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut ranges: Vec<Range<u64>> = vec![];
+ let mut row_offset = 0;
+
+ let mut pages = page_locations.iter().peekable();
+ let mut selectors = selectors.into_iter();
+ let mut current_selector = selectors.next();
+ let mut current_page = pages.next();
+
+ let mut current_page_included = false;
+
+ while let Some((selector, page)) = current_selector.as_mut().zip(current_page) {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ current_page_included = true;
+ }
+
+ if let Some(next_page) = pages.peek() {
+ if row_offset + selector.row_count > next_page.first_row_index as usize {
+ let remaining_in_page = next_page.first_row_index as usize - row_offset;
+ selector.row_count -= remaining_in_page;
+ row_offset += remaining_in_page;
+ current_page = pages.next();
+ current_page_included = false;
+
+ continue;
+ } else {
+ if row_offset + selector.row_count == next_page.first_row_index as usize {
+ current_page = pages.next();
+ current_page_included = false;
+ }
+ row_offset += selector.row_count;
+ current_selector = selectors.next();
+ }
+ } else {
+ if !(selector.skip || current_page_included) {
+ let start = page.offset as u64;
+ let end = start + page.compressed_page_size as u64;
+ ranges.push(start..end);
+ }
+ current_selector = selectors.next()
+ }
+ }
+
+ ranges
+}
+
+#[inline]
+fn expand_to_batch_boundaries_from_selectors<I>(
+ selectors: I,
+ batch_size: usize,
+ total_rows: usize,
+) -> RowSelection
+where
+ I: IntoIterator<Item = RowSelector>,
+{
+ let mut expanded_ranges = Vec::new();
+ let mut row_offset = 0;
+
+ for selector in selectors {
+ if selector.skip {
+ row_offset += selector.row_count;
+ } else {
+ let start = row_offset;
+ let end = row_offset + selector.row_count;
+
+ // Expand start to batch boundary
+ let expanded_start = (start / batch_size) * batch_size;
+ // Expand end to batch boundary
+ let expanded_end = end.div_ceil(batch_size) * batch_size;
+ let expanded_end = expanded_end.min(total_rows);
+
+ expanded_ranges.push(expanded_start..expanded_end);
+ row_offset += selector.row_count;
+ }
+ }
+
+ // Sort ranges by start position
+ expanded_ranges.sort_by_key(|range| range.start);
+
+ // Merge overlapping or consecutive ranges
+ let mut merged_ranges: Vec<Range<usize>> = Vec::new();
+ for range in expanded_ranges {
+ if let Some(last) = merged_ranges.last_mut() {
+ if range.start <= last.end {
+ // Overlapping or consecutive - merge them
+ last.end = last.end.max(range.end);
+ } else {
+ // No overlap - add new range
+ merged_ranges.push(range);
+ }
+ } else {
+ // First range
+ merged_ranges.push(range);
+ }
+ }
+
+ RowSelection::from_consecutive_ranges(merged_ranges.into_iter(), total_rows)
}
impl RowSelection {
+ fn from_selectors(selectors: Vec<RowSelector>) -> Self {
+ Self {
+ inner: RowSelectionInner::Selectors(selectors),
+ }
+ }
+
+ /// Create a [`RowSelection`] from a packed [`BooleanBuffer`].
+ ///
+ /// Each set bit selects a row, each unset bit skips one. Unlike
+ /// [`Self::from_filters`], the bitmap is kept as-is rather than
+ /// eagerly run-length-encoded. [`Self::iter`] materializes and caches the
+ /// RLE form on first use; use [`MaskRunIter`] to stream the RLE form
+ /// directly from the bitmap.
+ pub fn from_boolean_buffer(mask: BooleanBuffer) -> Self {
+ Self {
+ inner: RowSelectionInner::Mask(Box::new(MaskSelection::new(mask))),
+ }
+ }
+
+ /// Returns the underlying mask if this selection is mask-backed.
+ ///
+ /// Public so that engines composing selections (e.g. DataFusion's
+ /// `ParquetAccessPlan::into_overall_row_selection`) can concatenate
+ /// mask-backed selections without materialising the RLE form.
+ pub fn as_mask(&self) -> Option<&BooleanBuffer> {
+ match &self.inner {
+ RowSelectionInner::Mask(m) => Some(m.mask()),
+ _ => None,
+ }
+ }
+
+ /// Consume the selection and return its internal storage.
+ pub(crate) fn into_inner(self) -> RowSelectionInner {
+ self.inner
+ }
+
+ /// Choose the automatic materialisation strategy without converting between
+ /// selector and mask backing.
+ #[inline]
+ pub(crate) fn auto_selection_strategy(&self, threshold: usize) -> RowSelectionStrategy {
+ let (total_rows, effective_count) = match &self.inner {
+ RowSelectionInner::Selectors(selectors) => {
+ selectors.iter().fold((0usize, 0usize), |(rows, count), s| {
+ if s.row_count > 0 {
+ (rows + s.row_count, count + 1)
+ } else {
+ (rows, count)
+ }
+ })
+ }
+ RowSelectionInner::Mask(mask) => {
+ let mask = mask.mask();
+ let total_rows = mask.len();
+ (total_rows, mask_run_count(mask))
+ }
+ };
+
+ if effective_count == 0 {
+ return RowSelectionStrategy::Mask;
+ }
+
+ if total_rows < effective_count.saturating_mul(threshold) {
+ RowSelectionStrategy::Mask
+ } else {
+ RowSelectionStrategy::Selectors
+ }
+ }
+
+ #[cfg(test)]
+ fn selectors(&self) -> Vec<RowSelector> {
+ self.iter().copied().collect()
+ }
+
+ fn into_selectors_vec(self) -> Vec<RowSelector> {
Review Comment:
track in https://github.com/apache/arrow-rs/issues/10422
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4795054381
run benchmark arrow_reader_clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4748859669
Thanks @alamb for the pointers. I looked at #6624, #7454, and the work that landed in #8733.
My understanding is:
- #6624 also used a boolean mask for row selection, but it added a separate `BooleanRowSelection` type instead of making `RowSelection` able to keep a `BooleanBuffer`.
- #7454 tried a broader reader design: it used bitmap/range selections during predicate pushdown, coordinated predicate and projection decoding, and added a small page cache.
- #8733 added the production path for choosing selector vs mask execution through `RowSelectionPolicy` and `RowSelectionCursor`.
- **This PR is smaller**: it lets `RowSelection` keep a caller-provided `BooleanBuffer` and pass it directly to the mask cursor.
This does not replace selector-backed selections. Selectors are still better for clustered/page-index-style selections.
The case this helps is when the caller already has a row-level bitmap, such as from an external index, FTS index, or bitmap index. Today that bitmap has to go through `BooleanBuffer -> Vec<RowSelector> -> mask` in some paths. Keeping the original `BooleanBuffer` avoids that extra conversion and avoids building a large selector list for fragmented masks.
So I see this as a small representation-layer improvement that complements #8733. #8733 added mask execution; this PR adds a direct bitmap-backed input path for it. It does not try to implement the broader adaptive predicate-pushdown/page-cache design from #7454.
cc @hhhizzz @XiangpengHao @zhuqi-lucas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3462896701
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -135,12 +143,439 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(BooleanBuffer),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a == b,
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Iterator over the [`RowSelector`]s of a [`RowSelection`].
+///
+/// Yields owned [`RowSelector`] values; mask-backed selections stream the
+/// run-length form via [`BitSliceIterator`] without allocation.
+#[derive(Debug)]
+pub enum RowSelectionIter<'a> {
+ Selectors(std::iter::Copied<std::slice::Iter<'a, RowSelector>>),
+ Mask(MaskRunIter<'a>),
+}
+
+impl Iterator for RowSelectionIter<'_> {
+ type Item = RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<RowSelector> {
+ match self {
+ Self::Selectors(it) => it.next(),
+ Self::Mask(it) => it.next(),
+ }
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`].
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
+ if let Some(p) = self.pending.take() {
+ return Some(p);
+ }
+ if self.finished {
+ return None;
+ }
+ match self.slices.next() {
Review Comment:
Much of this code appears untested
I ran code coverage like
```shell
cargo llvm-cov --html test -p parquet --features arrow
```
<img width="1111" height="942" alt="Image" src="https://github.com/user-attachments/assets/bbc86094-2da0-4240-bf32-0d94af99a87e" />
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -135,12 +143,439 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(BooleanBuffer),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a == b,
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Iterator over the [`RowSelector`]s of a [`RowSelection`].
+///
+/// Yields owned [`RowSelector`] values; mask-backed selections stream the
+/// run-length form via [`BitSliceIterator`] without allocation.
+#[derive(Debug)]
+pub enum RowSelectionIter<'a> {
+ Selectors(std::iter::Copied<std::slice::Iter<'a, RowSelector>>),
+ Mask(MaskRunIter<'a>),
+}
+
+impl Iterator for RowSelectionIter<'_> {
+ type Item = RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<RowSelector> {
+ match self {
Review Comment:
In general I think this will add a branch in the main loops - every chunk of bits will require matching -- it might make sense to figure out how to have iterators for the two different types, so that code paths that want to iterate over the selection can match outside
```rust
match iter {
RowSelectionIter::Selectors(iter) => { for i in iter ... }, // special loop for selector backed
RowSelectionIter::Mask(iter) => { for i in iter ... }, // special loop for mask backed
}
```
##########
parquet/src/arrow/push_decoder/reader_builder/mod.rs:
##########
@@ -916,7 +916,7 @@ mod tests {
#[test]
// Verify that the size of RowGroupDecoderState does not grow too large
fn test_structure_size() {
- assert_eq!(std::mem::size_of::<RowGroupDecoderState>(), 232);
+ assert_eq!(std::mem::size_of::<RowGroupDecoderState>(), 256);
Review Comment:
I think this is bigger because a `RowSelection` is now bigger.
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -412,7 +868,12 @@ impl RowSelection {
///
/// returned: NNNNNNNNYYNYN
pub fn intersection(&self, other: &Self) -> Self {
- intersect_row_selections(&self.selectors, &other.selectors)
+ if let (Some(l), Some(r)) = (self.as_mask(), other.as_mask()) {
+ return Self::from_boolean_buffer(intersect_masks(l, r));
+ }
+ let l = self.materialize_for_combine();
Review Comment:
for example, I think and_then can use some of the fancy BMI instructions that @devanbenz is investigating in https://github.com/apache/arrow-rs/pull/10136
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -135,12 +143,439 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(BooleanBuffer),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a == b,
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Iterator over the [`RowSelector`]s of a [`RowSelection`].
+///
+/// Yields owned [`RowSelector`] values; mask-backed selections stream the
+/// run-length form via [`BitSliceIterator`] without allocation.
+#[derive(Debug)]
+pub enum RowSelectionIter<'a> {
+ Selectors(std::iter::Copied<std::slice::Iter<'a, RowSelector>>),
+ Mask(MaskRunIter<'a>),
+}
+
+impl Iterator for RowSelectionIter<'_> {
+ type Item = RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<RowSelector> {
+ match self {
+ Self::Selectors(it) => it.next(),
+ Self::Mask(it) => it.next(),
+ }
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`].
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
Review Comment:
I think this is the main API that is not compatible -- it need to return `&RowSelector` -- if we could change that we can merge this PR before the next major release (with breaking API changes)
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -412,7 +868,12 @@ impl RowSelection {
///
/// returned: NNNNNNNNYYNYN
pub fn intersection(&self, other: &Self) -> Self {
- intersect_row_selections(&self.selectors, &other.selectors)
+ if let (Some(l), Some(r)) = (self.as_mask(), other.as_mask()) {
+ return Self::from_boolean_buffer(intersect_masks(l, r));
+ }
+ let l = self.materialize_for_combine();
Review Comment:
this is a clever idea -- as a follow on we could potentially implement specialized versions of each of these implementations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783521397
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783482356)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
and_then 1.48 862.5±15.70µs ? ?/sec 1.00 581.9±2.03µs ? ?/sec
from_filters 1.00 370.2±1.96µs ? ?/sec 1.01 372.2±5.17µs ? ?/sec
intersection 2.03 2.7±0.02ms ? ?/sec 1.00 1338.6±2.46µs ? ?/sec
union 2.38 3.1±0.03ms ? ?/sec 1.00 1294.3±3.77µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 14.0 MiB |
| Avg memory | 10.4 MiB |
| CPU user | 44.6s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 45.0s |
| Peak memory | 13.5 MiB |
| Avg memory | 10.8 MiB |
| CPU user | 30.9s |
| CPU sys | 9.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4784012350
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783293361)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 847.9±15.41µs ? ?/sec 1.01 852.8±14.53µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 953.1±7.08µs ? ?/sec 1.00 950.6±6.80µs ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 852.4±15.50µs ? ?/sec 1.01 857.3±14.60µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs 1.03 284.2±0.29µs ? ?/sec 1.00 276.7±0.27µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, half NULLs 1.02 388.3±0.73µs ? ?/sec 1.00 382.4±1.08µs ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, no NULLs 1.03 287.7±0.30µs ? ?/sec 1.00 279.4±0.19µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, mandatory, no NULLs 1.04 434.2±2.38µs ? ?/sec 1.00 416.2±4.07µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, half NULLs 1.00 463.6±2.76µs ? ?/sec 1.00 464.3±3.94µs ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, no NULLs 1.00 409.3±4.29µs ? ?/sec 1.03 419.7±7.41µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, mandatory, no NULLs 1.00 79.7±0.05µs ? ?/sec 1.00 79.8±0.19µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, half NULLs 1.02 110.5±0.17µs ? ?/sec 1.00 108.4±0.40µs ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, no NULLs 1.00 83.5±0.04µs ? ?/sec 1.00 83.6±0.17µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs 1.01 148.9±0.98µs ? ?/sec 1.00 147.3±1.17µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs, short string 1.00 142.8±0.88µs ? ?/sec 1.07 152.4±3.25µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, half NULLs 1.07 155.3±3.56µs ? ?/sec 1.00 145.4±0.77µs ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, no NULLs 1.00 153.2±1.54µs ? ?/sec 1.00 153.1±1.29µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.10 1064.5±13.79µs ? ?/sec 1.00 964.9±0.60µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 669.5±1.79µs ? ?/sec 1.00 670.5±0.32µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 964.8±2.22µs ? ?/sec 1.00 969.1±0.56µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 182.1±0.47µs ? ?/sec 1.04 188.6±0.37µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs 1.00 291.2±0.47µs ? ?/sec 1.01 293.4±0.35µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs 1.00 191.1±0.61µs ? ?/sec 1.01 193.2±0.30µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, mandatory, no NULLs 1.11 143.1±11.33µs ? ?/sec 1.00 128.9±0.42µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, half NULLs 1.00 183.0±0.42µs ? ?/sec 1.03 187.7±3.05µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, no NULLs 1.00 131.5±0.55µs ? ?/sec 1.01 132.7±0.26µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, mandatory, no NULLs 1.00 60.2±0.11µs ? ?/sec 1.02 61.1±0.06µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, half NULLs 1.00 147.9±0.32µs ? ?/sec 1.00 147.6±0.30µs ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, no NULLs 1.00 63.0±0.12µs ? ?/sec 1.02 64.0±0.04µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, mandatory, no NULLs 1.00 802.0±3.12µs ? ?/sec 1.01 813.5±13.88µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, half NULLs 1.00 511.8±0.94µs ? ?/sec 1.00 509.8±0.37µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, no NULLs 1.00 805.8±0.52µs ? ?/sec 1.00 803.9±0.57µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, mandatory, no NULLs 1.00 22.8±0.13µs ? ?/sec 1.01 23.1±0.10µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, half NULLs 1.01 132.4±0.30µs ? ?/sec 1.00 131.0±0.23µs ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, no NULLs 1.00 27.3±0.14µs ? ?/sec 1.01 27.5±0.08µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, mandatory, no NULLs 1.00 75.4±0.25µs ? ?/sec 1.00 75.2±0.18µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, half NULLs 1.01 128.3±0.22µs ? ?/sec 1.00 127.1±0.13µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, no NULLs 1.02 77.8±0.19µs ? ?/sec 1.00 76.5±0.24µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, mandatory, no NULLs 1.00 5.3±0.03µs ? ?/sec 1.01 5.3±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, half NULLs 1.02 94.1±0.18µs ? ?/sec 1.00 92.6±0.07µs ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, no NULLs 1.00 8.8±0.02µs ? ?/sec 1.00 8.8±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, mandatory, no NULLs 1.03 148.9±0.52µs ? ?/sec 1.00 144.9±0.41µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, half NULLs 1.00 196.3±0.54µs ? ?/sec 1.01 197.5±0.17µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, no NULLs 1.06 169.3±16.77µs ? ?/sec 1.00 160.3±12.95µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, mandatory, no NULLs 1.03 7.6±0.01µs ? ?/sec 1.00 7.4±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, half NULLs 1.00 127.3±0.09µs ? ?/sec 1.00 127.7±0.10µs ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, no NULLs 1.00 11.0±0.03µs ? ?/sec 1.00 10.9±0.01µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, mandatory, no NULLs 1.01 301.5±1.16µs ? ?/sec 1.00 297.8±0.90µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, half NULLs 1.00 265.6±2.80µs ? ?/sec 1.00 265.3±0.70µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, no NULLs 1.03 312.1±16.04µs ? ?/sec 1.00 301.9±0.87µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, mandatory, no NULLs 1.00 12.2±0.04µs ? ?/sec 1.00 12.1±0.03µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, half NULLs 1.01 125.4±0.26µs ? ?/sec 1.00 123.6±0.15µs ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, no NULLs 1.03 16.2±0.04µs ? ?/sec 1.00 15.8±0.02µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed increasing value 1.01 87.4±0.42µs ? ?/sec 1.00 86.4±0.42µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed single value 1.02 78.6±0.27µs ? ?/sec 1.00 77.3±0.46µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip increasing value 1.01 50.7±0.24µs ? ?/sec 1.00 50.2±0.23µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip single value 1.01 46.0±0.29µs ? ?/sec 1.00 45.3±0.28µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip stepped increasing value 1.00 73.9±0.07µs ? ?/sec 1.00 73.9±0.23µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 87.7±0.21µs ? ?/sec 1.00 87.3±0.22µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs 1.00 89.2±0.32µs ? ?/sec 1.00 89.7±0.16µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs 1.00 89.9±0.31µs ? ?/sec 1.00 89.8±0.26µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed stepped increasing value 1.00 106.4±0.26µs ? ?/sec 1.00 106.7±0.19µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, mandatory, no NULLs 1.00 123.3±0.68µs ? ?/sec 1.00 123.4±0.38µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, half NULLs 1.00 147.4±0.29µs ? ?/sec 1.00 147.9±0.30µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, no NULLs 1.00 127.2±0.85µs ? ?/sec 1.00 127.4±0.52µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 54.0±0.05µs ? ?/sec 1.00 54.0±0.06µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 111.7±0.17µs ? ?/sec 1.00 111.2±0.14µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.00 57.5±0.07µs ? ?/sec 1.00 57.6±0.06µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.00 84.1±0.18µs ? ?/sec 1.00 84.1±0.07µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, half NULLs 1.01 129.0±0.18µs ? ?/sec 1.00 128.4±0.18µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 87.8±0.07µs ? ?/sec 1.00 87.7±0.06µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, mandatory, no NULLs 1.01 47.0±0.09µs ? ?/sec 1.00 46.6±0.08µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, half NULLs 1.01 109.2±0.14µs ? ?/sec 1.00 108.7±0.14µs ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, no NULLs 1.00 50.7±0.07µs ? ?/sec 1.04 52.6±0.06µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed increasing value 1.00 77.3±0.19µs ? ?/sec 1.00 77.6±0.26µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed single value 1.00 74.0±0.32µs ? ?/sec 1.00 74.3±0.20µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip increasing value 1.00 43.2±0.10µs ? ?/sec 1.00 43.3±0.13µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip single value 1.00 41.5±0.10µs ? ?/sec 1.00 41.5±0.12µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip stepped increasing value 1.00 67.2±0.06µs ? ?/sec 1.00 67.4±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs 1.00 79.6±0.06µs ? ?/sec 1.00 79.4±0.16µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs 1.00 86.2±0.10µs ? ?/sec 1.00 86.0±0.12µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs 1.00 81.6±0.07µs ? ?/sec 1.00 81.8±0.16µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed stepped increasing value 1.00 99.1±0.13µs ? ?/sec 1.00 99.1±0.18µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, mandatory, no NULLs 1.00 114.7±0.11µs ? ?/sec 1.00 114.3±0.30µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, half NULLs 1.00 144.3±0.22µs ? ?/sec 1.00 144.3±0.20µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, no NULLs 1.00 118.1±0.19µs ? ?/sec 1.01 119.3±0.33µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs 1.00 83.1±0.11µs ? ?/sec 1.00 83.3±0.10µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, half NULLs 1.00 128.5±0.14µs ? ?/sec 1.00 128.4±0.20µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, no NULLs 1.01 88.3±0.12µs ? ?/sec 1.00 87.4±0.12µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, mandatory, no NULLs 1.01 86.9±0.09µs ? ?/sec 1.00 86.4±0.08µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, half NULLs 1.01 133.0±0.17µs ? ?/sec 1.00 131.7±0.17µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, no NULLs 1.00 90.8±0.11µs ? ?/sec 1.00 90.4±0.07µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, mandatory, no NULLs 1.00 54.1±0.08µs ? ?/sec 1.00 54.2±0.08µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, half NULLs 1.00 113.5±0.12µs ? ?/sec 1.00 113.7±0.17µs ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, no NULLs 1.00 58.1±0.08µs ? ?/sec 1.00 58.2±0.08µs ? ?/sec
arrow_array_reader/Int16Array/binary packed increasing value 1.02 53.4±0.54µs ? ?/sec 1.00 52.3±0.41µs ? ?/sec
arrow_array_reader/Int16Array/binary packed single value 1.03 45.2±0.29µs ? ?/sec 1.00 43.9±0.46µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip increasing value 1.02 33.4±0.24µs ? ?/sec 1.00 32.6±0.20µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip single value 1.02 28.6±0.18µs ? ?/sec 1.00 28.0±0.26µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip stepped increasing value 1.00 56.7±0.08µs ? ?/sec 1.00 56.5±0.06µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs 1.00 70.2±0.36µs ? ?/sec 1.00 70.4±0.35µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs 1.00 72.0±0.18µs ? ?/sec 1.00 72.2±0.17µs ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs 1.00 72.1±0.38µs ? ?/sec 1.01 72.5±0.33µs ? ?/sec
arrow_array_reader/Int16Array/binary packed stepped increasing value 1.00 73.2±0.13µs ? ?/sec 1.00 73.0±0.11µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, mandatory, no NULLs 1.00 89.1±0.68µs ? ?/sec 1.00 89.3±0.69µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, half NULLs 1.00 113.1±0.34µs ? ?/sec 1.01 114.0±0.38µs ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, no NULLs 1.00 92.5±0.66µs ? ?/sec 1.00 92.8±0.66µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, mandatory, no NULLs 1.01 21.1±0.04µs ? ?/sec 1.00 20.9±0.04µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, half NULLs 1.00 78.2±0.11µs ? ?/sec 1.00 78.4±0.08µs ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.7±0.04µs ? ?/sec 1.00 24.7±0.05µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, mandatory, no NULLs 1.01 51.1±0.05µs ? ?/sec 1.00 50.8±0.04µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, half NULLs 1.00 95.6±0.13µs ? ?/sec 1.00 95.7±0.08µs ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, no NULLs 1.00 54.7±0.05µs ? ?/sec 1.00 54.6±0.06µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, mandatory, no NULLs 1.00 13.7±0.04µs ? ?/sec 1.01 13.9±0.03µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, half NULLs 1.00 75.2±0.09µs ? ?/sec 1.00 75.5±0.09µs ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, no NULLs 1.00 17.5±0.03µs ? ?/sec 1.01 17.6±0.05µs ? ?/sec
arrow_array_reader/Int32Array/binary packed increasing value 1.00 49.9±0.44µs ? ?/sec 1.00 50.0±0.57µs ? ?/sec
arrow_array_reader/Int32Array/binary packed single value 1.01 41.7±0.25µs ? ?/sec 1.00 41.4±0.24µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip increasing value 1.01 31.7±0.19µs ? ?/sec 1.00 31.4±0.32µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip single value 1.00 27.0±0.11µs ? ?/sec 1.00 26.9±0.14µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip stepped increasing value 1.00 54.7±0.05µs ? ?/sec 1.00 54.6±0.06µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs 1.00 68.8±0.19µs ? ?/sec 1.00 69.0±0.17µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs 1.00 71.2±0.13µs ? ?/sec 1.00 71.3±0.23µs ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs 1.00 71.0±0.16µs ? ?/sec 1.01 71.6±0.18µs ? ?/sec
arrow_array_reader/Int32Array/binary packed stepped increasing value 1.00 69.4±0.11µs ? ?/sec 1.00 69.4±0.13µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, mandatory, no NULLs 1.00 86.2±0.34µs ? ?/sec 1.01 86.9±0.40µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, half NULLs 1.00 111.1±0.25µs ? ?/sec 1.01 111.6±0.26µs ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, no NULLs 1.00 90.1±0.32µs ? ?/sec 1.01 91.0±0.37µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, mandatory, no NULLs 1.00 17.4±0.04µs ? ?/sec 1.00 17.4±0.04µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, half NULLs 1.01 75.5±0.09µs ? ?/sec 1.00 75.1±0.10µs ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, no NULLs 1.00 20.8±0.02µs ? ?/sec 1.02 21.2±0.03µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, mandatory, no NULLs 1.00 47.3±0.04µs ? ?/sec 1.00 47.4±0.05µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, half NULLs 1.00 92.1±0.09µs ? ?/sec 1.01 92.7±0.11µs ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, no NULLs 1.00 51.1±0.04µs ? ?/sec 1.00 51.2±0.05µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, mandatory, no NULLs 1.00 9.9±0.02µs ? ?/sec 1.02 10.1±0.05µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, half NULLs 1.00 72.5±0.08µs ? ?/sec 1.00 72.4±0.09µs ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, no NULLs 1.00 13.7±0.03µs ? ?/sec 1.00 13.7±0.04µs ? ?/sec
arrow_array_reader/Int64Array/binary packed increasing value 1.00 40.5±0.19µs ? ?/sec 1.01 40.8±0.14µs ? ?/sec
arrow_array_reader/Int64Array/binary packed single value 1.00 37.6±0.22µs ? ?/sec 1.01 38.1±0.16µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip increasing value 1.00 24.2±0.08µs ? ?/sec 1.01 24.4±0.09µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip single value 1.00 22.7±0.09µs ? ?/sec 1.01 22.8±0.07µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip stepped increasing value 1.00 48.1±0.06µs ? ?/sec 1.00 48.1±0.08µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs 1.00 60.4±0.05µs ? ?/sec 1.00 60.4±0.05µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs 1.00 67.5±0.08µs ? ?/sec 1.00 67.2±0.07µs ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs 1.00 62.5±0.08µs ? ?/sec 1.00 62.6±0.06µs ? ?/sec
arrow_array_reader/Int64Array/binary packed stepped increasing value 1.00 62.1±0.13µs ? ?/sec 1.01 62.6±0.20µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, mandatory, no NULLs 1.00 77.3±0.13µs ? ?/sec 1.00 77.2±0.12µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, half NULLs 1.01 108.2±0.13µs ? ?/sec 1.00 107.4±0.14µs ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, no NULLs 1.00 81.4±0.14µs ? ?/sec 1.00 81.4±0.13µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, mandatory, no NULLs 1.00 45.5±0.04µs ? ?/sec 1.01 46.0±0.04µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, half NULLs 1.00 91.7±0.10µs ? ?/sec 1.00 91.4±0.14µs ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, no NULLs 1.00 49.7±0.06µs ? ?/sec 1.00 49.9±0.05µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, mandatory, no NULLs 1.00 49.6±0.04µs ? ?/sec 1.00 49.6±0.03µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, half NULLs 1.00 94.8±0.11µs ? ?/sec 1.00 94.5±0.13µs ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, no NULLs 1.01 53.7±0.07µs ? ?/sec 1.00 53.3±0.05µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, mandatory, no NULLs 1.00 16.2±0.03µs ? ?/sec 1.02 16.5±0.03µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, half NULLs 1.00 77.4±0.11µs ? ?/sec 1.00 77.1±0.14µs ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, no NULLs 1.00 20.2±0.03µs ? ?/sec 1.01 20.5±0.02µs ? ?/sec
arrow_array_reader/Int8Array/binary packed increasing value 1.00 53.4±0.37µs ? ?/sec 1.00 53.3±0.58µs ? ?/sec
arrow_array_reader/Int8Array/binary packed single value 1.00 44.9±0.21µs ? ?/sec 1.00 44.9±0.25µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip increasing value 1.00 33.3±0.19µs ? ?/sec 1.00 33.3±0.30µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip single value 1.00 28.6±0.19µs ? ?/sec 1.00 28.5±0.14µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip stepped increasing value 1.00 56.4±0.07µs ? ?/sec 1.00 56.3±0.06µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs 1.00 70.0±0.32µs ? ?/sec 1.00 69.9±0.29µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs 1.00 71.8±0.23µs ? ?/sec 1.00 71.8±0.18µs ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs 1.00 72.2±0.32µs ? ?/sec 1.00 72.0±0.31µs ? ?/sec
arrow_array_reader/Int8Array/binary packed stepped increasing value 1.00 72.8±0.15µs ? ?/sec 1.00 72.7±0.16µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, mandatory, no NULLs 1.00 88.7±0.53µs ? ?/sec 1.00 88.8±0.55µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, half NULLs 1.00 112.8±0.32µs ? ?/sec 1.00 112.6±0.33µs ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, no NULLs 1.00 92.3±0.57µs ? ?/sec 1.00 92.5±0.55µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, mandatory, no NULLs 1.02 21.1±0.05µs ? ?/sec 1.00 20.6±0.03µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, half NULLs 1.01 78.2±0.08µs ? ?/sec 1.00 77.4±0.11µs ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, no NULLs 1.01 24.4±0.04µs ? ?/sec 1.00 24.2±0.04µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, mandatory, no NULLs 1.00 50.5±0.06µs ? ?/sec 1.01 50.9±0.06µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, half NULLs 1.00 95.6±0.11µs ? ?/sec 1.00 95.8±0.14µs ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, no NULLs 1.00 54.4±0.06µs ? ?/sec 1.00 54.7±0.05µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, mandatory, no NULLs 1.00 13.7±0.03µs ? ?/sec 1.00 13.6±0.04µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, half NULLs 1.00 74.6±0.12µs ? ?/sec 1.00 74.8±0.12µs ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, no NULLs 1.00 17.5±0.04µs ? ?/sec 1.02 17.8±0.03µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/90pct NULLs 1.04 1005.8±9.32µs ? ?/sec 1.00 967.6±8.68µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/99pct NULLs 1.08 456.7±11.87µs ? ?/sec 1.00 421.5±12.01µs ? ?/sec
arrow_array_reader/ListArray/Fixed32List/half NULLs 1.00 2.7±0.02ms ? ?/sec 1.01 2.7±0.01ms ? ?/sec
arrow_array_reader/ListArray/Fixed32List/no NULLs 1.00 4.3±0.04ms ? ?/sec 1.02 4.4±0.04ms ? ?/sec
arrow_array_reader/ListArray/Int32List/90pct NULLs 1.00 923.2±5.45µs ? ?/sec 1.00 923.0±5.43µs ? ?/sec
arrow_array_reader/ListArray/Int32List/99pct NULLs 1.01 412.4±11.96µs ? ?/sec 1.00 410.3±11.51µs ? ?/sec
arrow_array_reader/ListArray/Int32List/half NULLs 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
arrow_array_reader/ListArray/Int32List/no NULLs 1.00 2.9±0.01ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
arrow_array_reader/ListArray/StringList/90pct NULLs 1.00 1095.2±5.87µs ? ?/sec 1.00 1095.0±6.39µs ? ?/sec
arrow_array_reader/ListArray/StringList/99pct NULLs 1.00 425.6±12.75µs ? ?/sec 1.00 424.5±13.11µs ? ?/sec
arrow_array_reader/ListArray/StringList/half NULLs 1.00 3.9±0.01ms ? ?/sec 1.00 3.9±0.02ms ? ?/sec
arrow_array_reader/ListArray/StringList/no NULLs 1.00 7.1±0.04ms ? ?/sec 1.00 7.1±0.05ms ? ?/sec
arrow_array_reader/StringArray/const delta byte array encoded, mandatory, no NULLs 1.00 538.6±2.22µs ? ?/sec 1.01 541.8±2.59µs ? ?/sec
arrow_array_reader/StringArray/const delta length byte array encoded, mandatory, no NULLs 1.01 221.1±1.06µs ? ?/sec 1.00 219.6±1.35µs ? ?/sec
arrow_array_reader/StringArray/const prefix delta byte array encoded, mandatory, no NULLs 1.00 758.7±39.57µs ? ?/sec 1.03 782.9±44.69µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, mandatory, no NULLs 1.05 283.7±0.27µs ? ?/sec 1.00 269.7±0.27µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, half NULLs 1.03 389.4±0.64µs ? ?/sec 1.00 379.6±0.65µs ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, no NULLs 1.05 287.1±0.26µs ? ?/sec 1.00 273.4±0.24µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, mandatory, no NULLs 1.00 454.7±4.75µs ? ?/sec 1.02 463.6±6.82µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, half NULLs 1.00 488.6±2.45µs ? ?/sec 1.01 491.9±4.18µs ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, no NULLs 1.00 459.6±5.21µs ? ?/sec 1.01 466.2±7.76µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, mandatory, no NULLs 1.00 265.6±0.33µs ? ?/sec 1.00 266.7±0.71µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, half NULLs 1.00 276.0±0.34µs ? ?/sec 1.00 275.5±0.48µs ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, no NULLs 1.00 269.9±0.26µs ? ?/sec 1.00 270.1±0.43µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, mandatory, no NULLs 1.00 79.8±0.08µs ? ?/sec 1.00 79.9±0.11µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, half NULLs 1.02 110.9±0.11µs ? ?/sec 1.00 109.0±0.35µs ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, no NULLs 1.00 83.6±0.08µs ? ?/sec 1.00 83.6±0.06µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, mandatory, no NULLs 1.00 218.8±0.31µs ? ?/sec 1.00 218.9±0.41µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, half NULLs 1.01 184.1±0.25µs ? ?/sec 1.00 181.8±0.34µs ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, no NULLs 1.00 225.8±0.40µs ? ?/sec 1.00 226.3±0.40µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed increasing value 1.00 53.2±0.59µs ? ?/sec 1.00 53.5±0.60µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed single value 1.00 44.9±0.29µs ? ?/sec 1.00 45.1±0.26µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip increasing value 1.00 33.4±0.28µs ? ?/sec 1.00 33.5±0.33µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip single value 1.00 28.7±0.12µs ? ?/sec 1.00 28.6±0.12µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip stepped increasing value 1.00 56.5±0.07µs ? ?/sec 1.00 56.5±0.07µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs 1.00 73.4±0.13µs ? ?/sec 1.00 73.4±0.10µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs 1.00 74.3±0.12µs ? ?/sec 1.00 74.3±0.10µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs 1.00 75.6±0.14µs ? ?/sec 1.00 75.6±0.17µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed stepped increasing value 1.00 73.4±0.12µs ? ?/sec 1.00 73.1±0.15µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, mandatory, no NULLs 1.00 92.5±0.25µs ? ?/sec 1.00 92.9±0.26µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, half NULLs 1.00 116.3±0.21µs ? ?/sec 1.00 115.7±0.22µs ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, no NULLs 1.00 96.3±0.23µs ? ?/sec 1.00 96.5±0.26µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.9±0.04µs ? ?/sec 1.00 20.9±0.02µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, half NULLs 1.01 78.4±0.09µs ? ?/sec 1.00 78.0±0.08µs ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, no NULLs 1.00 24.8±0.03µs ? ?/sec 1.00 24.7±0.02µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, mandatory, no NULLs 1.00 51.1±0.05µs ? ?/sec 1.00 51.1±0.06µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, half NULLs 1.00 95.9±0.16µs ? ?/sec 1.00 95.7±0.07µs ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, no NULLs 1.00 54.9±0.06µs ? ?/sec 1.00 54.8±0.04µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, mandatory, no NULLs 1.02 14.1±0.04µs ? ?/sec 1.00 13.8±0.03µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, half NULLs 1.01 75.6±0.24µs ? ?/sec 1.00 75.2±0.09µs ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, no NULLs 1.00 17.7±0.03µs ? ?/sec 1.01 17.8±0.04µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed increasing value 1.00 50.0±0.57µs ? ?/sec 1.00 50.1±0.55µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed single value 1.00 41.6±0.28µs ? ?/sec 1.00 41.8±0.24µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip increasing value 1.00 31.5±0.27µs ? ?/sec 1.00 31.5±0.26µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip single value 1.01 27.0±0.11µs ? ?/sec 1.00 26.8±0.13µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip stepped increasing value 1.00 54.8±0.05µs ? ?/sec 1.00 54.7±0.06µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs 1.00 68.9±0.20µs ? ?/sec 1.00 69.0±0.16µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs 1.00 70.8±0.11µs ? ?/sec 1.00 70.9±0.11µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs 1.00 70.7±0.19µs ? ?/sec 1.00 71.0±0.15µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed stepped increasing value 1.00 69.6±0.13µs ? ?/sec 1.00 69.5±0.13µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, mandatory, no NULLs 1.00 86.6±0.37µs ? ?/sec 1.00 87.0±0.39µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, half NULLs 1.00 110.9±0.22µs ? ?/sec 1.00 111.3±0.24µs ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, no NULLs 1.00 90.4±0.35µs ? ?/sec 1.00 90.8±0.29µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, mandatory, no NULLs 1.00 17.5±0.02µs ? ?/sec 1.00 17.5±0.03µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, half NULLs 1.01 75.4±0.09µs ? ?/sec 1.00 74.6±0.10µs ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, no NULLs 1.00 21.1±0.04µs ? ?/sec 1.00 21.1±0.02µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, mandatory, no NULLs 1.01 47.7±0.04µs ? ?/sec 1.00 47.4±0.03µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, half NULLs 1.00 92.1±0.11µs ? ?/sec 1.01 92.7±0.10µs ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, no NULLs 1.00 51.2±0.04µs ? ?/sec 1.02 52.2±0.04µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, mandatory, no NULLs 1.00 10.1±0.03µs ? ?/sec 1.01 10.2±0.02µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, half NULLs 1.00 72.8±0.10µs ? ?/sec 1.00 72.9±0.16µs ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, no NULLs 1.00 14.0±0.04µs ? ?/sec 1.00 13.9±0.02µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed increasing value 1.00 40.6±0.16µs ? ?/sec 1.02 41.4±0.18µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed single value 1.00 37.6±0.19µs ? ?/sec 1.01 38.1±0.11µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip increasing value 1.00 24.3±0.05µs ? ?/sec 1.01 24.5±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip single value 1.00 22.9±0.09µs ? ?/sec 1.01 23.1±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip stepped increasing value 1.00 48.2±0.05µs ? ?/sec 1.00 48.2±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs 1.00 60.6±0.06µs ? ?/sec 1.00 60.3±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs 1.00 67.6±0.10µs ? ?/sec 1.00 67.4±0.08µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs 1.00 62.6±0.06µs ? ?/sec 1.00 62.5±0.05µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed stepped increasing value 1.00 62.3±0.13µs ? ?/sec 1.00 62.4±0.12µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, mandatory, no NULLs 1.00 77.5±0.12µs ? ?/sec 1.00 77.5±0.11µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, half NULLs 1.01 108.4±0.18µs ? ?/sec 1.00 107.8±0.17µs ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, no NULLs 1.00 81.5±0.12µs ? ?/sec 1.00 81.6±0.10µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, mandatory, no NULLs 1.01 46.0±0.03µs ? ?/sec 1.00 45.7±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, half NULLs 1.00 91.8±0.17µs ? ?/sec 1.00 91.5±0.12µs ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, no NULLs 1.00 49.8±0.07µs ? ?/sec 1.00 49.7±0.05µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, mandatory, no NULLs 1.00 49.7±0.06µs ? ?/sec 1.00 49.9±0.04µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, half NULLs 1.00 95.0±0.14µs ? ?/sec 1.00 94.8±0.12µs ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, no NULLs 1.01 53.8±0.09µs ? ?/sec 1.00 53.4±0.06µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, mandatory, no NULLs 1.00 16.3±0.04µs ? ?/sec 1.02 16.5±0.03µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, half NULLs 1.00 76.8±0.10µs ? ?/sec 1.00 76.7±0.10µs ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, no NULLs 1.00 20.4±0.03µs ? ?/sec 1.01 20.6±0.05µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed increasing value 1.00 52.0±0.44µs ? ?/sec 1.04 53.9±0.40µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed single value 1.00 43.7±0.53µs ? ?/sec 1.03 45.0±0.17µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip increasing value 1.00 32.8±0.18µs ? ?/sec 1.02 33.4±0.20µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip single value 1.00 28.0±0.32µs ? ?/sec 1.02 28.6±0.13µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip stepped increasing value 1.00 56.6±0.09µs ? ?/sec 1.00 56.4±0.09µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs 1.00 72.2±0.34µs ? ?/sec 1.00 72.5±0.21µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs 1.00 73.8±0.23µs ? ?/sec 1.00 73.5±0.16µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs 1.00 74.6±0.43µs ? ?/sec 1.00 74.8±0.23µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed stepped increasing value 1.00 73.1±0.17µs ? ?/sec 1.00 72.9±0.14µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, mandatory, no NULLs 1.00 90.9±0.66µs ? ?/sec 1.01 92.0±0.41µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, half NULLs 1.00 115.0±0.31µs ? ?/sec 1.00 115.3±0.29µs ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, no NULLs 1.00 95.0±0.66µs ? ?/sec 1.01 95.7±0.44µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, mandatory, no NULLs 1.00 20.8±0.05µs ? ?/sec 1.00 20.8±0.04µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, half NULLs 1.00 77.8±0.09µs ? ?/sec 1.00 77.7±0.10µs ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, no NULLs 1.00 24.5±0.05µs ? ?/sec 1.02 24.9±0.03µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, mandatory, no NULLs 1.00 50.7±0.05µs ? ?/sec 1.00 50.7±0.05µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, half NULLs 1.01 96.1±0.19µs ? ?/sec 1.00 95.4±0.12µs ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, no NULLs 1.00 54.6±0.08µs ? ?/sec 1.00 54.4±0.05µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, mandatory, no NULLs 1.00 13.7±0.04µs ? ?/sec 1.01 13.8±0.02µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, half NULLs 1.01 76.0±0.09µs ? ?/sec 1.00 75.4±0.11µs ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, no NULLs 1.00 17.4±0.03µs ? ?/sec 1.00 17.4±0.02µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, half NULLs 1.00 72.8±0.09µs ? ?/sec 1.00 72.9±0.10µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, mandatory struct, optional data, no NULLs 1.00 13.9±0.03µs ? ?/sec 1.01 14.0±0.02µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, half NULLs 1.00 133.3±0.17µs ? ?/sec 1.00 133.0±0.14µs ? ?/sec
arrow_array_reader/struct/Int32Array/plain encoded, optional struct, optional data, no NULLs 1.00 65.4±0.06µs ? ?/sec 1.00 65.1±0.05µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 2930.7s |
| Peak memory | 71.7 MiB |
| Avg memory | 28.6 MiB |
| CPU user | 2926.6s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 2925.7s |
| Peak memory | 71.6 MiB |
| Avg memory | 25.6 MiB |
| CPU user | 2918.9s |
| CPU sys | 0.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787493427
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787465734)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4787465734-650-xqjtv 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (0ffbf26319f13b931c40678b0480c6a3ea9cbcaa) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..0ffbf26319f13b931c40678b0480c6a3ea9cbcaa)
BENCH_NAME=row_selector
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench row_selector
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787465734
run benchmark arrow_reader_clickbench arrow_reader_row_filter row_selector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3465891615
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -412,7 +868,12 @@ impl RowSelection {
///
/// returned: NNNNNNNNYYNYN
pub fn intersection(&self, other: &Self) -> Self {
- intersect_row_selections(&self.selectors, &other.selectors)
+ if let (Some(l), Some(r)) = (self.as_mask(), other.as_mask()) {
+ return Self::from_boolean_buffer(intersect_masks(l, r));
+ }
+ let l = self.materialize_for_combine();
Review Comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5029890469
resolve conflict and added test case in [d4fa2be](https://github.com/apache/arrow-rs/pull/10141/commits/d4fa2be9c0a1f26348e1980bf846bdb83ecc6287)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783502348
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783482356)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4783482356-640-s7mgp 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (7b102f41c4e92c03788a991dfd7d59be4fd67b21) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..7b102f41c4e92c03788a991dfd7d59be4fd67b21)
BENCH_NAME=row_selection_cursor
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench row_selection_cursor
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787709681
run benchmark arrow_reader_clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787747341
the `arrow_reader_row_filter` `row_selector` benchmark almost same to main branch, but `arrow_reader_clickbench` still slower than main branch(the not equal to main branch result show in below), i retrigger again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783526299
row_selector benchmark is also quite a bit slower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5015837950
> Could we rerun the complete L4–L40 curve in row_selection_cursor on the current head and compare:
>
> 1. the Selector/Mask crossover against the merge-base;
> 2. the strategy selected by Auto at each point; and
> 3. Auto's runtime against the faster forced strategy?
>
> Since this PR adds a second input representation, it would be particularly
> useful to run the matrix for both selector-backed and mask-backed
> RowSelections. That would show whether Auto should use a different threshold
> or cost model when choosing a strategy requires first converting the backing
> representation.
the command for below figure benchmark that [325d774](https://github.com/apache/arrow-rs/pull/10141/commits/325d774420a49430e05399f401d7945ee93901e1) added the bitmap input representation into row_selection_cursor
```
cargo bench -p parquet --bench row_selection_cursor --features arrow,test_common -- "len-uniform50-uniform50" --save-baseline 10141-auto
# selector-backed input
critcmp 10141-auto -g '.*?/(.*)' -f "read_(auto|selector|mask)/len-uniform50-uniform50"
# mask-backed input
critcmp 10141-auto -g '.*?/(.*)' -f "mask_backed/len-uniform50-uniform50"
```
<img width="1182" height="461" alt="image" src="https://github.com/user-attachments/assets/3d983aa9-8dc1-4f9c-a009-2e995e39a650" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3643101412
##########
parquet/src/arrow/arrow_reader/selection/mod.rs:
##########
@@ -135,12 +149,297 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(Box<MaskSelection>),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.mask().len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a.mask() == b.mask(),
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.mask().len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.mask().set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Borrowed iterator over the [`RowSelector`]s of a [`RowSelection`].
+#[derive(Debug)]
+pub struct RowSelectionIter<'a>(std::slice::Iter<'a, RowSelector>);
+
+impl<'a> Iterator for RowSelectionIter<'a> {
+ type Item = &'a RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<Self::Item> {
Review Comment:
apply in [86610b5](https://github.com/apache/arrow-rs/pull/10141/commits/86610b5bbb3a2a91c7cae070eddbb57fff24e015) and [bdf2726](https://github.com/apache/arrow-rs/pull/10141/commits/bdf2726c21930d5264b49c751c9b5c8f0fc569c4)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3643180870
##########
parquet/src/arrow/arrow_reader/selection/boolean.rs:
##########
@@ -0,0 +1,1308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::{LoadedRowRanges, RowSelection, RowSelectionInner, RowSelector};
+use crate::errors::ParquetError;
+use arrow_array::BooleanArray;
+use arrow_buffer::bit_iterator::BitSliceIterator;
+use arrow_buffer::{BooleanBuffer, BooleanBufferBuilder, Buffer};
+use std::cmp::Ordering;
+use std::sync::{Arc, OnceLock};
+
+/// Mask-backed [`RowSelection`] storage.
+///
+/// `selectors` is only populated if callers use the borrowed [`RowSelection::iter`]
+/// compatibility API. Internal paths that can stream or consume the bitmap avoid
+/// this cache.
+///
+/// `count` caches the popcount; `RowSelection::split_off` propagates it to
+/// both halves so repeated `row_count()` calls do not rescan the bitmap.
+#[derive(Debug)]
+pub(crate) struct MaskSelection {
+ mask: BooleanBuffer,
+ selectors: OnceLock<Vec<RowSelector>>,
+ count: OnceLock<usize>,
+}
+
+impl MaskSelection {
+ pub(super) fn new(mask: BooleanBuffer) -> Self {
+ Self {
+ mask,
+ selectors: OnceLock::new(),
+ count: OnceLock::new(),
+ }
+ }
+
+ /// Create a selection whose selected-row count is already known.
+ pub(super) fn with_count(mask: BooleanBuffer, count: usize) -> Self {
+ debug_assert!(count <= mask.len());
+ let cell = OnceLock::new();
+ let _ = cell.set(count);
+ Self {
+ mask,
+ selectors: OnceLock::new(),
+ count: cell,
+ }
+ }
+
+ pub(crate) fn mask(&self) -> &BooleanBuffer {
+ &self.mask
+ }
+
+ pub(crate) fn into_mask(self) -> BooleanBuffer {
+ let Self { mask, .. } = self;
+ mask
+ }
+
+ /// Number of selected rows, computed once and cached.
+ pub(super) fn count(&self) -> usize {
+ *self.count.get_or_init(|| self.mask.count_set_bits())
+ }
+
+ /// The cached selected-row count, if it has been computed.
+ pub(super) fn cached_count(&self) -> Option<usize> {
+ self.count.get().copied()
+ }
+
+ pub(super) fn selectors(&self) -> &[RowSelector] {
+ self.selectors
+ .get_or_init(|| mask_to_selectors(&self.mask))
+ .as_slice()
+ }
+}
+
+impl Clone for MaskSelection {
+ fn clone(&self) -> Self {
+ // Drop the selector cache but keep the cheap count cache.
+ Self {
+ mask: self.mask.clone(),
+ selectors: OnceLock::new(),
+ count: self.count.clone(),
+ }
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`], yielding owned [`RowSelector`]s
+/// without allocation.
+///
+/// Useful as a zero-cost alternative to [`RowSelection::iter`] for mask-backed
+/// selections, via [`RowSelection::as_mask`]:
+///
+/// ```ignore
+/// if let Some(mask) = selection.as_mask() {
+/// for run in MaskRunIter::new(mask) { ... }
+/// }
+/// ```
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ /// Create a streaming RLE iterator over a [`BooleanBuffer`].
+ pub fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
+ if let Some(p) = self.pending.take() {
+ return Some(p);
+ }
+ if self.finished {
+ return None;
+ }
+ match self.slices.next() {
+ Some((start, end)) => {
+ let select = RowSelector::select(end - start);
+ if start > self.cursor {
+ let skip = RowSelector::skip(start - self.cursor);
+ self.pending = Some(select);
+ self.cursor = end;
+ Some(skip)
+ } else {
+ self.cursor = end;
+ Some(select)
+ }
+ }
+ None => {
+ self.finished = true;
+ if self.cursor < self.total {
+ let skip = RowSelector::skip(self.total - self.cursor);
+ self.cursor = self.total;
+ Some(skip)
+ } else {
+ None
+ }
+ }
+ }
+ }
+}
+
+/// Cursor for iterating a mask-backed [`RowSelection`]
+///
+/// This is best for dense selections where there are many small skips
+/// or selections. For example, selecting every other row.
+///
+/// When page pruning produces sparse column data, `loaded_row_ranges` limits
+/// each decoded chunk to rows whose pages are loaded for every projected leaf.
+/// For example, two projected columns can have different page boundaries:
+///
+/// ```text
+/// Row ranges: [0, 4) [4, 6) [6, 8) [8, 10) [10, 12)
+/// Selection mask: 1000 00 00 00 01
+/// Column A pages: loaded | missing [4, 8) | loaded [8, 12)
+/// Column B pages: loaded [0, 6) | missing [6, 10) | loaded
+/// LoadedRowRanges: [0, 4) [10, 12)
+/// ```
+///
+/// The first chunk decodes `[0, 4)` with mask `1000`. The next chunk skips to
+/// row 11 and decodes `[11, 12)` with mask `1`. The loaded ranges are decode
+/// boundaries, not output batch boundaries: [`ParquetRecordBatchReader`]
+/// accumulates both chunks and applies the combined mask `10001` once.
+///
+/// [`ParquetRecordBatchReader`]: crate::arrow::arrow_reader::ParquetRecordBatchReader
+#[derive(Debug)]
+pub struct MaskCursor {
+ pub(super) mask: BooleanBuffer,
+ /// Current absolute offset into the selection
+ pub(super) position: usize,
+ /// Row ranges whose backing pages are loaded for every projected column.
+ pub(super) loaded_row_ranges: Option<Arc<LoadedRowRanges>>,
+}
+
+impl MaskCursor {
+ /// Returns `true` when no further rows remain
+ pub fn is_empty(&self) -> bool {
+ self.position >= self.mask.len()
+ }
+
+ /// Advance through the mask representation, producing the next chunk summary
+ pub fn next_mask_chunk(&mut self, batch_size: usize) -> Option<MaskChunk> {
+ if self.is_empty() {
+ return None;
+ }
+
+ Some(self.next_mask_chunk_non_empty(batch_size))
+ }
+
+ /// Produces the next chunk for a non-empty, trailing-skip-free mask.
+ fn next_mask_chunk_non_empty(&mut self, batch_size: usize) -> MaskChunk {
+ debug_assert!(!self.is_empty());
+
+ let (initial_skip, chunk_rows, selected_rows, mask_start, end_position) = {
+ let mask = &self.mask;
+ let start_position = self.position;
+ let mut cursor = start_position;
+ let mut initial_skip = 0;
+
+ while cursor < mask.len() && !mask.value(cursor) {
+ initial_skip += 1;
+ cursor += 1;
+ }
+ debug_assert!(
+ cursor < mask.len(),
+ "ReadPlan must remove trailing skips from Mask selections"
+ );
+
+ let mask_start = cursor;
+ let mut chunk_rows = 0;
+ let mut selected_rows = 0;
+
+ // Advance until enough rows have been selected to satisfy the batch size,
+ // or until the mask is exhausted. This mirrors the behaviour of the legacy
+ // `RowSelector` queue-based iteration.
+ while cursor < mask.len() && selected_rows < batch_size {
+ chunk_rows += 1;
+ if mask.value(cursor) {
+ selected_rows += 1;
+ }
+ cursor += 1;
+ }
+
+ (initial_skip, chunk_rows, selected_rows, mask_start, cursor)
+ };
+
+ self.position = end_position;
+
+ MaskChunk {
+ initial_skip,
+ chunk_rows,
+ selected_rows,
+ mask_start,
+ }
+ }
+
+ /// Returns the next non-empty mask chunk without crossing an unloaded row range.
+ ///
+ /// The [`ReadPlan`](crate::arrow::arrow_reader::ReadPlan) removes trailing
+ /// skips before constructing this cursor. Callers therefore only invoke
+ /// this method for a non-empty mask that has another selected row.
+ pub(crate) fn next_chunk(&mut self, batch_size: usize) -> Result<MaskChunk, ParquetError> {
+ debug_assert!(batch_size > 0);
+ debug_assert!(!self.is_empty());
+
+ if self.loaded_row_ranges.is_none() {
+ return Ok(self.next_mask_chunk_non_empty(batch_size));
+ }
+
+ let start_position = self.position;
+ let mut cursor = start_position;
+ while cursor < self.mask.len() && !self.mask.value(cursor) {
+ cursor += 1;
+ }
+
+ debug_assert!(
+ cursor < self.mask.len(),
+ "ReadPlan must remove trailing skips from Mask selections"
+ );
+
+ let loaded_range_end = self
+ .loaded_row_ranges
+ .as_ref()
+ .and_then(|ranges| ranges.end_containing(cursor))
+ .ok_or_else(|| {
+ ParquetError::General(format!(
+ "Internal Error: selected row {cursor} has no loaded page range"
+ ))
+ })?;
+
+ let mask_start = cursor;
+ let mut selected_rows = 0;
+ while cursor < loaded_range_end && cursor < self.mask.len() && selected_rows < batch_size {
+ if self.mask.value(cursor) {
+ selected_rows += 1;
+ }
+ cursor += 1;
+ }
+
+ self.position = cursor;
+ Ok(MaskChunk {
+ initial_skip: mask_start - start_position,
+ chunk_rows: cursor - mask_start,
+ selected_rows,
+ mask_start,
+ })
+ }
+
+ /// Materialise the boolean values for a mask-backed chunk
+ pub fn mask_values_for(&self, chunk: &MaskChunk) -> Result<BooleanArray, ParquetError> {
+ if chunk.mask_start.saturating_add(chunk.chunk_rows) > self.mask.len() {
+ return Err(ParquetError::General(
+ "Internal Error: MaskChunk exceeds mask length".to_string(),
+ ));
+ }
+ Ok(BooleanArray::from(
+ self.mask.slice(chunk.mask_start, chunk.chunk_rows),
+ ))
+ }
+}
+
+/// Result of computing the next chunk to read when using a [`MaskCursor`]
+#[derive(Debug)]
+pub struct MaskChunk {
+ /// Number of leading rows to skip before reaching selected rows
+ pub initial_skip: usize,
+ /// Total rows covered by this chunk (selected + skipped)
+ pub chunk_rows: usize,
+ /// Rows actually selected within the chunk
+ pub selected_rows: usize,
+ /// Starting offset within the mask where the chunk begins
+ pub mask_start: usize,
+}
+
+/// Materialize a [`BooleanBuffer`] into its RLE form.
+pub(crate) fn mask_to_selectors(mask: &BooleanBuffer) -> Vec<RowSelector> {
+ let total_rows = mask.len();
+ if total_rows == 0 {
+ return Vec::new();
+ }
+ let mut selectors: Vec<RowSelector> = Vec::new();
+ let mut last_end = 0;
+ for (start, end) in mask.set_slices() {
+ if start > last_end {
+ selectors.push(RowSelector::skip(start - last_end));
+ }
+ selectors.push(RowSelector::select(end - start));
+ last_end = end;
+ }
+ if last_end != total_rows {
+ selectors.push(RowSelector::skip(total_rows - last_end));
+ }
+ selectors
+}
+
+/// Returns whether `mask` contains at least `min_runs` alternating set/unset runs.
+///
+/// Stops as soon as the requested number of runs is found, avoiding a full scan
+/// when callers only need to know whether a boundary has been crossed.
+pub(super) fn mask_has_at_least_runs(mask: &BooleanBuffer, min_runs: usize) -> bool {
+ if min_runs == 0 {
+ return true;
+ }
+
+ let total_rows = mask.len();
+ if total_rows == 0 {
+ return false;
+ }
+
+ let mut run_count = 0;
+ let mut last_end = 0;
+ for (start, end) in mask.set_slices() {
+ run_count += usize::from(start > last_end) + 1;
+ if run_count >= min_runs {
+ return true;
+ }
+ last_end = end;
+ }
+
+ run_count + usize::from(last_end < total_rows) >= min_runs
+}
+
+/// Bitwise AND of two mask-backed selections. Longer side's tail passes through.
+pub(super) fn intersect_masks(l: &BooleanBuffer, r: &BooleanBuffer) -> BooleanBuffer {
+ if l.len() == r.len() {
+ return l & r;
+ }
+ let common = l.len().min(r.len());
+ let head = &l.slice(0, common) & &r.slice(0, common);
+ let (longer, longer_len) = if l.len() > r.len() {
+ (l, l.len())
+ } else {
+ (r, r.len())
+ };
+ let tail = longer.slice(common, longer_len - common);
+ let mut builder = BooleanBufferBuilder::new(longer_len);
+ builder.append_buffer(&head);
+ builder.append_buffer(&tail);
+ builder.finish()
+}
+
+/// Bitwise OR of two mask-backed selections. Longer side's tail passes through.
+pub(super) fn union_masks(l: &BooleanBuffer, r: &BooleanBuffer) -> BooleanBuffer {
+ if l.len() == r.len() {
+ return l | r;
+ }
+ let common = l.len().min(r.len());
Review Comment:
track in https://github.com/apache/arrow-rs/issues/10425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5100674024
> was this PR still considered an API change? i took a brief look and noticed:
>
> * `RowSelection` no longer implements `Debug`, `Eq` and `PartialEq`
> * `RowSelection::iter` return changed from `impl Iterator<Item = &RowSelector>` to `RowSelectionIter<'_>`
>
> i think both are these are considered breaking changes
Hi,@Jefffrey ,`RowSelection` still implements `Debug`, `PartialEq`, and `Eq`; they were changed from derives to manual implementations. The `iter()` signature did change syntactically, but exposing a concrete type that preserves the previously promised `Iterator<Item = &RowSelector>` contract is source-compatible for existing callers. It does create a new concrete-type commitment going forward, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5258799414
> he leaf kernel is fast — 2.6x–4.6x against stock get_batch_with_dict + arrow::compute::filter on captured low-survival pages — but I never established a query-level benefit. My two fully-covered queries ran at 100% coverage over 65M rows each and showed no reproducible direction across two rounds. And for sizing: the paper's 3.1x on Q6 is against a 2023 C++ baseline, while modern arrow-rs's full-decode path has already absorbed most of that headroom, so the residual against today's main should be expected to be much smaller.
I guess what I was thinking was that I didn't think the parquet reader supports reading some subset of struct column (which is a group type in parquet) and applying a filter to two of its fields
So like data
```json
{
"a": "foo",
"b": "bar",
"c": "baz"
}
...
```
Then a query that is like
```sql
select struct.c where struct.a = 'f' and struct.b = 'b'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783401366
Merging up to fix CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4789494532
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4789243487)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_clickbench/async/Q1 1.01 1097.8±6.48µs ? ?/sec 1.00 1086.7±7.43µs ? ?/sec
arrow_reader_clickbench/async/Q10 1.02 7.3±0.11ms ? ?/sec 1.00 7.1±0.09ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.00 8.2±0.10ms ? ?/sec 1.00 8.2±0.09ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.01 15.2±0.08ms ? ?/sec 1.00 15.1±0.07ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.03 18.3±0.21ms ? ?/sec 1.00 17.8±0.12ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.03 16.9±0.14ms ? ?/sec 1.00 16.4±0.08ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.01 3.3±0.05ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.00 75.6±0.49ms ? ?/sec 1.11 84.1±0.37ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.00 83.6±0.78ms ? ?/sec 1.30 108.5±0.51ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.00 117.9±2.19ms ? ?/sec 1.20 141.8±1.65ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.05 265.7±2.72ms ? ?/sec 1.00 254.0±2.21ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.04 21.0±0.26ms ? ?/sec 1.00 20.1±0.14ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.04 60.6±0.99ms ? ?/sec 1.00 58.5±0.61ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.06 61.7±0.64ms ? ?/sec 1.00 58.4±0.30ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.01 19.2±0.22ms ? ?/sec 1.00 18.9±0.08ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.04 16.4±0.41ms ? ?/sec 1.00 15.7±0.21ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.01 5.7±0.04ms ? ?/sec 1.00 5.7±0.04ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.06 14.6±0.34ms ? ?/sec 1.00 13.8±0.17ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.06 26.0±0.67ms ? ?/sec 1.00 24.5±0.25ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.02 6.1±0.08ms ? ?/sec 1.00 6.0±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.01 5.2±0.05ms ? ?/sec 1.00 5.2±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.00 3.6±0.03ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q1 1.01 1070.7±5.67µs ? ?/sec 1.00 1065.3±5.27µs ? ?/sec
arrow_reader_clickbench/async_object_store/Q10 1.00 7.0±0.09ms ? ?/sec 1.01 7.1±0.10ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q11 1.00 8.1±0.07ms ? ?/sec 1.01 8.2±0.14ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q12 1.01 15.2±0.13ms ? ?/sec 1.00 15.1±0.16ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q13 1.00 17.9±0.17ms ? ?/sec 1.00 17.9±0.19ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q14 1.01 16.8±0.15ms ? ?/sec 1.00 16.7±0.17ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q19 1.00 3.1±0.03ms ? ?/sec 1.01 3.2±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q20 1.01 74.2±0.90ms ? ?/sec 1.00 73.2±0.62ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q21 1.02 83.0±0.80ms ? ?/sec 1.00 81.6±0.65ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q22 1.03 105.1±3.83ms ? ?/sec 1.00 101.6±0.77ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q23 1.03 252.9±1.98ms ? ?/sec 1.00 246.2±5.50ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q24 1.00 20.3±0.20ms ? ?/sec 1.00 20.3±0.24ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q27 1.02 59.3±0.90ms ? ?/sec 1.00 58.2±0.64ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q28 1.01 59.9±0.94ms ? ?/sec 1.00 59.3±0.43ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q30 1.00 18.8±0.15ms ? ?/sec 1.00 18.7±0.12ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q36 1.00 15.6±0.30ms ? ?/sec 1.03 16.1±0.22ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q37 1.00 5.6±0.04ms ? ?/sec 1.02 5.7±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q38 1.01 13.8±0.27ms ? ?/sec 1.00 13.6±0.10ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q39 1.02 24.9±0.74ms ? ?/sec 1.00 24.4±0.26ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q40 1.01 5.9±0.06ms ? ?/sec 1.00 5.8±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q41 1.01 5.1±0.04ms ? ?/sec 1.00 5.0±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q42 1.00 3.5±0.02ms ? ?/sec 1.00 3.5±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.01 902.6±10.22µs ? ?/sec 1.00 894.7±2.58µs ? ?/sec
arrow_reader_clickbench/sync/Q10 1.01 5.3±0.02ms ? ?/sec 1.00 5.2±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.01 6.3±0.02ms ? ?/sec 1.00 6.2±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.02 22.6±0.13ms ? ?/sec 1.00 22.1±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.22 30.5±0.27ms ? ?/sec 1.00 25.0±0.12ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.02 24.0±0.13ms ? ?/sec 1.00 23.5±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.01 2.9±0.04ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.06 130.3±1.37ms ? ?/sec 1.00 123.1±0.48ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.05 102.3±0.52ms ? ?/sec 1.00 97.7±0.24ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.04 153.3±1.19ms ? ?/sec 1.00 147.2±0.51ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.00 304.3±4.98ms ? ?/sec 1.01 307.4±9.26ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.02 28.1±0.17ms ? ?/sec 1.00 27.5±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.04 112.8±0.37ms ? ?/sec 1.00 108.3±0.31ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.05 110.7±0.37ms ? ?/sec 1.00 105.3±0.26ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.02 19.4±0.07ms ? ?/sec 1.00 18.9±0.09ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.02 23.0±0.06ms ? ?/sec 1.00 22.5±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.02 7.1±0.02ms ? ?/sec 1.00 6.9±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.02 12.0±0.05ms ? ?/sec 1.00 11.7±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.04 21.9±0.14ms ? ?/sec 1.00 21.0±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.04 5.6±0.04ms ? ?/sec 1.00 5.3±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.02 5.8±0.03ms ? ?/sec 1.00 5.7±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.01 4.4±0.02ms ? ?/sec 1.00 4.4±0.02ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 800.2s |
| Peak memory | 286.3 MiB |
| Avg memory | 137.7 MiB |
| CPU user | 707.9s |
| CPU sys | 88.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 795.2s |
| Peak memory | 286.2 MiB |
| Avg memory | 153.8 MiB |
| CPU user | 696.7s |
| CPU sys | 92.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4789243487
run benchmark arrow_reader_clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783327071
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783293361)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4783293361-636-bs95k 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (1720537307f8a4422ddd8d295b35ea0f9d8f0adc) to 2e035fd (merge-base) [diff](https://github.com/apache/arrow-rs/compare/2e035fd4d9e030473ef757e37ff9a5eae7524f40..1720537307f8a4422ddd8d295b35ea0f9d8f0adc)
BENCH_NAME=arrow_reader_row_filter
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_row_filter
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783326252
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783293361)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4783293361-635-hxxkf 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (1720537307f8a4422ddd8d295b35ea0f9d8f0adc) to 2e035fd (merge-base) [diff](https://github.com/apache/arrow-rs/compare/2e035fd4d9e030473ef757e37ff9a5eae7524f40..1720537307f8a4422ddd8d295b35ea0f9d8f0adc)
BENCH_NAME=arrow_reader
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787507764
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787465734)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
and_then 1.00 582.5±2.11µs ? ?/sec 1.00 580.9±2.23µs ? ?/sec
from_filters 1.00 370.5±2.14µs ? ?/sec 1.00 370.8±1.81µs ? ?/sec
intersection 1.00 1302.6±1.35µs ? ?/sec 1.03 1337.8±1.64µs ? ?/sec
union 1.03 1331.1±2.70µs ? ?/sec 1.00 1294.7±1.88µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 14.0 MiB |
| Avg memory | 10.4 MiB |
| CPU user | 44.7s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 13.9 MiB |
| Avg memory | 10.4 MiB |
| CPU user | 44.0s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787614815
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787465734)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_row_filter/float64 <= 99.0/all_columns/async 1.00 5.3±0.02ms ? ?/sec 1.00 5.3±0.02ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/all_columns/sync 1.00 5.3±0.08ms ? ?/sec 1.01 5.3±0.10ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/async 1.00 4.7±0.06ms ? ?/sec 1.00 4.7±0.03ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/sync 1.00 4.5±0.02ms ? ?/sec 1.00 4.5±0.02ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/async 1.00 4.2±0.01ms ? ?/sec 1.00 4.1±0.01ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/sync 1.00 4.5±0.01ms ? ?/sec 1.01 4.6±0.10ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/async 1.02 3.8±0.09ms ? ?/sec 1.00 3.8±0.08ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/sync 1.00 3.7±0.01ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/async 1.00 5.4±0.11ms ? ?/sec 1.01 5.5±0.10ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/sync 1.00 5.4±0.18ms ? ?/sec 1.00 5.4±0.16ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/async 1.00 4.6±0.02ms ? ?/sec 1.00 4.6±0.02ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/sync 1.00 4.6±0.14ms ? ?/sec 1.00 4.6±0.13ms ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/async 1.00 904.0±14.03µs ? ?/sec 1.00 907.6±18.66µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/sync 1.00 2.1±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/async 1.00 814.5±5.68µs ? ?/sec 1.00 813.6±9.14µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/sync 1.00 2.1±0.01ms ? ?/sec 1.00 2.1±0.01ms ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/async 1.00 7.7±0.02ms ? ?/sec 1.00 7.7±0.02ms ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/sync 1.01 6.8±0.02ms ? ?/sec 1.00 6.8±0.02ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/async 1.01 7.3±0.07ms ? ?/sec 1.00 7.2±0.07ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/sync 1.01 6.5±0.16ms ? ?/sec 1.00 6.5±0.17ms ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/async 1.00 5.0±0.03ms ? ?/sec 1.01 5.0±0.04ms ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/sync 1.00 5.3±0.02ms ? ?/sec 1.00 5.3±0.02ms ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/async 1.00 4.6±0.01ms ? ?/sec 1.01 4.6±0.09ms ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/sync 1.01 4.6±0.05ms ? ?/sec 1.00 4.6±0.02ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/async 1.00 3.5±0.10ms ? ?/sec 1.02 3.6±0.11ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/sync 1.00 3.6±0.01ms ? ?/sec 1.00 3.6±0.01ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/async 1.00 3.2±0.06ms ? ?/sec 1.01 3.2±0.07ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/sync 1.00 3.2±0.04ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/async 1.01 8.6±0.03ms ? ?/sec 1.00 8.6±0.02ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/sync 1.01 9.4±0.08ms ? ?/sec 1.00 9.4±0.03ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/async 1.01 7.2±0.02ms ? ?/sec 1.00 7.2±0.03ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/sync 1.02 6.5±0.09ms ? ?/sec 1.00 6.4±0.07ms ? ?/sec
arrow_reader_row_filter_limit/float64 <= 99.0/all_columns/limit10/async 1.01 347.2±5.20µs ? ?/sec 1.00 342.1±7.85µs ? ?/sec
arrow_reader_row_filter_limit/float64 <= 99.0/exclude_filter_column/limit10/async 1.02 336.5±1.09µs ? ?/sec 1.00 329.6±1.62µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0 AND ts >= 9000/all_columns/limit10/async 1.03 391.5±6.28µs ? ?/sec 1.00 380.0±3.23µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0 AND ts >= 9000/exclude_filter_column/limit10/async 1.00 364.6±0.69µs ? ?/sec 1.00 363.7±5.43µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0/all_columns/limit10/async 1.00 346.1±2.47µs ? ?/sec 1.00 345.3±8.56µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0/exclude_filter_column/limit10/async 1.01 337.5±2.20µs ? ?/sec 1.00 335.0±5.57µs ? ?/sec
arrow_reader_row_filter_limit/int64 > 90/all_columns/limit10/async 1.02 338.8±1.26µs ? ?/sec 1.00 332.9±1.29µs ? ?/sec
arrow_reader_row_filter_limit/int64 > 90/exclude_filter_column/limit10/async 1.02 336.0±5.63µs ? ?/sec 1.00 327.8±5.34µs ? ?/sec
arrow_reader_row_filter_limit/ts < 9000/all_columns/limit10/async 1.01 342.9±6.32µs ? ?/sec 1.00 338.4±6.19µs ? ?/sec
arrow_reader_row_filter_limit/ts < 9000/exclude_filter_column/limit10/async 1.01 332.5±6.11µs ? ?/sec 1.00 328.1±5.09µs ? ?/sec
arrow_reader_row_filter_limit/ts >= 9000/all_columns/limit10/async 1.01 350.5±3.12µs ? ?/sec 1.00 346.2±1.81µs ? ?/sec
arrow_reader_row_filter_limit/ts >= 9000/exclude_filter_column/limit10/async 1.02 341.9±3.63µs ? ?/sec 1.00 335.7±3.60µs ? ?/sec
arrow_reader_row_filter_limit/utf8View <> ''/all_columns/limit10/async 1.01 362.4±0.84µs ? ?/sec 1.00 360.0±1.98µs ? ?/sec
arrow_reader_row_filter_limit/utf8View <> ''/exclude_filter_column/limit10/async 1.02 351.6±2.22µs ? ?/sec 1.00 346.0±1.92µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 480.1s |
| Peak memory | 44.8 MiB |
| Avg memory | 30.6 MiB |
| CPU user | 474.1s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 475.1s |
| Peak memory | 45.6 MiB |
| Avg memory | 31.0 MiB |
| CPU user | 469.6s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5015752318
Thanks for the thorough review @hhhizzz — I verified all six findings and they are accurate. The three blocking items are fixed in [496d7cb](https://github.com/apache/arrow-rs/pull/10141/commits/496d7cb62a2aff8064ac35c2bcdc5d682e78d291); [325d774](https://github.com/apache/arrow-rs/pull/10141/commits/325d774420a49430e05399f401d7945ee93901e1) extends the `row_selection_cursor` bench with a `read_auto` mode and mask-backed inputs.
## 1. Regression — fixed
Profiling showed the regression was entirely in the selector→bitmap conversion: after the module split, LLVM stopped inlining `BooleanBufferBuilder::append_n` there (self time 3.21% → 5.43%, matching the ~2% regression); the decode loop profiles identically on both sides. `boolean_mask_from_selectors` now fills the bitmap bytes directly instead of calling `append_n` per selector, with a fuzz test asserting bit-exact equivalence.
L4–L40 matrix, 10141-auto [325d774](https://github.com/apache/arrow-rs/pull/10141/commits/325d774420a49430e05399f401d7945ee93901e1) vs main `ee30b61` (medians, Apple M2 Max):
<img width="1003" height="390" alt="image" src="https://github.com/user-attachments/assets/d4ec292f-1fde-423a-86fc-fbb8f0ce83ec" />
Both forced paths are now faster than the main at every point, and the Selectors/Mask crossover (between L12 and L16 on this machine) is unchanged by this PR.
## 2. Dense-mask trimming — fixed
`trim_mask` no longer popcounts or walks set bits: it fast-paths a set final bit, otherwise reverse-scans bytes for the last set bit. Near-dense masks are now O(1) to trim. Fuzz-tested including sliced (non-zero offset) masks.
## 3. Repeated popcounts — fixed
`MaskSelection` caches the selected count in a `OnceLock<usize>`. `split_off` popcounts only the head and derives the tail by subtraction, so the row-group frontier is O(bitmap) total instead of O(row_groups × bitmap); `trim`/`offset`/`limit` propagate the cache, and `Clone` keeps it so `peek_next_row_group`'s frontier clone answers `row_count()` for free. Unit tests cover the propagation.
## 4–6
These are valid findings as well. To keep this PR from growing too large I'd like to address them in follow-up PRs, and I'll file issues for them so they don't get lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5026886653
> i will add those test case after https://github.com/apache/arrow-rs/pull/10288 merge.
I just merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5100991012
> they were changed from derives to manual implementations
thanks i missed that
> The `iter()` signature did change syntactically, but exposing a concrete type that preserves the previously promised `Iterator<Item = &RowSelector>` contract is source-compatible for existing callers. It does create a new concrete-type commitment going forward, though.
thoughts on if we should revert to the old signature (if possible) just to be on the safe side? i think only really niche use cases would be broken but maybe its good to keep flexibility if ever we need it 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5101272326
> @haohuaijin, would you mind preparing a follow-up PR to restore the old iter() signature and make RowSelectionIter internal again?
sure, it is okay to restore the old `iter()` signature, i will make a pr later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5100475876
was this PR still considered an API change? i took a brief look and noticed:
- `RowSelection` no longer implements `Debug`, `Eq` and `PartialEq`
- `RowSelection::iter` return changed from `impl Iterator<Item = &RowSelector>` to `RowSelectionIter<'_>`
i think both are these are considered breaking changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5084180649
I think we are very close to being able to implement something like this
- https://dl.acm.org/doi/10.1145/3589323
We probably already do it for unnested (leaf) nodes -- but need a bit more cleverness for structured nodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788216344
run benchmark arrow_reader_clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787923901
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787709681)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_clickbench/async/Q1 1.00 1086.0±7.05µs ? ?/sec 1.00 1084.8±6.38µs ? ?/sec
arrow_reader_clickbench/async/Q10 1.00 6.5±0.05ms ? ?/sec 1.01 6.6±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.00 7.5±0.05ms ? ?/sec 1.01 7.5±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.02 14.5±0.09ms ? ?/sec 1.00 14.3±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.01 17.1±0.15ms ? ?/sec 1.00 16.9±0.10ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.02 16.1±0.11ms ? ?/sec 1.00 15.8±0.09ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.00 3.1±0.03ms ? ?/sec 1.01 3.1±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.11 92.7±0.85ms ? ?/sec 1.00 83.9±0.37ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.01 106.6±1.26ms ? ?/sec 1.00 105.0±7.45ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.02 141.4±7.98ms ? ?/sec 1.00 139.0±9.48ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.00 258.0±2.20ms ? ?/sec 1.00 257.3±7.13ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.02 19.6±0.16ms ? ?/sec 1.00 19.3±0.14ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.01 59.1±0.42ms ? ?/sec 1.00 58.5±0.32ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.02 59.4±0.56ms ? ?/sec 1.00 58.3±0.37ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.01 18.5±0.11ms ? ?/sec 1.00 18.4±0.09ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.01 15.8±0.32ms ? ?/sec 1.00 15.6±0.34ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.01 5.5±0.03ms ? ?/sec 1.00 5.4±0.04ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.00 13.7±0.31ms ? ?/sec 1.00 13.7±0.32ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.00 24.7±0.62ms ? ?/sec 1.01 24.9±0.53ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.00 5.7±0.05ms ? ?/sec 1.01 5.7±0.06ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.00 5.0±0.03ms ? ?/sec 1.00 5.0±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.00 3.5±0.02ms ? ?/sec 1.00 3.5±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q1 1.01 1066.9±3.66µs ? ?/sec 1.00 1056.5±3.18µs ? ?/sec
arrow_reader_clickbench/async_object_store/Q10 1.00 6.4±0.05ms ? ?/sec 1.00 6.4±0.06ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q11 1.01 7.4±0.06ms ? ?/sec 1.00 7.3±0.06ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q12 1.02 14.5±0.14ms ? ?/sec 1.00 14.2±0.07ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q13 1.02 17.1±0.16ms ? ?/sec 1.00 16.8±0.14ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q14 1.01 15.9±0.11ms ? ?/sec 1.00 15.7±0.07ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q19 1.00 3.0±0.04ms ? ?/sec 1.01 3.0±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q20 1.00 72.4±0.74ms ? ?/sec 1.00 72.6±0.54ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q21 1.01 81.9±0.65ms ? ?/sec 1.00 80.9±0.54ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q22 1.01 100.4±1.01ms ? ?/sec 1.00 99.6±0.72ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q23 1.06 244.6±0.61ms ? ?/sec 1.00 230.2±3.28ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q24 1.02 19.3±0.19ms ? ?/sec 1.00 19.0±0.14ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q27 1.02 57.9±0.77ms ? ?/sec 1.00 57.0±0.51ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q28 1.03 58.4±0.73ms ? ?/sec 1.00 56.8±0.38ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q30 1.01 18.3±0.14ms ? ?/sec 1.00 18.0±0.09ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q36 1.01 15.0±0.27ms ? ?/sec 1.00 14.9±0.21ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q37 1.02 5.4±0.02ms ? ?/sec 1.00 5.3±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q38 1.00 13.2±0.34ms ? ?/sec 1.00 13.3±0.21ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q39 1.02 24.1±0.63ms ? ?/sec 1.00 23.7±0.42ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q40 1.01 5.6±0.06ms ? ?/sec 1.00 5.5±0.08ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q41 1.00 4.8±0.04ms ? ?/sec 1.01 4.9±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q42 1.00 3.4±0.02ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.00 892.4±1.84µs ? ?/sec 1.00 889.8±2.26µs ? ?/sec
arrow_reader_clickbench/sync/Q10 1.00 5.1±0.01ms ? ?/sec 1.00 5.1±0.10ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.00 6.0±0.02ms ? ?/sec 1.01 6.1±0.11ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.00 21.6±0.08ms ? ?/sec 1.00 21.6±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.18 28.6±0.22ms ? ?/sec 1.00 24.3±0.11ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.01 23.0±0.08ms ? ?/sec 1.00 22.8±0.09ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.00 2.7±0.02ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.02 126.5±0.27ms ? ?/sec 1.00 124.1±0.31ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.03 100.3±0.17ms ? ?/sec 1.00 97.7±0.20ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.01 148.4±0.29ms ? ?/sec 1.00 147.0±0.43ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.01 309.5±8.60ms ? ?/sec 1.00 306.6±10.33ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.01 26.9±0.14ms ? ?/sec 1.00 26.6±0.09ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.02 110.7±0.44ms ? ?/sec 1.00 108.6±0.18ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.03 108.2±0.20ms ? ?/sec 1.00 105.4±0.13ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.02 18.8±0.09ms ? ?/sec 1.00 18.5±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.00 22.0±0.12ms ? ?/sec 1.00 22.1±0.11ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.00 6.8±0.01ms ? ?/sec 1.00 6.8±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.00 11.4±0.06ms ? ?/sec 1.00 11.4±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.00 20.7±0.12ms ? ?/sec 1.00 20.7±0.10ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.00 5.2±0.02ms ? ?/sec 1.00 5.2±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.00 5.6±0.02ms ? ?/sec 1.01 5.6±0.04ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.00 4.3±0.02ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 790.2s |
| Peak memory | 282.2 MiB |
| Avg memory | 126.1 MiB |
| CPU user | 700.0s |
| CPU sys | 84.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 795.2s |
| Peak memory | 286.1 MiB |
| Avg memory | 130.1 MiB |
| CPU user | 697.9s |
| CPU sys | 90.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4795067063
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4795054381)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4795054381-675-lmz82 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (ee2df31354821271252272b819cc827a9f2c74ac) to 7fb7751 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/7fb775196456bbc7892cecec2d07c655608f7bb5..ee2df31354821271252272b819cc827a9f2c74ac)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_clickbench
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4788477163
run benchmark arrow_reader_clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4729511099
Some related work
- https://github.com/apache/arrow-rs/pull/6624 from @XiangpengHao
- https://github.com/apache/arrow-rs/pull/7454 from @zhuqi-lucas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783449255
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783293361)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_row_filter/float64 <= 99.0/all_columns/async 1.02 5.4±0.03ms ? ?/sec 1.00 5.3±0.03ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/all_columns/sync 1.01 5.2±0.04ms ? ?/sec 1.00 5.2±0.06ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/async 1.01 4.6±0.04ms ? ?/sec 1.00 4.6±0.03ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/sync 1.00 4.5±0.04ms ? ?/sec 1.00 4.5±0.05ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/async 1.01 4.2±0.04ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/sync 1.01 4.5±0.04ms ? ?/sec 1.00 4.5±0.04ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/async 1.02 3.7±0.03ms ? ?/sec 1.00 3.7±0.01ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/sync 1.01 3.7±0.04ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/async 1.01 5.4±0.03ms ? ?/sec 1.00 5.3±0.03ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/sync 1.01 5.2±0.05ms ? ?/sec 1.00 5.2±0.05ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/async 1.00 4.6±0.03ms ? ?/sec 1.00 4.6±0.03ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/sync 1.00 4.5±0.03ms ? ?/sec 1.00 4.5±0.04ms ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/async 1.02 897.9±2.45µs ? ?/sec 1.00 878.6±2.32µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/sync 1.00 2.2±0.02ms ? ?/sec 1.00 2.2±0.02ms ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/async 1.03 822.5±2.61µs ? ?/sec 1.00 798.2±2.02µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/sync 1.00 2.1±0.02ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/async 1.09 8.3±0.12ms ? ?/sec 1.00 7.6±0.05ms ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/sync 1.01 6.9±0.06ms ? ?/sec 1.00 6.8±0.04ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/async 1.08 7.7±0.12ms ? ?/sec 1.00 7.2±0.05ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/sync 1.01 6.4±0.05ms ? ?/sec 1.00 6.3±0.04ms ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/async 1.01 5.0±0.04ms ? ?/sec 1.00 5.0±0.04ms ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/sync 1.00 5.3±0.04ms ? ?/sec 1.00 5.3±0.04ms ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/async 1.00 4.5±0.04ms ? ?/sec 1.01 4.6±0.03ms ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/sync 1.01 4.5±0.04ms ? ?/sec 1.00 4.5±0.04ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/async 1.00 3.4±0.03ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/sync 1.00 3.6±0.03ms ? ?/sec 1.01 3.6±0.02ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/async 1.00 3.1±0.02ms ? ?/sec 1.02 3.1±0.03ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/sync 1.00 3.2±0.03ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/async 1.06 9.1±0.08ms ? ?/sec 1.00 8.6±0.06ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/sync 1.01 9.4±0.06ms ? ?/sec 1.00 9.3±0.06ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/async 1.06 7.6±0.07ms ? ?/sec 1.00 7.2±0.04ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/sync 1.00 6.4±0.03ms ? ?/sec 1.00 6.4±0.04ms ? ?/sec
arrow_reader_row_filter_limit/float64 <= 99.0/all_columns/limit10/async 1.03 347.8±1.64µs ? ?/sec 1.00 336.2±0.81µs ? ?/sec
arrow_reader_row_filter_limit/float64 <= 99.0/exclude_filter_column/limit10/async 1.05 344.0±1.71µs ? ?/sec 1.00 326.9±0.57µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0 AND ts >= 9000/all_columns/limit10/async 1.03 384.5±1.68µs ? ?/sec 1.00 374.1±1.44µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0 AND ts >= 9000/exclude_filter_column/limit10/async 1.04 371.2±2.32µs ? ?/sec 1.00 356.9±1.02µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0/all_columns/limit10/async 1.03 348.1±1.56µs ? ?/sec 1.00 337.2±1.29µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0/exclude_filter_column/limit10/async 1.05 345.4±1.73µs ? ?/sec 1.00 327.9±0.97µs ? ?/sec
arrow_reader_row_filter_limit/int64 > 90/all_columns/limit10/async 1.03 341.7±2.09µs ? ?/sec 1.00 330.5±1.30µs ? ?/sec
arrow_reader_row_filter_limit/int64 > 90/exclude_filter_column/limit10/async 1.04 336.2±1.81µs ? ?/sec 1.00 324.6±1.56µs ? ?/sec
arrow_reader_row_filter_limit/ts < 9000/all_columns/limit10/async 1.04 338.6±1.61µs ? ?/sec 1.00 326.5±1.17µs ? ?/sec
arrow_reader_row_filter_limit/ts < 9000/exclude_filter_column/limit10/async 1.05 335.8±1.76µs ? ?/sec 1.00 321.1±1.14µs ? ?/sec
arrow_reader_row_filter_limit/ts >= 9000/all_columns/limit10/async 1.03 354.8±1.67µs ? ?/sec 1.00 344.6±1.16µs ? ?/sec
arrow_reader_row_filter_limit/ts >= 9000/exclude_filter_column/limit10/async 1.05 349.7±1.31µs ? ?/sec 1.00 334.1±0.97µs ? ?/sec
arrow_reader_row_filter_limit/utf8View <> ''/all_columns/limit10/async 1.06 373.9±1.52µs ? ?/sec 1.00 353.8±0.99µs ? ?/sec
arrow_reader_row_filter_limit/utf8View <> ''/exclude_filter_column/limit10/async 1.03 353.8±1.27µs ? ?/sec 1.00 343.5±0.93µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 480.1s |
| Peak memory | 45.0 MiB |
| Avg memory | 30.9 MiB |
| CPU user | 473.1s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 470.1s |
| Peak memory | 44.8 MiB |
| Avg memory | 31.1 MiB |
| CPU user | 467.5s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783482356
run benchmark row_selector row_selection_cursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783531978
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783522864)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4783522864-641-ld6wl 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (7b102f41c4e92c03788a991dfd7d59be4fd67b21) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..7b102f41c4e92c03788a991dfd7d59be4fd67b21)
BENCH_NAME=arrow_reader_row_filter
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_row_filter
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5018312161
#10288 is about to land and changes Mask execution across page-pruned data. When rebasing this PR, could we preserve its `LoadedRowRanges`-aware Mask path and thread `loaded_row_ranges` into both cursor constructors, especially:
```rust
new_mask_from_buffer(mask, loaded_row_ranges)
```
#10288 only has `new_mask_from_selectors`, while this PR introduces the direct `BooleanBuffer -> MaskCursor` path. If the ranges are not propagated there, `RowSelection::from_boolean_buffer(...)` can still cross an unloaded page and reproduce the sparse-page offset failure.
Could we also add a sparse-page regression test starting from a direct, preferably non-byte-aligned, `BooleanBuffer::slice(...)`? The existing #10288 tests all start from `Vec<RowSelector>`, so they do not cover this integration seam.
The old `should_force_selectors` / `override_selector_strategy_if_needed` fallback should be removed after the rebase, since #10288 makes Auto remain Mask safely across skipped pages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4795181140
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4795054381)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_clickbench/async/Q1 1.01 1098.7±5.61µs ? ?/sec 1.00 1085.6±4.15µs ? ?/sec
arrow_reader_clickbench/async/Q10 1.02 6.5±0.05ms ? ?/sec 1.00 6.4±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.01 7.6±0.12ms ? ?/sec 1.00 7.5±0.07ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.01 14.3±0.08ms ? ?/sec 1.00 14.1±0.04ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.01 17.0±0.14ms ? ?/sec 1.00 16.7±0.10ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.01 15.8±0.10ms ? ?/sec 1.00 15.6±0.04ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.01 3.1±0.04ms ? ?/sec 1.00 3.0±0.02ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.29 92.8±1.15ms ? ?/sec 1.00 72.2±0.33ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.10 107.7±1.24ms ? ?/sec 1.00 98.1±7.30ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.11 137.2±8.76ms ? ?/sec 1.00 123.5±0.66ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.03 254.6±2.18ms ? ?/sec 1.00 248.0±1.25ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.02 19.4±0.17ms ? ?/sec 1.00 18.9±0.10ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.02 58.6±0.56ms ? ?/sec 1.00 57.4±0.26ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.03 59.1±0.52ms ? ?/sec 1.00 57.4±0.34ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.02 18.5±0.10ms ? ?/sec 1.00 18.2±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.00 15.3±0.24ms ? ?/sec 1.01 15.4±0.30ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.00 5.4±0.04ms ? ?/sec 1.01 5.5±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.01 13.6±0.27ms ? ?/sec 1.00 13.4±0.26ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.01 25.1±0.49ms ? ?/sec 1.00 24.9±0.44ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.02 5.8±0.06ms ? ?/sec 1.00 5.6±0.04ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.01 5.0±0.03ms ? ?/sec 1.00 5.0±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.00 3.5±0.02ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q1 1.00 1068.1±6.19µs ? ?/sec 1.01 1078.6±5.12µs ? ?/sec
arrow_reader_clickbench/async_object_store/Q10 1.00 6.3±0.05ms ? ?/sec 1.00 6.4±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q11 1.00 7.3±0.04ms ? ?/sec 1.00 7.3±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q12 1.01 14.3±0.10ms ? ?/sec 1.00 14.1±0.08ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q13 1.01 16.8±0.14ms ? ?/sec 1.00 16.6±0.11ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q14 1.00 15.6±0.07ms ? ?/sec 1.00 15.6±0.07ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q19 1.02 3.0±0.03ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q20 1.01 72.1±0.72ms ? ?/sec 1.00 71.6±0.47ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q21 1.01 81.2±0.72ms ? ?/sec 1.00 80.1±0.42ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q22 1.04 100.4±1.84ms ? ?/sec 1.00 96.8±0.44ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q23 1.03 243.4±19.11ms ? ?/sec 1.00 236.4±0.44ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q24 1.01 19.0±0.13ms ? ?/sec 1.00 18.8±0.09ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q27 1.02 58.0±0.65ms ? ?/sec 1.00 57.0±0.37ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q28 1.02 57.6±0.72ms ? ?/sec 1.00 56.5±0.40ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q30 1.02 18.3±0.45ms ? ?/sec 1.00 17.9±0.08ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q36 1.02 15.0±0.38ms ? ?/sec 1.00 14.6±0.24ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q37 1.00 5.4±0.02ms ? ?/sec 1.00 5.4±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q38 1.03 13.1±0.25ms ? ?/sec 1.00 12.8±0.19ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q39 1.04 24.3±0.79ms ? ?/sec 1.00 23.4±0.46ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q40 1.01 5.5±0.04ms ? ?/sec 1.00 5.5±0.06ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q41 1.00 4.8±0.03ms ? ?/sec 1.00 4.8±0.05ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q42 1.00 3.4±0.02ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.00 888.8±1.49µs ? ?/sec 1.01 897.7±4.51µs ? ?/sec
arrow_reader_clickbench/sync/Q10 1.00 5.1±0.01ms ? ?/sec 1.00 5.1±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.00 6.0±0.02ms ? ?/sec 1.01 6.1±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.00 21.5±0.06ms ? ?/sec 1.00 21.6±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.17 28.3±0.18ms ? ?/sec 1.00 24.2±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.01 23.0±0.12ms ? ?/sec 1.00 22.7±0.09ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.03 2.7±0.03ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.03 126.6±0.27ms ? ?/sec 1.00 122.3±0.28ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.04 100.5±0.54ms ? ?/sec 1.00 96.8±0.17ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.05 148.9±0.41ms ? ?/sec 1.00 142.0±0.29ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.03 308.9±8.43ms ? ?/sec 1.00 298.8±10.61ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.03 26.8±0.13ms ? ?/sec 1.00 26.1±0.10ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.03 110.4±0.34ms ? ?/sec 1.00 107.2±0.21ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.04 108.4±0.27ms ? ?/sec 1.00 104.3±0.25ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.03 18.6±0.10ms ? ?/sec 1.00 18.2±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.01 22.0±0.09ms ? ?/sec 1.00 21.7±0.09ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.00 6.8±0.01ms ? ?/sec 1.00 6.7±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.01 11.3±0.05ms ? ?/sec 1.00 11.2±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.01 20.6±0.12ms ? ?/sec 1.00 20.4±0.16ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.01 5.2±0.02ms ? ?/sec 1.00 5.1±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.00 5.6±0.02ms ? ?/sec 1.00 5.6±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.00 4.2±0.02ms ? ?/sec 1.00 4.2±0.03ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 785.2s |
| Peak memory | 286.2 MiB |
| Avg memory | 154.2 MiB |
| CPU user | 709.4s |
| CPU sys | 73.5s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 790.2s |
| Peak memory | 286.3 MiB |
| Avg memory | 132.2 MiB |
| CPU user | 696.4s |
| CPU sys | 89.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4810205790
I had similar issues with `arrow_reader_clickbench` #9659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5017907694
I reran the forced-Selectors benchmark on an AMD EPYC 9254 using the exact same historical 2-mode benchmark source for both `cc83f7e` and `6c98404` (5 alternating pairs, CPU pinned). The **regression** still reproduces: L32 is `+2.49%` and L40 is `+5.58%`.
I then split L40 into selection clone, reader creation, read-plan build, and reader drain. Clone (`-3.06%`), `try_new` (`-2.99%`), and build (`-0.70%`) do not regress; the difference is entirely in drain, at roughly `+5%` after excluding one noisy round.
However, **this does not appear to be additional work introduced by the selector cursor.** The selector hot path is unchanged, and the normalized `ParquetRecordBatchReader::next` machine-code stream is identical on both revisions (732 instructions). The result is also highly codegen-sensitive: a debug/frame-pointer build reverses the result, and with `codegen-units=1` the stable gap disappears. Forcing function alignment changes it again. I also tried adding `#[inline]` to the affected `read_records` helper; that restored the base-like symbol layout but made performance another `2.25%` worse.
My conclusion is that the **residual forced-Selectors difference is a default multi-CGU inlining/code-layout artifact** rather than a cursor algorithm regression. I would recommend **not** change `SelectorsCursor`, the L4–L40 sweep, or the Auto threshold based on this result. The mask-backed Auto decision cost from fully enumerating `mask_run_count`, on the other hand, remains independently reproducible within the same binary and is still worth optimizing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "hhhizzz (via GitHub)" <gi...@apache.org>.
hhhizzz commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5101053950
Yes, I agree that restoring the opaque return type would preserve the original abstraction boundary and give us more flexibility in the future.
@haohuaijin, would you mind preparing a follow-up PR to restore the old `iter()` signature and make `RowSelectionIter` internal again?
cc @alamb for thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5076460926
Thanks again for the reviews and suggestions @hhhizzz @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on code in PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#discussion_r3465730044
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -135,12 +143,439 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(BooleanBuffer),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a == b,
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Iterator over the [`RowSelector`]s of a [`RowSelection`].
+///
+/// Yields owned [`RowSelector`] values; mask-backed selections stream the
+/// run-length form via [`BitSliceIterator`] without allocation.
+#[derive(Debug)]
+pub enum RowSelectionIter<'a> {
+ Selectors(std::iter::Copied<std::slice::Iter<'a, RowSelector>>),
+ Mask(MaskRunIter<'a>),
+}
+
+impl Iterator for RowSelectionIter<'_> {
+ type Item = RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<RowSelector> {
+ match self {
Review Comment:
updated in [fff3ab3](https://github.com/apache/arrow-rs/pull/10141/commits/fff3ab3278d378b3bdd77b421ea52ec7b74b7a13)
##########
parquet/src/arrow/arrow_reader/selection.rs:
##########
@@ -135,12 +143,439 @@ impl RowSelector {
/// * Consecutive [`RowSelector`]s alternate skipping or selecting rows
///
/// [`PageIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-#[derive(Debug, Clone, Default, Eq, PartialEq)]
+#[derive(Default, Clone)]
pub struct RowSelection {
- selectors: Vec<RowSelector>,
+ inner: RowSelectionInner,
+}
+
+/// Internal storage for [`RowSelection`].
+#[derive(Debug, Clone)]
+pub(crate) enum RowSelectionInner {
+ Selectors(Vec<RowSelector>),
+ Mask(BooleanBuffer),
+}
+
+impl Default for RowSelectionInner {
+ fn default() -> Self {
+ Self::Selectors(Vec::new())
+ }
+}
+
+impl std::fmt::Debug for RowSelection {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match &self.inner {
+ RowSelectionInner::Selectors(s) => f
+ .debug_struct("RowSelection")
+ .field("selectors", s)
+ .finish(),
+ RowSelectionInner::Mask(m) => f
+ .debug_struct("RowSelection")
+ .field("mask_len", &m.len())
+ .finish_non_exhaustive(),
+ }
+ }
+}
+
+impl PartialEq for RowSelection {
+ fn eq(&self, other: &Self) -> bool {
+ match (&self.inner, &other.inner) {
+ (RowSelectionInner::Selectors(a), RowSelectionInner::Selectors(b)) => a == b,
+ (RowSelectionInner::Mask(a), RowSelectionInner::Mask(b)) => a == b,
+ (RowSelectionInner::Mask(mask), RowSelectionInner::Selectors(selectors))
+ | (RowSelectionInner::Selectors(selectors), RowSelectionInner::Mask(mask)) => {
+ if selectors
+ .iter()
+ .try_fold(0usize, |acc, selector| acc.checked_add(selector.row_count))
+ != Some(mask.len())
+ {
+ return false;
+ }
+
+ let mut slices = mask.set_slices().peekable();
+ let mut cursor = 0usize;
+
+ for selector in selectors {
+ let end = cursor + selector.row_count;
+
+ if selector.skip {
+ if slices.peek().is_some_and(|(start, _)| *start < end) {
+ return false;
+ }
+ } else {
+ match slices.next() {
+ Some((start, slice_end)) if start == cursor && slice_end == end => {}
+ _ => return false,
+ }
+ }
+
+ cursor = end;
+ }
+
+ slices.next().is_none()
+ }
+ }
+ }
+}
+
+impl Eq for RowSelection {}
+
+/// Iterator over the [`RowSelector`]s of a [`RowSelection`].
+///
+/// Yields owned [`RowSelector`] values; mask-backed selections stream the
+/// run-length form via [`BitSliceIterator`] without allocation.
+#[derive(Debug)]
+pub enum RowSelectionIter<'a> {
+ Selectors(std::iter::Copied<std::slice::Iter<'a, RowSelector>>),
+ Mask(MaskRunIter<'a>),
+}
+
+impl Iterator for RowSelectionIter<'_> {
+ type Item = RowSelector;
+
+ #[inline]
+ fn next(&mut self) -> Option<RowSelector> {
+ match self {
+ Self::Selectors(it) => it.next(),
+ Self::Mask(it) => it.next(),
+ }
+ }
+}
+
+/// Streaming RLE view of a [`BooleanBuffer`].
+#[derive(Debug)]
+pub struct MaskRunIter<'a> {
+ slices: BitSliceIterator<'a>,
+ cursor: usize,
+ total: usize,
+ pending: Option<RowSelector>,
+ finished: bool,
+}
+
+impl<'a> MaskRunIter<'a> {
+ fn new(mask: &'a BooleanBuffer) -> Self {
+ Self {
+ slices: mask.set_slices(),
+ cursor: 0,
+ total: mask.len(),
+ pending: None,
+ finished: false,
+ }
+ }
+}
+
+impl Iterator for MaskRunIter<'_> {
+ type Item = RowSelector;
+
+ fn next(&mut self) -> Option<RowSelector> {
+ if let Some(p) = self.pending.take() {
+ return Some(p);
+ }
+ if self.finished {
+ return None;
+ }
+ match self.slices.next() {
Review Comment:
added in [fff3ab3](https://github.com/apache/arrow-rs/pull/10141/commits/fff3ab3278d378b3bdd77b421ea52ec7b74b7a13)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787493291
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4787465734)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4787465734-648-84cnc 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (0ffbf26319f13b931c40678b0480c6a3ea9cbcaa) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..0ffbf26319f13b931c40678b0480c6a3ea9cbcaa)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_clickbench
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783521780
🤔 looks like arrow_reader_row_filter is slightly slower. Rerunning to try an reproduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783617324
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783400488)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_clickbench/async/Q1 1.00 1084.9±5.43µs ? ?/sec 1.00 1084.8±7.61µs ? ?/sec
arrow_reader_clickbench/async/Q10 1.00 6.6±0.21ms ? ?/sec 1.01 6.6±0.26ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.02 7.6±0.26ms ? ?/sec 1.00 7.4±0.18ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.00 14.3±0.21ms ? ?/sec 1.01 14.5±0.28ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.02 17.3±0.50ms ? ?/sec 1.00 17.1±0.32ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.04 16.4±0.40ms ? ?/sec 1.00 15.7±0.27ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.02 3.2±0.09ms ? ?/sec 1.00 3.1±0.08ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.09 91.8±1.07ms ? ?/sec 1.00 84.3±0.96ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.02 105.2±1.79ms ? ?/sec 1.00 103.1±7.55ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.00 140.1±2.65ms ? ?/sec 1.00 139.5±1.63ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.00 255.0±3.90ms ? ?/sec 1.01 256.5±5.04ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.00 19.3±0.29ms ? ?/sec 1.01 19.5±0.44ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.01 58.9±0.60ms ? ?/sec 1.00 58.2±0.73ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.01 59.6±1.10ms ? ?/sec 1.00 58.9±0.77ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.01 18.7±0.20ms ? ?/sec 1.00 18.5±0.17ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.00 15.5±0.50ms ? ?/sec 1.03 16.0±0.52ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.03 5.6±0.08ms ? ?/sec 1.00 5.5±0.11ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.00 13.7±0.38ms ? ?/sec 1.00 13.6±0.44ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.02 25.0±0.84ms ? ?/sec 1.00 24.5±0.67ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.11 6.3±0.19ms ? ?/sec 1.00 5.7±0.10ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.02 5.2±0.10ms ? ?/sec 1.00 5.1±0.11ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.00 3.5±0.04ms ? ?/sec 1.00 3.5±0.07ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q1 1.00 1059.3±5.36µs ? ?/sec 1.00 1054.9±5.42µs ? ?/sec
arrow_reader_clickbench/async_object_store/Q10 1.00 6.4±0.18ms ? ?/sec 1.00 6.4±0.18ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q11 1.00 7.3±0.19ms ? ?/sec 1.00 7.4±0.25ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q12 1.04 14.8±0.38ms ? ?/sec 1.00 14.2±0.22ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q13 1.02 17.2±0.36ms ? ?/sec 1.00 16.8±0.35ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q14 1.00 15.8±0.28ms ? ?/sec 1.03 16.3±0.38ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q19 1.00 3.0±0.05ms ? ?/sec 1.00 3.0±0.04ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q20 1.00 72.4±0.71ms ? ?/sec 1.00 72.3±0.87ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q21 1.01 81.6±0.84ms ? ?/sec 1.00 81.2±0.64ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q22 1.02 100.9±1.44ms ? ?/sec 1.00 99.0±0.99ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q23 1.00 221.2±2.22ms ? ?/sec 1.11 244.4±4.21ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q24 1.00 19.0±0.29ms ? ?/sec 1.00 19.1±0.36ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q27 1.00 57.7±1.04ms ? ?/sec 1.00 57.6±0.62ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q28 1.02 59.4±1.16ms ? ?/sec 1.00 58.3±0.86ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q30 1.00 18.3±0.27ms ? ?/sec 1.01 18.5±0.27ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q36 1.00 15.0±0.56ms ? ?/sec 1.01 15.1±0.49ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q37 1.02 5.5±0.08ms ? ?/sec 1.00 5.3±0.07ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q38 1.01 13.0±0.40ms ? ?/sec 1.00 12.9±0.38ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q39 1.05 24.4±0.88ms ? ?/sec 1.00 23.3±0.52ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q40 1.08 5.9±0.20ms ? ?/sec 1.00 5.5±0.17ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q41 1.02 5.0±0.11ms ? ?/sec 1.00 4.9±0.12ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q42 1.02 3.5±0.05ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.00 888.5±2.09µs ? ?/sec 1.00 889.0±1.51µs ? ?/sec
arrow_reader_clickbench/sync/Q10 1.00 5.1±0.06ms ? ?/sec 1.00 5.1±0.06ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.01 6.1±0.13ms ? ?/sec 1.00 6.0±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.02 22.0±0.28ms ? ?/sec 1.00 21.5±0.19ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.16 28.6±0.53ms ? ?/sec 1.00 24.6±0.45ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.00 22.9±0.30ms ? ?/sec 1.01 23.2±0.30ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.02 2.7±0.06ms ? ?/sec 1.00 2.7±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.01 125.4±1.07ms ? ?/sec 1.00 123.7±0.92ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.04 101.5±1.27ms ? ?/sec 1.00 98.0±0.56ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.00 148.0±1.81ms ? ?/sec 1.00 147.7±2.06ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.01 307.7±9.93ms ? ?/sec 1.00 306.2±9.61ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.01 27.1±0.40ms ? ?/sec 1.00 26.8±0.54ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.00 109.8±0.95ms ? ?/sec 1.00 110.3±1.21ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.00 107.7±1.03ms ? ?/sec 1.00 107.2±1.06ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.00 18.6±0.23ms ? ?/sec 1.00 18.5±0.27ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.04 22.7±0.38ms ? ?/sec 1.00 21.8±0.24ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.02 6.9±0.06ms ? ?/sec 1.00 6.8±0.09ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.00 11.4±0.20ms ? ?/sec 1.00 11.4±0.21ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.01 20.6±0.38ms ? ?/sec 1.00 20.3±0.30ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.05 5.4±0.09ms ? ?/sec 1.00 5.1±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.05 5.9±0.11ms ? ?/sec 1.00 5.6±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.03 4.3±0.07ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 795.2s |
| Peak memory | 286.4 MiB |
| Avg memory | 132.9 MiB |
| CPU user | 704.6s |
| CPU sys | 86.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 790.2s |
| Peak memory | 286.3 MiB |
| Avg memory | 137.2 MiB |
| CPU user | 700.3s |
| CPU sys | 88.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783640912
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783522864)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group export-mask main
----- ----------- ----
arrow_reader_row_filter/float64 <= 99.0/all_columns/async 1.01 5.4±0.02ms ? ?/sec 1.00 5.4±0.01ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/all_columns/sync 1.01 5.4±0.03ms ? ?/sec 1.00 5.4±0.02ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/async 1.01 4.8±0.01ms ? ?/sec 1.00 4.7±0.01ms ? ?/sec
arrow_reader_row_filter/float64 <= 99.0/exclude_filter_column/sync 1.00 4.7±0.02ms ? ?/sec 1.00 4.6±0.02ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/async 1.01 4.3±0.01ms ? ?/sec 1.00 4.3±0.01ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/all_columns/sync 1.01 4.7±0.02ms ? ?/sec 1.00 4.6±0.03ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/async 1.02 3.8±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0 AND ts >= 9000/exclude_filter_column/sync 1.01 3.8±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/async 1.01 5.5±0.02ms ? ?/sec 1.00 5.4±0.01ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/all_columns/sync 1.01 5.5±0.02ms ? ?/sec 1.00 5.4±0.03ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/async 1.01 4.8±0.02ms ? ?/sec 1.00 4.7±0.01ms ? ?/sec
arrow_reader_row_filter/float64 > 99.0/exclude_filter_column/sync 1.01 4.7±0.02ms ? ?/sec 1.00 4.6±0.01ms ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/async 1.01 904.3±2.94µs ? ?/sec 1.00 895.5±3.19µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/all_columns/sync 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/async 1.00 814.5±2.11µs ? ?/sec 1.00 814.7±2.01µs ? ?/sec
arrow_reader_row_filter/int64 == 9999/exclude_filter_column/sync 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/async 1.07 8.4±0.11ms ? ?/sec 1.00 7.8±0.03ms ? ?/sec
arrow_reader_row_filter/int64 > 90/all_columns/sync 1.00 7.0±0.02ms ? ?/sec 1.01 7.1±0.17ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/async 1.06 7.8±0.09ms ? ?/sec 1.00 7.4±0.03ms ? ?/sec
arrow_reader_row_filter/int64 > 90/exclude_filter_column/sync 1.00 6.5±0.01ms ? ?/sec 1.01 6.6±0.16ms ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/async 1.00 5.1±0.01ms ? ?/sec 1.00 5.1±0.02ms ? ?/sec
arrow_reader_row_filter/ts < 9000/all_columns/sync 1.00 5.5±0.02ms ? ?/sec 1.00 5.5±0.02ms ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/async 1.00 4.7±0.01ms ? ?/sec 1.00 4.7±0.02ms ? ?/sec
arrow_reader_row_filter/ts < 9000/exclude_filter_column/sync 1.01 4.7±0.02ms ? ?/sec 1.00 4.7±0.02ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/async 1.00 3.5±0.01ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/all_columns/sync 1.00 3.8±0.02ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/async 1.00 3.3±0.01ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
arrow_reader_row_filter/ts >= 9000/exclude_filter_column/sync 1.00 3.3±0.01ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/async 1.05 9.2±0.09ms ? ?/sec 1.00 8.7±0.03ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/all_columns/sync 1.00 9.5±0.02ms ? ?/sec 1.00 9.6±0.13ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/async 1.06 7.7±0.07ms ? ?/sec 1.00 7.2±0.02ms ? ?/sec
arrow_reader_row_filter/utf8View <> ''/exclude_filter_column/sync 1.01 6.6±0.03ms ? ?/sec 1.00 6.5±0.13ms ? ?/sec
arrow_reader_row_filter_limit/float64 <= 99.0/all_columns/limit10/async 1.01 342.2±1.54µs ? ?/sec 1.00 338.4±1.63µs ? ?/sec
arrow_reader_row_filter_limit/float64 <= 99.0/exclude_filter_column/limit10/async 1.01 333.0±1.64µs ? ?/sec 1.00 330.4±1.12µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0 AND ts >= 9000/all_columns/limit10/async 1.00 382.0±1.65µs ? ?/sec 1.00 380.3±2.06µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0 AND ts >= 9000/exclude_filter_column/limit10/async 1.02 363.8±1.45µs ? ?/sec 1.00 357.4±1.46µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0/all_columns/limit10/async 1.01 342.1±1.63µs ? ?/sec 1.00 339.0±1.34µs ? ?/sec
arrow_reader_row_filter_limit/float64 > 99.0/exclude_filter_column/limit10/async 1.01 333.4±1.27µs ? ?/sec 1.00 330.9±1.19µs ? ?/sec
arrow_reader_row_filter_limit/int64 > 90/all_columns/limit10/async 1.00 335.7±1.45µs ? ?/sec 1.00 336.3±1.55µs ? ?/sec
arrow_reader_row_filter_limit/int64 > 90/exclude_filter_column/limit10/async 1.00 328.9±1.78µs ? ?/sec 1.00 330.3±1.58µs ? ?/sec
arrow_reader_row_filter_limit/ts < 9000/all_columns/limit10/async 1.00 331.9±1.54µs ? ?/sec 1.00 332.2±1.13µs ? ?/sec
arrow_reader_row_filter_limit/ts < 9000/exclude_filter_column/limit10/async 1.00 326.2±0.86µs ? ?/sec 1.01 327.9±1.33µs ? ?/sec
arrow_reader_row_filter_limit/ts >= 9000/all_columns/limit10/async 1.00 348.2±1.28µs ? ?/sec 1.00 348.3±1.15µs ? ?/sec
arrow_reader_row_filter_limit/ts >= 9000/exclude_filter_column/limit10/async 1.00 341.1±1.48µs ? ?/sec 1.00 340.8±1.25µs ? ?/sec
arrow_reader_row_filter_limit/utf8View <> ''/all_columns/limit10/async 1.01 360.9±1.13µs ? ?/sec 1.00 357.6±1.51µs ? ?/sec
arrow_reader_row_filter_limit/utf8View <> ''/exclude_filter_column/limit10/async 1.02 352.5±1.31µs ? ?/sec 1.00 346.6±1.69µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 480.1s |
| Peak memory | 45.5 MiB |
| Avg memory | 30.4 MiB |
| CPU user | 474.1s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 480.1s |
| Peak memory | 45.5 MiB |
| Avg memory | 30.5 MiB |
| CPU user | 475.6s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4784260759
Benchmark for [this request](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783482356) hit the 7200s job deadline before finishing.
Benchmarks requested: `row_selection_cursor`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783507058
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/10141#issuecomment-4783482356)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4783482356-639-b2k9m 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing export-mask (7b102f41c4e92c03788a991dfd7d59be4fd67b21) to 6ba533d (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6ba533d09ad6131f5289a981a8286ce42f91c780..7b102f41c4e92c03788a991dfd7d59be4fd67b21)
BENCH_NAME=row_selector
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench row_selector
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(parquet): `RowSelection` can be backed by a `BooleanBuffer` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10141:
URL: https://github.com/apache/arrow-rs/pull/10141#issuecomment-5107952722
Thanks for the catch @Jefffrey @haohuaijin and @hhhizzz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org