You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/13 21:36:23 UTC
(datafusion) branch gh-readonly-queue/main/pr-25494-2423b91850f995ebef8de4bea569cfd8aa3eabac created (now 2cbdc4f914)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-25494-2423b91850f995ebef8de4bea569cfd8aa3eabac
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 2cbdc4f914 perf: copy bitmaps a word at a time in `pre_selection_scatter` (#25494)
This branch includes the following new commits:
new 2cbdc4f914 perf: copy bitmaps a word at a time in `pre_selection_scatter` (#25494)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: perf: copy bitmaps a word at a time in `pre_selection_scatter` (#25494)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-25494-2423b91850f995ebef8de4bea569cfd8aa3eabac
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 2cbdc4f914eaae76285d2bd15d7685f254704e8b
Author: Yousef Mahmoud <yo...@gmail.com>
AuthorDate: Tue Oct 13 21:06:02 2026 +0000
perf: copy bitmaps a word at a time in `pre_selection_scatter` (#25494)
## Which issue does this PR close?
- N/A
## Rationale for this change
`pre_selection_scatter` puts the short-circuited RHS result back at full
batch length. For each run of selected rows it slices the RHS and
appends one `Option<bool>` at a time, even though both sides are
bit-packed, and every `slice` clones and drops an `Arc`.
Since #25771, `AND` chains of cheap comparisons no longer reach this
function, so TPC-H is unchanged. It still runs for `OR`, and for `AND`
with a conjunct that is not cheap.
TPC-H SF30 `lineitem`, 24 threads, `l_quantity > K or l_discount >
0.05`, by share of rows that reach the RHS:
- 4%: 86.0ms -> 82.0ms
- 8%: 96.5ms -> 89.5ms
- 12%: 107.5ms -> 96.0ms
- 16%: 115.0ms -> 102.0ms
- 20%: 102.0ms -> 92.0ms
`l_quantity < K and l_comment like '%special%'` is 1-2% faster. The 22
TPC-H queries are within noise.
## What changes are included in this PR?
Borrow the RHS buffers once, copy each run with `append_packed_range`,
and only build a validity bitmap when the RHS has nulls.
## Are these changes tested?
Yes, by the existing tests, including `test_pre_selection_scatter`.
## Are there any user-facing changes?
No.
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
---
datafusion/physical-expr/src/expressions/binary.rs | 48 +++++++++++++++++++---
1 file changed, 42 insertions(+), 6 deletions(-)
diff --git a/datafusion/physical-expr/src/expressions/binary.rs b/datafusion/physical-expr/src/expressions/binary.rs
index 44df100ad8..053850b836 100644
--- a/datafusion/physical-expr/src/expressions/binary.rs
+++ b/datafusion/physical-expr/src/expressions/binary.rs
@@ -27,6 +27,7 @@ use std::hash::Hash;
use std::sync::Arc;
use arrow::array::*;
+use arrow::buffer::NullBuffer;
use arrow::compute::kernels::boolean::{and_kleene, or_kleene};
use arrow::compute::kernels::concat_elements::concat_elements_dyn;
use arrow::compute::{SlicesIterator, cast, filter, filter_record_batch};
@@ -1360,21 +1361,56 @@ fn pre_selection_scatter(
let mut last_end = 0;
match right_result {
Some(right_result) => {
+ // Borrow the RHS buffers once. `BooleanArray::slice` clones and
+ // drops an `Arc` for every run, which dominates this loop when the
+ // mask selects many short runs.
+ let right_values = right_result.values();
+ let right_offset = right_values.offset();
+ let right_bytes = right_values.values();
+ let right_nulls = right_result.nulls();
+
+ let mut values = BooleanBufferBuilder::new(result_len);
+ // Only build a validity bitmap when the RHS has nulls. Otherwise
+ // every run pays a second pass to write an all-ones mask that is
+ // then discarded.
+ let mut validity = right_nulls.map(|_| BooleanBufferBuilder::new(result_len));
+
SlicesIterator::new(mask).for_each(|(start, end)| {
if start > last_end {
- result_array_builder.append_n(start - last_end, fill_value);
+ let gap = start - last_end;
+ values.append_n(gap, fill_value);
+ if let Some(v) = validity.as_mut() {
+ v.append_n(gap, true);
+ }
}
- // Copy the selected RHS slice.
+ // Both sides are bit-packed, so copy the run a word at a
+ // time. Iterating the RHS would yield one `Option<bool>` and
+ // set one bit per row.
let len = end - start;
- right_result
- .slice(right_array_pos, len)
- .iter()
- .for_each(|v| result_array_builder.append_option(v));
+ let from = right_offset + right_array_pos;
+ values.append_packed_range(from..from + len, right_bytes);
+ if let (Some(v), Some(nulls)) = (validity.as_mut(), right_nulls) {
+ let nb = nulls.inner();
+ let nfrom = nb.offset() + right_array_pos;
+ v.append_packed_range(nfrom..nfrom + len, nb.values());
+ }
right_array_pos += len;
last_end = end;
});
+
+ if last_end < result_len {
+ let gap = result_len - last_end;
+ values.append_n(gap, fill_value);
+ if let Some(v) = validity.as_mut() {
+ v.append_n(gap, true);
+ }
+ }
+
+ let nulls = validity.map(|mut v| NullBuffer::new(v.finish()));
+ let boolean_result = BooleanArray::new(values.finish(), nulls);
+ return Ok(ColumnarValue::Array(Arc::new(boolean_result)));
}
None => SlicesIterator::new(mask).for_each(|(start, end)| {
if start > last_end {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org