You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/14 03:46:19 UTC
(datafusion) branch gh-readonly-queue/main/pr-25494-19b34a249e9b6651db249dba01cb66f864e377a1 created (now ff14758d93)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-25494-19b34a249e9b6651db249dba01cb66f864e377a1
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at ff14758d93 perf: copy bitmaps a word at a time in `pre_selection_scatter` (#25494)
This branch includes the following new commits:
new ff14758d93 perf: copy bitmaps a word at a time in `pre_selection_scatter` (#25494)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: perf: copy bitmaps a word at a time in `pre_selection_scatter` (#25494)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-25494-19b34a249e9b6651db249dba01cb66f864e377a1
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit ff14758d9383b4cb6264ec616081faf3762fb2d0
Author: Yousef Mahmoud <yo...@gmail.com>
AuthorDate: Wed Oct 14 03:45:55 2026 +0000
perf: copy bitmaps a word at a time in `pre_selection_scatter` (#25494)
## Which issue does this PR close?
- N/A
## Rationale for this change
`pre_selection_scatter` puts the short-circuited RHS result back at full
batch length. For each run of selected rows it slices the RHS and
appends one `Option<bool>` at a time, even though both sides are
bit-packed, and every `slice` clones and drops an `Arc`.
Since #25771, `AND` chains of cheap comparisons no longer reach this
function, so TPC-H is unchanged. It still runs for `OR`, and for `AND`
with a conjunct that is not cheap.
TPC-H SF30 `lineitem`, 24 threads, `l_quantity > K or l_discount >
0.05`, by share of rows that reach the RHS:
- 4%: 86.0ms -> 82.0ms
- 8%: 96.5ms -> 89.5ms
- 12%: 107.5ms -> 96.0ms
- 16%: 115.0ms -> 102.0ms
- 20%: 102.0ms -> 92.0ms
`l_quantity < K and l_comment like '%special%'` is 1-2% faster. The 22
TPC-H queries are within noise.
## What changes are included in this PR?
Borrow the RHS buffers once, copy each run with `append_packed_range`,
and only build a validity bitmap when the RHS has nulls.
## Are these changes tested?
Yes, by the existing tests, including `test_pre_selection_scatter`.
## Are there any user-facing changes?
No.
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
---
datafusion/physical-expr/src/expressions/binary.rs | 48 +++++++++++++++++++---
1 file changed, 42 insertions(+), 6 deletions(-)
diff --git a/datafusion/physical-expr/src/expressions/binary.rs b/datafusion/physical-expr/src/expressions/binary.rs
index 44df100ad8..053850b836 100644
--- a/datafusion/physical-expr/src/expressions/binary.rs
+++ b/datafusion/physical-expr/src/expressions/binary.rs
@@ -27,6 +27,7 @@ use std::hash::Hash;
use std::sync::Arc;
use arrow::array::*;
+use arrow::buffer::NullBuffer;
use arrow::compute::kernels::boolean::{and_kleene, or_kleene};
use arrow::compute::kernels::concat_elements::concat_elements_dyn;
use arrow::compute::{SlicesIterator, cast, filter, filter_record_batch};
@@ -1360,21 +1361,56 @@ fn pre_selection_scatter(
let mut last_end = 0;
match right_result {
Some(right_result) => {
+ // Borrow the RHS buffers once. `BooleanArray::slice` clones and
+ // drops an `Arc` for every run, which dominates this loop when the
+ // mask selects many short runs.
+ let right_values = right_result.values();
+ let right_offset = right_values.offset();
+ let right_bytes = right_values.values();
+ let right_nulls = right_result.nulls();
+
+ let mut values = BooleanBufferBuilder::new(result_len);
+ // Only build a validity bitmap when the RHS has nulls. Otherwise
+ // every run pays a second pass to write an all-ones mask that is
+ // then discarded.
+ let mut validity = right_nulls.map(|_| BooleanBufferBuilder::new(result_len));
+
SlicesIterator::new(mask).for_each(|(start, end)| {
if start > last_end {
- result_array_builder.append_n(start - last_end, fill_value);
+ let gap = start - last_end;
+ values.append_n(gap, fill_value);
+ if let Some(v) = validity.as_mut() {
+ v.append_n(gap, true);
+ }
}
- // Copy the selected RHS slice.
+ // Both sides are bit-packed, so copy the run a word at a
+ // time. Iterating the RHS would yield one `Option<bool>` and
+ // set one bit per row.
let len = end - start;
- right_result
- .slice(right_array_pos, len)
- .iter()
- .for_each(|v| result_array_builder.append_option(v));
+ let from = right_offset + right_array_pos;
+ values.append_packed_range(from..from + len, right_bytes);
+ if let (Some(v), Some(nulls)) = (validity.as_mut(), right_nulls) {
+ let nb = nulls.inner();
+ let nfrom = nb.offset() + right_array_pos;
+ v.append_packed_range(nfrom..nfrom + len, nb.values());
+ }
right_array_pos += len;
last_end = end;
});
+
+ if last_end < result_len {
+ let gap = result_len - last_end;
+ values.append_n(gap, fill_value);
+ if let Some(v) = validity.as_mut() {
+ v.append_n(gap, true);
+ }
+ }
+
+ let nulls = validity.map(|mut v| NullBuffer::new(v.finish()));
+ let boolean_result = BooleanArray::new(values.finish(), nulls);
+ return Ok(ColumnarValue::Array(Arc::new(boolean_result)));
}
None => SlicesIterator::new(mask).for_each(|(start, end)| {
if start > last_end {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org