You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "abokhalill (via GitHub)" <gi...@apache.org> on 2026/09/19 06:04:43 UTC
[PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
abokhalill opened a new pull request, #25494:
URL: https://github.com/apache/datafusion/pull/25494
## Which issue does this PR close?
- N/A
## Rationale for this change
`pre_selection_scatter` scatters the short-circuited RHS result back to the full batch length. For each run of selected rows it slices the RHS array and iterates it, appending one `Option<bool>` at a time:
```rust
right_result
.slice(right_array_pos, len)
.iter()
.for_each(|v| result_array_builder.append_option(v));
```
Both the RHS and the builder are bit-packed, so this sets one bit per row where `BooleanBufferBuilder::append_packed_range` copies a word at a time. The per-run `slice` also clones and drops an `Arc`, which shows up as the hottest instruction in the function when the mask selects many short runs.
Borrowing the RHS buffers once outside the loop removes the slice entirely, and the copy becomes a range copy for both the values and the validity bitmap.
Benchmarks: TPC-H SF30, parquet, 24 threads
q6, 388.7 ms → 356.5 ms, -8.3%
q12, 471.2 ms → 451.7 ms, -4.1%
q19, 657.0 ms → 644.0 ms, -2.0%
q3, 723.6 ms → 727.5 ms, +0.6%
q7, 1350.9 ms → 1359.6 ms, +0.7%
q10, 910.0 ms → 911.9 ms, +0.2%
The first three win every round of six; the rest are within noise.
The function is only reached for short-circuit AND/OR, so the gain depends on how much of the filter reaches the RHS. Varying selectivity with `l_quantity < K` on `lineitem`:
4% of rows selected, -1.3%
8%, -3.6%
16%, -7.1%
32%, -5.7%
48% and above, within noise
It peaks where the mask has many short runs, and decays at both ends: a nearly empty mask has little to copy, and a nearly full one is a single long run.
## What changes are included in this PR?
* Borrow the RHS values and nulls buffers once instead of slicing per run
* Copy each run with `append_packed_range` rather than per-row `append_option`
* Only build a validity bitmap when the RHS actually has nulls
## What is the testing strategy for this PR?
Existing tests pass, including `test_pre_selection_scatter`.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6076571656
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/25494#issuecomment-6076368077)
**Instance:** `c4a-highcpu-48` (12 vCPU / 65 GiB)
Comparing perf-pre-selection-scatter-bulk-copy (7a60002fae0d919dd1a09510828524abd167583e) to f9b7f34 (merge-base) [diff](https://github.com/apache/datafusion/compare/f9b7f3482a4166b7de7ba66b3ce3da39543751ca..7a60002fae0d919dd1a09510828524abd167583e)
<details><summary>Run configuration</summary>
```yaml
run benchmark binary_op
env:
BENCH_FILTER: "short_circuit"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 48
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 3 MiB (48 instances)
L1i cache: 3 MiB (48 instances)
L2 cache: 96 MiB (48 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD perf-pre-selection-scatter-bulk-copy
----- ---- ------------------------------------
short_circuit/and/all_false 1.04 66.4±1.61ns ? ?/sec 1.00 63.8±0.68ns ? ?/sec
short_circuit/and/all_true_in_and 1.00 693.7±2.92µs ? ?/sec 1.00 691.5±2.74µs ? ?/sec
short_circuit/and/one_true_first 1.00 298.1±0.91µs ? ?/sec 1.00 299.4±1.27µs ? ?/sec
short_circuit/and/one_true_last 1.00 269.9±1.46µs ? ?/sec 1.00 269.6±1.26µs ? ?/sec
short_circuit/and/one_true_middle 1.00 269.0±1.12µs ? ?/sec 1.01 271.7±1.27µs ? ?/sec
short_circuit/and/one_true_middle_left 1.00 265.8±0.66µs ? ?/sec 1.00 264.9±1.07µs ? ?/sec
short_circuit/and/one_true_middle_right 1.00 270.2±0.79µs ? ?/sec 1.00 269.3±1.06µs ? ?/sec
short_circuit/or/all_false_in_or 1.00 693.3±2.84µs ? ?/sec 1.08 749.3±85.43µs ? ?/sec
short_circuit/or/all_true 1.06 74.5±3.31ns ? ?/sec 1.00 70.5±0.35ns ? ?/sec
short_circuit/or/one_false_first 1.00 270.8±0.83µs ? ?/sec 1.00 269.8±0.72µs ? ?/sec
short_circuit/or/one_false_last 1.00 297.4±0.69µs ? ?/sec 1.00 297.4±0.77µs ? ?/sec
short_circuit/or/one_false_middle 1.00 296.7±0.69µs ? ?/sec 1.00 297.6±0.73µs ? ?/sec
short_circuit/or/one_false_middle_left 1.00 293.5±0.69µs ? ?/sec 1.00 292.1±0.73µs ? ?/sec
short_circuit/or/one_false_middle_right 1.00 296.7±0.78µs ? ?/sec 1.00 297.5±0.79µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**binary_op — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 310.1s |
| Peak memory | 84.4 MiB |
| Avg memory | 39.6 MiB |
| CPU user | 187.4s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
**binary_op — branch**
| Metric | Value |
|--------|-------|
| Wall time | 305.1s |
| Peak memory | 84.1 MiB |
| Avg memory | 39.9 MiB |
| CPU user | 184.8s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6076368077
run benchmark binary_op
```yaml
env:
BENCH_FILTER: short_circuit
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6035275502
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25494?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 82.37%. Comparing base ([`a522cd5`](https://app.codecov.io/gh/apache/datafusion/commit/a522cd5bcedabd6238a0462b881a7ef4cb5c6f66?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b1a7913`](https://app.codecov.io/gh/apache/datafusion/commit/b1a7913e8c4d55343f27a38bb5099dbb6f1aeadc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 256 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25494 +/- ##
==========================================
- Coverage 82.37% 82.37% -0.01%
==========================================
Files 1138 1138
Lines 433506 433529 +23
Branches 433506 433529 +23
==========================================
+ Hits 357102 357107 +5
- Misses 54850 54862 +12
- Partials 21554 21560 +6
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25494?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6141689847
🚀
@abokhalill
Thank you for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6085057184
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/25494#issuecomment-6084992423)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6084992423-3245-29csd 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing perf-pre-selection-scatter-bulk-copy (7a60002fae0d919dd1a09510828524abd167583e) to f9b7f34 (merge-base) [diff](https://github.com/apache/datafusion/compare/f9b7f3482a4166b7de7ba66b3ce3da39543751ca..7a60002fae0d919dd1a09510828524abd167583e)
<details><summary>Run configuration</summary>
```yaml
run benchmark binary_op
env:
BENCH_FILTER: "short_circuit"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#discussion_r4227847253
##########
datafusion/physical-expr/src/expressions/binary.rs:
##########
@@ -1360,21 +1361,56 @@ fn pre_selection_scatter(
let mut last_end = 0;
match right_result {
Some(right_result) => {
+ // Borrow the RHS buffers once. `BooleanArray::slice` clones and
+ // drops an `Arc` for every run, which dominates this loop when the
+ // mask selects many short runs.
+ let right_values = right_result.values();
+ let right_offset = right_values.offset();
+ let right_bytes = right_values.values();
+ let right_nulls = right_result.nulls();
+
+ let mut values = BooleanBufferBuilder::new(result_len);
Review Comment:
Could we move `result_array_builder` initialization into the `None` branch? The new `Some` branch allocates its own bitmap builder and returns without using the original one, so this would avoid an unnecessary allocation and free.
##########
datafusion/physical-expr/src/expressions/binary.rs:
##########
@@ -1360,21 +1361,56 @@ fn pre_selection_scatter(
let mut last_end = 0;
match right_result {
Some(right_result) => {
+ // Borrow the RHS buffers once. `BooleanArray::slice` clones and
+ // drops an `Arc` for every run, which dominates this loop when the
+ // mask selects many short runs.
+ let right_values = right_result.values();
+ let right_offset = right_values.offset();
+ let right_bytes = right_values.values();
+ let right_nulls = right_result.nulls();
+
+ let mut values = BooleanBufferBuilder::new(result_len);
+ // Only build a validity bitmap when the RHS has nulls. Otherwise
+ // every run pays a second pass to write an all-ones mask that is
+ // then discarded.
+ let mut validity = right_nulls.map(|_| BooleanBufferBuilder::new(result_len));
+
SlicesIterator::new(mask).for_each(|(start, end)| {
if start > last_end {
- result_array_builder.append_n(start - last_end, fill_value);
+ let gap = start - last_end;
+ values.append_n(gap, fill_value);
+ if let Some(v) = validity.as_mut() {
+ v.append_n(gap, true);
+ }
}
- // Copy the selected RHS slice.
+ // Both sides are bit-packed, so copy the run a word at a
+ // time. Iterating the RHS would yield one `Option<bool>` and
+ // set one bit per row.
let len = end - start;
- right_result
- .slice(right_array_pos, len)
- .iter()
- .for_each(|v| result_array_builder.append_option(v));
+ let from = right_offset + right_array_pos;
+ values.append_packed_range(from..from + len, right_bytes);
Review Comment:
Could we add a scatter regression test covering nullable RHS values, non-byte-aligned offsets, runs crossing byte and 64-bit boundaries, and leading, intermediate, and trailing gaps with both fill values? Please include independently offset value and validity buffers, since ordinary slices usually share the same offset, and compare the output against a simple row-wise reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6085173963
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/25494#issuecomment-6084992423)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing perf-pre-selection-scatter-bulk-copy (7a60002fae0d919dd1a09510828524abd167583e) to f9b7f34 (merge-base) [diff](https://github.com/apache/datafusion/compare/f9b7f3482a4166b7de7ba66b3ce3da39543751ca..7a60002fae0d919dd1a09510828524abd167583e)
<details><summary>Run configuration</summary>
```yaml
run benchmark binary_op
env:
BENCH_FILTER: "short_circuit"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD perf-pre-selection-scatter-bulk-copy
----- ---- ------------------------------------
short_circuit/and/all_false 1.04 66.6±1.82ns ? ?/sec 1.00 63.9±0.72ns ? ?/sec
short_circuit/and/all_true_in_and 1.01 696.0±3.04µs ? ?/sec 1.00 689.2±2.87µs ? ?/sec
short_circuit/and/one_true_first 1.00 298.2±1.55µs ? ?/sec 1.01 299.9±0.87µs ? ?/sec
short_circuit/and/one_true_last 1.00 270.3±1.18µs ? ?/sec 1.00 269.8±0.93µs ? ?/sec
short_circuit/and/one_true_middle 1.00 270.5±0.95µs ? ?/sec 1.00 269.7±0.80µs ? ?/sec
short_circuit/and/one_true_middle_left 1.01 266.8±0.82µs ? ?/sec 1.00 265.5±0.68µs ? ?/sec
short_circuit/and/one_true_middle_right 1.01 272.3±1.12µs ? ?/sec 1.00 269.9±0.98µs ? ?/sec
short_circuit/or/all_false_in_or 1.00 693.3±3.13µs ? ?/sec 1.00 691.6±3.55µs ? ?/sec
short_circuit/or/all_true 1.00 68.9±0.28ns ? ?/sec 1.03 70.9±0.36ns ? ?/sec
short_circuit/or/one_false_first 1.00 269.9±1.19µs ? ?/sec 1.00 269.6±0.69µs ? ?/sec
short_circuit/or/one_false_last 1.00 297.0±1.30µs ? ?/sec 1.00 296.9±0.66µs ? ?/sec
short_circuit/or/one_false_middle 1.00 296.1±0.80µs ? ?/sec 1.01 298.0±1.44µs ? ?/sec
short_circuit/or/one_false_middle_left 1.00 293.6±0.72µs ? ?/sec 1.00 293.9±1.63µs ? ?/sec
short_circuit/or/one_false_middle_right 1.01 299.5±0.74µs ? ?/sec 1.00 296.2±0.59µs ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**binary_op — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 220.1s |
| Peak memory | 83.5 MiB |
| Avg memory | 54.9 MiB |
| CPU user | 185.5s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
**binary_op — branch**
| Metric | Value |
|--------|-------|
| Wall time | 215.0s |
| Peak memory | 84.7 MiB |
| Avg memory | 56.8 MiB |
| CPU user | 184.1s |
| CPU sys | 0.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6076429084
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/25494#issuecomment-6076368077)
**Instance:** `c4a-highcpu-48` (12 vCPU / 65 GiB) | `Linux bench-c6076368077-3230-btb2m 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 48
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 3 MiB (48 instances)
L1i cache: 3 MiB (48 instances)
L2 cache: 96 MiB (48 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing perf-pre-selection-scatter-bulk-copy (7a60002fae0d919dd1a09510828524abd167583e) to f9b7f34 (merge-base) [diff](https://github.com/apache/datafusion/compare/f9b7f3482a4166b7de7ba66b3ce3da39543751ca..7a60002fae0d919dd1a09510828524abd167583e)
<details><summary>Run configuration</summary>
```yaml
run benchmark binary_op
env:
BENCH_FILTER: "short_circuit"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6034776940
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-physical-expr v55.1.0 (current)
Built [ 58.613s] (current)
Parsing datafusion-physical-expr v55.1.0 (current)
Parsed [ 0.048s] (current)
Building datafusion-physical-expr v55.1.0 (baseline)
Built [ 30.443s] (baseline)
Parsing datafusion-physical-expr v55.1.0 (baseline)
Parsed [ 0.050s] (baseline)
Checking datafusion-physical-expr v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.363s] 223 checks: 221 pass, 2 fail, 0 warn, 31 skip
--- failure inherent_method_missing: pub method removed or renamed ---
Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/inherent_method_missing.ron
Failed in:
RangePartitioning::try_new_with_samples, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/ade6fce556fc3cf9ea18014d48fe532012268e7b/datafusion/physical-expr/src/partitioning.rs:277
RangePartitioning::samples, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/ade6fce556fc3cf9ea18014d48fe532012268e7b/datafusion/physical-expr/src/partitioning.rs:305
RangePartitioning::max_partition_count, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/ade6fce556fc3cf9ea18014d48fe532012268e7b/datafusion/physical-expr/src/partitioning.rs:320
RangePartitioning::has_same_layout, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/ade6fce556fc3cf9ea18014d48fe532012268e7b/datafusion/physical-expr/src/partitioning.rs:330
RangePartitioning::scale, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/ade6fce556fc3cf9ea18014d48fe532012268e7b/datafusion/physical-expr/src/partitioning.rs:343
DynamicFilterPhysicalExpr::is_complete, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/ade6fce556fc3cf9ea18014d48fe532012268e7b/datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs:403
ProjectionExprs::project_statistics_with_input_schema, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/ade6fce556fc3cf9ea18014d48fe532012268e7b/datafusion/physical-expr/src/projection.rs:722
--- failure type_method_marked_deprecated: type method #[deprecated] added ---
Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/type_method_marked_deprecated.ron
Failed in:
method datafusion_physical_expr::scalar_subquery::ScalarSubqueryExpr::index in /home/runner/work/datafusion/datafusion/datafusion/physical-expr/src/scalar_subquery.rs:103
Summary semver requires new major version: 1 major and 1 minor checks failed
Finished [ 91.052s] datafusion-physical-expr
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6034791199
@abokhalill
Thanks for your contribution.
Can you resolve the merge conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6084992423
run benchmark binary_op
```yaml
env:
BENCH_FILTER: short_circuit
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "abokhalill (via GitHub)" <gi...@apache.org>.
abokhalill commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6043767293
@kosiew done. the speedups are actually even more prominent now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #25494:
URL: https://github.com/apache/datafusion/pull/25494#issuecomment-6084996145
Trying to see if performance changes are reproducable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: copy bitmaps a word at a time in `pre_selection_scatter` [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #25494:
URL: https://github.com/apache/datafusion/pull/25494
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org