You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "liamzwbao (via GitHub)" <gi...@apache.org> on 2026/05/16 16:30:35 UTC
[PR] [arrow-select] Replace `ArrayData` with direct `Array` construction [arrow-rs]
liamzwbao opened a new pull request, #9986:
URL: https://github.com/apache/arrow-rs/pull/9986
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
-->
- Part of #9298.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
- Replaces several `ArrayDataBuilder` paths in `arrow-select/src/filter.rs` with direct typed array constructors.
- Adds a small helper for filtered null buffers that reuses the already-computed null count.
# Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
If this PR claims a performance improvement, please include evidence such as benchmark results.
-->
Covered by exsiting tests
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please call them out.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501461992
@liamzwbao -- some of these results look great but some look like they got slightly slower. Can you investigate (perhaps by using `::new_unchecked` to skip redundant validation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501488450
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501463694)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4501463694-234-wjwbf 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing issue-9298-repalce-array-data-arrow-select (34b1837721f17052738e0e0ca18178b450118177) to accb1cf (merge-base) [diff](https://github.com/apache/arrow-rs/compare/accb1cf12a33878f179f8bb3c78c1a740fb97216..34b1837721f17052738e0e0ca18178b450118177)
BENCH_NAME=filter_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501219436
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501184291)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4501184291-232-42vmp 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing issue-9298-repalce-array-data-arrow-select (34b1837721f17052738e0e0ca18178b450118177) to accb1cf (merge-base) [diff](https://github.com/apache/arrow-rs/compare/accb1cf12a33878f179f8bb3c78c1a740fb97216..34b1837721f17052738e0e0ca18178b450118177)
BENCH_NAME=filter_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#discussion_r3276084881
##########
arrow-select/src/filter.rs:
##########
@@ -824,17 +825,10 @@ where
IterationStrategy::All | IterationStrategy::None => unreachable!(),
}
- let mut builder = ArrayDataBuilder::new(T::DATA_TYPE)
- .len(predicate.count)
- .add_buffer(filter.dst_offsets.into())
- .add_buffer(filter.dst_values.into());
+ let offsets = unsafe { OffsetBuffer::new_unchecked(filter.dst_offsets.into()) };
Review Comment:
Here it would also be nice to comment about why this is safe (what assumptions it relies on). However, I see the existing code doesn't have a safety comment
```rust
// Safety: offsets are correctly constructed
```
##########
arrow-select/src/filter.rs:
##########
@@ -579,6 +580,14 @@ fn filter_null_mask(
Some((null_count, nulls))
}
+/// Filters `nulls` and reuses the computed `null_count` to avoid scanning the bitmap.
+fn filter_nulls(nulls: Option<&NullBuffer>, predicate: &FilterPredicate) -> Option<NullBuffer> {
+ let (null_count, nulls) = filter_null_mask(nulls, predicate)?;
+ let buffer = BooleanBuffer::new(nulls, 0, predicate.count);
+
+ Some(unsafe { NullBuffer::new_unchecked(buffer, null_count) })
Review Comment:
can we please add a safety comment here explaining why this is safe:
```suggestion
// Safety: null_count return from filter_null_mas is correct
Some(unsafe { NullBuffer::new_unchecked(buffer, null_count) })
```
It might also be nice to add a debug assert here to verify in debug builds
```rust
debug_assert_eq!(null_count, nulls.num_zeros())
```
##########
arrow-select/src/filter.rs:
##########
@@ -843,17 +837,11 @@ fn filter_byte_view<T: ByteViewType>(
predicate: &FilterPredicate,
) -> GenericByteViewArray<T> {
let new_view_buffer = filter_native(array.views(), predicate);
+ let views = ScalarBuffer::new(new_view_buffer, 0, predicate.count);
Review Comment:
Here (and other places) you can probably use the unchecked variants too to skip some checks, if we need to get additional speed (`ScalarBuffer::new_unchecked`)
However, given your PR removes an allocation (the buffers array) I suspect this is already going to be faster and avoiding unsafe is a nice bonus ❤️
##########
arrow-select/src/filter.rs:
##########
@@ -579,6 +580,14 @@ fn filter_null_mask(
Some((null_count, nulls))
}
+/// Filters `nulls` and reuses the computed `null_count` to avoid scanning the bitmap.
+fn filter_nulls(nulls: Option<&NullBuffer>, predicate: &FilterPredicate) -> Option<NullBuffer> {
Review Comment:
What do you think about making this a method on `FilterPredicate`? That would make it easier to find / reuse I think.
```rust
impl FilterPredicate {
fn filter_nulls(&self, nulls: Option<&NullBuffer>) -> Option<NullBuffer> {
...
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4529720982
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9986#issuecomment-4529714415)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4529714415-307-jkdnf 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing issue-9298-repalce-array-data-arrow-select (c3a78c0c4d33e35c89fb921f166984183c233646) to 7b335b7 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/7b335b7712089c1270ab1a989336908a240cc812..c3a78c0c4d33e35c89fb921f166984183c233646)
BENCH_NAME=filter_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501964407
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501756916)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group issue-9298-repalce-array-data-arrow-select main
----- ------------------------------------------ ----
filter context decimal128 (kept 1/2) 1.00 20.3±0.06µs ? ?/sec 1.01 20.5±0.11µs ? ?/sec
filter context decimal128 high selectivity (kept 1023/1024) 1.00 19.7±0.10µs ? ?/sec 1.02 20.0±0.13µs ? ?/sec
filter context decimal128 low selectivity (kept 1/1024) 1.00 149.4±1.16ns ? ?/sec 1.35 201.3±9.12ns ? ?/sec
filter context f32 (kept 1/2) 1.00 83.5±5.57µs ? ?/sec 1.02 85.1±6.59µs ? ?/sec
filter context f32 high selectivity (kept 1023/1024) 1.00 5.5±0.01µs ? ?/sec 1.06 5.8±0.01µs ? ?/sec
filter context f32 low selectivity (kept 1/1024) 1.00 325.9±11.76ns ? ?/sec 1.34 437.4±17.67ns ? ?/sec
filter context fsb with value length 20 (kept 1/2) 1.00 71.5±5.33µs ? ?/sec 1.03 73.4±6.38µs ? ?/sec
filter context fsb with value length 20 high selectivity (kept 1023/1024) 1.00 71.6±5.40µs ? ?/sec 1.02 73.4±6.37µs ? ?/sec
filter context fsb with value length 20 low selectivity (kept 1/1024) 1.00 71.5±5.33µs ? ?/sec 1.03 73.4±6.36µs ? ?/sec
filter context fsb with value length 5 (kept 1/2) 1.00 71.6±5.33µs ? ?/sec 1.03 73.5±6.48µs ? ?/sec
filter context fsb with value length 5 high selectivity (kept 1023/1024) 1.00 71.5±5.39µs ? ?/sec 1.03 73.4±6.48µs ? ?/sec
filter context fsb with value length 5 low selectivity (kept 1/1024) 1.00 71.6±5.38µs ? ?/sec 1.03 73.4±6.46µs ? ?/sec
filter context fsb with value length 50 (kept 1/2) 1.00 71.5±5.33µs ? ?/sec 1.03 73.4±6.39µs ? ?/sec
filter context fsb with value length 50 high selectivity (kept 1023/1024) 1.00 71.3±5.44µs ? ?/sec 1.03 73.4±6.49µs ? ?/sec
filter context fsb with value length 50 low selectivity (kept 1/1024) 1.00 71.4±5.37µs ? ?/sec 1.03 73.4±6.38µs ? ?/sec
filter context i32 (kept 1/2) 1.00 12.3±0.01µs ? ?/sec 1.00 12.4±0.01µs ? ?/sec
filter context i32 high selectivity (kept 1023/1024) 1.00 3.7±0.00µs ? ?/sec 1.01 3.8±0.01µs ? ?/sec
filter context i32 low selectivity (kept 1/1024) 1.00 146.9±2.07ns ? ?/sec 1.38 202.7±8.87ns ? ?/sec
filter context i32 w NULLs (kept 1/2) 1.00 84.0±5.47µs ? ?/sec 1.04 87.3±6.59µs ? ?/sec
filter context i32 w NULLs high selectivity (kept 1023/1024) 1.00 5.5±0.01µs ? ?/sec 1.01 5.6±0.01µs ? ?/sec
filter context i32 w NULLs low selectivity (kept 1/1024) 1.00 326.3±11.86ns ? ?/sec 1.33 435.3±17.96ns ? ?/sec
filter context mixed string view (kept 1/2) 1.02 94.7±6.66µs ? ?/sec 1.00 92.7±5.27µs ? ?/sec
filter context mixed string view high selectivity (kept 1023/1024) 1.00 20.9±0.32µs ? ?/sec 1.07 22.4±0.07µs ? ?/sec
filter context mixed string view low selectivity (kept 1/1024) 1.00 420.7±17.74ns ? ?/sec 1.44 605.2±9.56ns ? ?/sec
filter context short string view (kept 1/2) 1.02 94.6±6.59µs ? ?/sec 1.00 92.8±5.31µs ? ?/sec
filter context short string view high selectivity (kept 1023/1024) 1.00 21.7±0.08µs ? ?/sec 1.00 21.8±0.13µs ? ?/sec
filter context short string view low selectivity (kept 1/1024) 1.00 354.2±11.27ns ? ?/sec 1.32 466.7±10.47ns ? ?/sec
filter context string (kept 1/2) 1.00 424.6±7.11µs ? ?/sec 1.00 422.9±6.59µs ? ?/sec
filter context string dictionary (kept 1/2) 1.01 12.9±0.02µs ? ?/sec 1.00 12.8±0.02µs ? ?/sec
filter context string dictionary high selectivity (kept 1023/1024) 1.01 4.2±0.02µs ? ?/sec 1.00 4.1±0.01µs ? ?/sec
filter context string dictionary low selectivity (kept 1/1024) 1.00 532.2±10.20ns ? ?/sec 1.10 587.2±4.78ns ? ?/sec
filter context string dictionary w NULLs (kept 1/2) 1.00 84.5±5.47µs ? ?/sec 1.02 86.4±6.58µs ? ?/sec
filter context string dictionary w NULLs high selectivity (kept 1023/1024) 1.00 6.0±0.01µs ? ?/sec 1.00 6.1±0.01µs ? ?/sec
filter context string dictionary w NULLs low selectivity (kept 1/1024) 1.00 711.8±11.63ns ? ?/sec 1.15 816.8±12.15ns ? ?/sec
filter context string high selectivity (kept 1023/1024) 1.00 312.4±5.00µs ? ?/sec 1.02 319.1±1.51µs ? ?/sec
filter context string low selectivity (kept 1/1024) 1.00 761.4±10.62ns ? ?/sec 1.19 904.2±8.94ns ? ?/sec
filter context u8 (kept 1/2) 1.16 14.0±0.08µs ? ?/sec 1.00 12.1±0.02µs ? ?/sec
filter context u8 high selectivity (kept 1023/1024) 1.00 1098.5±2.88ns ? ?/sec 1.00 1101.4±10.19ns ? ?/sec
filter context u8 low selectivity (kept 1/1024) 1.00 129.0±1.60ns ? ?/sec 1.44 185.6±9.43ns ? ?/sec
filter context u8 w NULLs (kept 1/2) 1.00 85.5±6.42µs ? ?/sec 1.00 85.7±6.48µs ? ?/sec
filter context u8 w NULLs high selectivity (kept 1023/1024) 1.00 2.9±0.01µs ? ?/sec 1.02 2.9±0.01µs ? ?/sec
filter context u8 w NULLs low selectivity (kept 1/1024) 1.00 319.4±13.02ns ? ?/sec 1.34 428.4±20.01ns ? ?/sec
filter decimal128 (kept 1/2) 1.00 35.7±0.10µs ? ?/sec 1.00 35.7±0.07µs ? ?/sec
filter decimal128 high selectivity (kept 1023/1024) 1.01 19.4±0.30µs ? ?/sec 1.00 19.1±0.16µs ? ?/sec
filter decimal128 low selectivity (kept 1/1024) 1.00 1568.0±2.98ns ? ?/sec 1.03 1609.3±3.83ns ? ?/sec
filter f32 (kept 1/2) 1.00 106.8±0.44µs ? ?/sec 1.01 108.2±0.35µs ? ?/sec
filter fsb with value length 20 (kept 1/2) 1.02 80.0±0.08µs ? ?/sec 1.00 78.3±0.09µs ? ?/sec
filter fsb with value length 20 high selectivity (kept 1023/1024) 1.04 25.3±0.57µs ? ?/sec 1.00 24.3±0.82µs ? ?/sec
filter fsb with value length 20 low selectivity (kept 1/1024) 1.00 1622.0±4.19ns ? ?/sec 1.02 1655.0±5.24ns ? ?/sec
filter fsb with value length 5 (kept 1/2) 1.02 79.4±0.05µs ? ?/sec 1.00 78.0±0.13µs ? ?/sec
filter fsb with value length 5 high selectivity (kept 1023/1024) 1.00 6.0±0.05µs ? ?/sec 1.00 6.0±0.09µs ? ?/sec
filter fsb with value length 5 low selectivity (kept 1/1024) 1.00 1582.0±1.87ns ? ?/sec 1.03 1628.3±10.51ns ? ?/sec
filter fsb with value length 50 (kept 1/2) 1.00 124.6±0.55µs ? ?/sec 1.00 124.6±0.18µs ? ?/sec
filter fsb with value length 50 high selectivity (kept 1023/1024) 1.08 92.7±6.98µs ? ?/sec 1.00 86.1±9.40µs ? ?/sec
filter fsb with value length 50 low selectivity (kept 1/1024) 1.00 1629.0±4.19ns ? ?/sec 1.04 1687.8±5.42ns ? ?/sec
filter i32 (kept 1/2) 1.00 29.2±0.04µs ? ?/sec 1.01 29.6±0.03µs ? ?/sec
filter i32 high selectivity (kept 1023/1024) 1.00 4.9±0.08µs ? ?/sec 1.00 4.9±0.07µs ? ?/sec
filter i32 low selectivity (kept 1/1024) 1.00 1505.5±4.99ns ? ?/sec 1.03 1545.8±2.55ns ? ?/sec
filter optimize (kept 1/2) 1.03 28.2±0.05µs ? ?/sec 1.00 27.5±0.04µs ? ?/sec
filter optimize high selectivity (kept 1023/1024) 1.01 1339.3±3.48ns ? ?/sec 1.00 1328.8±1.69ns ? ?/sec
filter optimize low selectivity (kept 1/1024) 1.01 1328.4±1.87ns ? ?/sec 1.00 1317.9±1.50ns ? ?/sec
filter run array (kept 1/2) 1.00 285.7±2.67µs ? ?/sec 1.04 296.0±1.15µs ? ?/sec
filter run array high selectivity (kept 1023/1024) 1.04 297.4±4.59µs ? ?/sec 1.00 285.8±5.59µs ? ?/sec
filter run array low selectivity (kept 1/1024) 1.00 236.3±0.94µs ? ?/sec 1.00 236.3±0.92µs ? ?/sec
filter single record batch 1.01 29.6±0.03µs ? ?/sec 1.00 29.4±0.06µs ? ?/sec
filter u8 (kept 1/2) 1.01 29.6±0.04µs ? ?/sec 1.00 29.3±0.04µs ? ?/sec
filter u8 high selectivity (kept 1023/1024) 1.00 2.2±0.04µs ? ?/sec 1.03 2.2±0.04µs ? ?/sec
filter u8 low selectivity (kept 1/1024) 1.00 1460.0±4.14ns ? ?/sec 1.05 1532.7±3.47ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 670.2s |
| Peak memory | 3.0 GiB |
| Avg memory | 3.0 GiB |
| CPU user | 662.6s |
| CPU sys | 0.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 685.2s |
| Peak memory | 3.0 GiB |
| Avg memory | 3.0 GiB |
| CPU user | 681.2s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501184291
run benchmark filter_kernels
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "liamzwbao (via GitHub)" <gi...@apache.org>.
liamzwbao commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4527125924
Hi @alamb, thanks for running those benchmarks! same for me that I didn't find a clear improvement or regression. Will review the comments tomorrow and investigate a bit of the new checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "liamzwbao (via GitHub)" <gi...@apache.org>.
liamzwbao commented on code in PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#discussion_r3295127150
##########
arrow-select/src/filter.rs:
##########
@@ -579,6 +580,14 @@ fn filter_null_mask(
Some((null_count, nulls))
}
+/// Filters `nulls` and reuses the computed `null_count` to avoid scanning the bitmap.
+fn filter_nulls(nulls: Option<&NullBuffer>, predicate: &FilterPredicate) -> Option<NullBuffer> {
Review Comment:
I think this is a good idea. Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4529714161
> Hi @alamb, thanks for running those benchmarks! From local testing, I didn't find a clear regression but some sort of improvements
I agree -- I think given we are down to optmizing away single allocations the overall benefit is likely to be hard to measure -- though in aggregate these allocations I suspect take up a non trivial amount of time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4502544924
I would say the benchmarks don't show a clear win / downside one way or the other
I do think it would be good to avoid the new checks just to be sure but otherwise this PR is ready to go.
Thank you again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4529714415
run benchmarks filter_kernels
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501778392
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501756916)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4501756916-238-fwm4b 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing issue-9298-repalce-array-data-arrow-select (34b1837721f17052738e0e0ca18178b450118177) to accb1cf (merge-base) [diff](https://github.com/apache/arrow-rs/compare/accb1cf12a33878f179f8bb3c78c1a740fb97216..34b1837721f17052738e0e0ca18178b450118177)
BENCH_NAME=filter_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench filter_kernels
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501666195
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501463694)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group issue-9298-repalce-array-data-arrow-select main
----- ------------------------------------------ ----
filter context decimal128 (kept 1/2) 1.00 19.9±0.12µs ? ?/sec 1.06 21.1±0.12µs ? ?/sec
filter context decimal128 high selectivity (kept 1023/1024) 1.00 19.4±0.24µs ? ?/sec 1.04 20.1±0.15µs ? ?/sec
filter context decimal128 low selectivity (kept 1/1024) 1.00 151.2±1.59ns ? ?/sec 1.31 198.8±1.41ns ? ?/sec
filter context f32 (kept 1/2) 1.00 85.2±5.69µs ? ?/sec 1.02 86.7±6.81µs ? ?/sec
filter context f32 high selectivity (kept 1023/1024) 1.03 5.7±0.01µs ? ?/sec 1.00 5.6±0.01µs ? ?/sec
filter context f32 low selectivity (kept 1/1024) 1.00 324.1±12.51ns ? ?/sec 1.34 435.1±12.10ns ? ?/sec
filter context fsb with value length 20 (kept 1/2) 1.00 73.7±14.65µs ? ?/sec 1.00 73.5±6.57µs ? ?/sec
filter context fsb with value length 20 high selectivity (kept 1023/1024) 1.00 73.6±14.67µs ? ?/sec 1.00 73.5±6.61µs ? ?/sec
filter context fsb with value length 20 low selectivity (kept 1/1024) 1.00 73.5±14.64µs ? ?/sec 1.00 73.4±6.48µs ? ?/sec
filter context fsb with value length 5 (kept 1/2) 1.00 73.7±14.62µs ? ?/sec 1.00 73.4±6.61µs ? ?/sec
filter context fsb with value length 5 high selectivity (kept 1023/1024) 1.00 73.7±14.65µs ? ?/sec 1.00 73.4±6.60µs ? ?/sec
filter context fsb with value length 5 low selectivity (kept 1/1024) 1.00 73.5±14.65µs ? ?/sec 1.00 73.2±6.55µs ? ?/sec
filter context fsb with value length 50 (kept 1/2) 1.00 73.5±14.68µs ? ?/sec 1.00 73.4±6.47µs ? ?/sec
filter context fsb with value length 50 high selectivity (kept 1023/1024) 1.00 73.4±14.66µs ? ?/sec 1.00 73.4±6.50µs ? ?/sec
filter context fsb with value length 50 low selectivity (kept 1/1024) 1.00 73.6±14.59µs ? ?/sec 1.00 73.3±6.52µs ? ?/sec
filter context i32 (kept 1/2) 1.00 12.3±0.01µs ? ?/sec 1.01 12.4±0.01µs ? ?/sec
filter context i32 high selectivity (kept 1023/1024) 1.00 3.7±0.00µs ? ?/sec 1.00 3.7±0.01µs ? ?/sec
filter context i32 low selectivity (kept 1/1024) 1.00 150.0±1.66ns ? ?/sec 1.33 198.9±1.48ns ? ?/sec
filter context i32 w NULLs (kept 1/2) 1.00 84.3±5.46µs ? ?/sec 1.02 85.9±6.60µs ? ?/sec
filter context i32 w NULLs high selectivity (kept 1023/1024) 1.00 5.5±0.01µs ? ?/sec 1.02 5.6±0.01µs ? ?/sec
filter context i32 w NULLs low selectivity (kept 1/1024) 1.00 328.9±12.03ns ? ?/sec 1.34 442.0±11.62ns ? ?/sec
filter context mixed string view (kept 1/2) 1.02 94.1±6.72µs ? ?/sec 1.00 92.1±5.57µs ? ?/sec
filter context mixed string view high selectivity (kept 1023/1024) 1.00 20.6±0.17µs ? ?/sec 1.01 20.9±0.38µs ? ?/sec
filter context mixed string view low selectivity (kept 1/1024) 1.00 417.7±17.91ns ? ?/sec 1.43 599.3±10.68ns ? ?/sec
filter context short string view (kept 1/2) 1.02 93.9±6.61µs ? ?/sec 1.00 92.1±5.47µs ? ?/sec
filter context short string view high selectivity (kept 1023/1024) 1.01 20.4±0.26µs ? ?/sec 1.00 20.1±0.31µs ? ?/sec
filter context short string view low selectivity (kept 1/1024) 1.00 354.2±12.74ns ? ?/sec 1.35 477.6±10.84ns ? ?/sec
filter context string (kept 1/2) 1.00 427.8±10.49µs ? ?/sec 1.01 430.1±8.03µs ? ?/sec
filter context string dictionary (kept 1/2) 1.02 13.0±0.65µs ? ?/sec 1.00 12.7±0.03µs ? ?/sec
filter context string dictionary high selectivity (kept 1023/1024) 1.00 4.1±0.01µs ? ?/sec 1.01 4.1±0.01µs ? ?/sec
filter context string dictionary low selectivity (kept 1/1024) 1.00 528.4±7.92ns ? ?/sec 1.10 579.4±4.23ns ? ?/sec
filter context string dictionary w NULLs (kept 1/2) 1.00 84.5±5.43µs ? ?/sec 1.02 86.4±6.57µs ? ?/sec
filter context string dictionary w NULLs high selectivity (kept 1023/1024) 1.00 6.0±0.01µs ? ?/sec 1.00 6.0±0.01µs ? ?/sec
filter context string dictionary w NULLs low selectivity (kept 1/1024) 1.00 718.2±11.94ns ? ?/sec 1.14 821.2±13.79ns ? ?/sec
filter context string high selectivity (kept 1023/1024) 1.01 327.9±27.24µs ? ?/sec 1.00 325.5±14.40µs ? ?/sec
filter context string low selectivity (kept 1/1024) 1.00 757.3±10.42ns ? ?/sec 1.19 903.1±9.32ns ? ?/sec
filter context u8 (kept 1/2) 1.00 12.0±0.01µs ? ?/sec 1.01 12.1±0.02µs ? ?/sec
filter context u8 high selectivity (kept 1023/1024) 1.00 1100.9±3.17ns ? ?/sec 1.00 1096.4±3.61ns ? ?/sec
filter context u8 low selectivity (kept 1/1024) 1.00 128.6±1.46ns ? ?/sec 1.42 182.9±1.28ns ? ?/sec
filter context u8 w NULLs (kept 1/2) 1.00 85.5±6.45µs ? ?/sec 1.00 85.8±6.55µs ? ?/sec
filter context u8 w NULLs high selectivity (kept 1023/1024) 1.00 2.9±0.01µs ? ?/sec 1.02 2.9±0.01µs ? ?/sec
filter context u8 w NULLs low selectivity (kept 1/1024) 1.00 314.4±12.14ns ? ?/sec 1.31 411.1±12.46ns ? ?/sec
filter decimal128 (kept 1/2) 1.00 34.9±0.08µs ? ?/sec 1.02 35.8±0.09µs ? ?/sec
filter decimal128 high selectivity (kept 1023/1024) 1.01 19.4±0.08µs ? ?/sec 1.00 19.2±0.11µs ? ?/sec
filter decimal128 low selectivity (kept 1/1024) 1.00 1567.2±1.70ns ? ?/sec 1.04 1624.5±5.10ns ? ?/sec
filter f32 (kept 1/2) 1.00 105.7±0.54µs ? ?/sec 1.02 107.8±0.42µs ? ?/sec
filter fsb with value length 20 (kept 1/2) 1.03 80.3±0.16µs ? ?/sec 1.00 78.1±0.22µs ? ?/sec
filter fsb with value length 20 high selectivity (kept 1023/1024) 1.03 25.2±0.73µs ? ?/sec 1.00 24.5±0.78µs ? ?/sec
filter fsb with value length 20 low selectivity (kept 1/1024) 1.00 1617.5±2.39ns ? ?/sec 1.02 1655.4±5.60ns ? ?/sec
filter fsb with value length 5 (kept 1/2) 1.02 79.6±0.07µs ? ?/sec 1.00 78.0±0.08µs ? ?/sec
filter fsb with value length 5 high selectivity (kept 1023/1024) 1.00 6.1±0.54µs ? ?/sec 1.03 6.3±0.09µs ? ?/sec
filter fsb with value length 5 low selectivity (kept 1/1024) 1.00 1586.3±4.48ns ? ?/sec 1.03 1638.7±3.39ns ? ?/sec
filter fsb with value length 50 (kept 1/2) 1.00 123.4±0.66µs ? ?/sec 1.00 123.5±0.20µs ? ?/sec
filter fsb with value length 50 high selectivity (kept 1023/1024) 1.03 90.4±5.97µs ? ?/sec 1.00 87.9±8.43µs ? ?/sec
filter fsb with value length 50 low selectivity (kept 1/1024) 1.00 1622.1±2.05ns ? ?/sec 1.03 1664.8±4.19ns ? ?/sec
filter i32 (kept 1/2) 1.00 29.3±0.02µs ? ?/sec 1.01 29.6±0.02µs ? ?/sec
filter i32 high selectivity (kept 1023/1024) 1.00 4.9±0.08µs ? ?/sec 1.01 4.9±0.07µs ? ?/sec
filter i32 low selectivity (kept 1/1024) 1.00 1516.5±6.90ns ? ?/sec 1.02 1550.4±3.70ns ? ?/sec
filter optimize (kept 1/2) 1.02 28.2±0.07µs ? ?/sec 1.00 27.6±0.12µs ? ?/sec
filter optimize high selectivity (kept 1023/1024) 1.00 1330.5±1.38ns ? ?/sec 1.00 1336.0±2.37ns ? ?/sec
filter optimize low selectivity (kept 1/1024) 1.01 1326.1±1.57ns ? ?/sec 1.00 1317.9±1.52ns ? ?/sec
filter run array (kept 1/2) 1.00 286.4±2.82µs ? ?/sec 1.00 286.1±3.29µs ? ?/sec
filter run array high selectivity (kept 1023/1024) 1.01 288.3±8.41µs ? ?/sec 1.00 286.8±5.90µs ? ?/sec
filter run array low selectivity (kept 1/1024) 1.01 237.4±2.88µs ? ?/sec 1.00 236.2±0.79µs ? ?/sec
filter single record batch 1.00 29.4±0.07µs ? ?/sec 1.01 29.6±0.05µs ? ?/sec
filter u8 (kept 1/2) 1.01 29.5±0.07µs ? ?/sec 1.00 29.2±0.03µs ? ?/sec
filter u8 high selectivity (kept 1023/1024) 1.00 2.2±0.04µs ? ?/sec 1.04 2.3±0.04µs ? ?/sec
filter u8 low selectivity (kept 1/1024) 1.00 1463.6±4.80ns ? ?/sec 1.04 1526.3±2.92ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 670.1s |
| Peak memory | 3.0 GiB |
| Avg memory | 3.0 GiB |
| CPU user | 667.5s |
| CPU sys | 0.8s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 685.2s |
| Peak memory | 3.0 GiB |
| Avg memory | 3.0 GiB |
| CPU user | 681.0s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501446723
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501184291)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group issue-9298-repalce-array-data-arrow-select main
----- ------------------------------------------ ----
filter context decimal128 (kept 1/2) 1.03 20.9±0.05µs ? ?/sec 1.00 20.2±0.11µs ? ?/sec
filter context decimal128 high selectivity (kept 1023/1024) 1.00 19.0±0.39µs ? ?/sec 1.01 19.2±0.12µs ? ?/sec
filter context decimal128 low selectivity (kept 1/1024) 1.00 148.3±1.39ns ? ?/sec 1.30 193.1±0.90ns ? ?/sec
filter context f32 (kept 1/2) 1.02 85.2±6.64µs ? ?/sec 1.00 83.2±5.34µs ? ?/sec
filter context f32 high selectivity (kept 1023/1024) 1.00 5.5±0.01µs ? ?/sec 1.03 5.7±0.01µs ? ?/sec
filter context f32 low selectivity (kept 1/1024) 1.00 326.3±14.90ns ? ?/sec 1.31 428.5±9.47ns ? ?/sec
filter context fsb with value length 20 (kept 1/2) 1.02 73.1±6.45µs ? ?/sec 1.00 71.4±5.26µs ? ?/sec
filter context fsb with value length 20 high selectivity (kept 1023/1024) 1.03 73.4±6.50µs ? ?/sec 1.00 71.0±5.30µs ? ?/sec
filter context fsb with value length 20 low selectivity (kept 1/1024) 1.02 73.1±6.45µs ? ?/sec 1.00 71.4±5.25µs ? ?/sec
filter context fsb with value length 5 (kept 1/2) 1.03 73.4±6.34µs ? ?/sec 1.00 71.6±5.14µs ? ?/sec
filter context fsb with value length 5 high selectivity (kept 1023/1024) 1.02 73.2±6.53µs ? ?/sec 1.00 71.6±5.26µs ? ?/sec
filter context fsb with value length 5 low selectivity (kept 1/1024) 1.03 73.3±6.37µs ? ?/sec 1.00 71.4±5.20µs ? ?/sec
filter context fsb with value length 50 (kept 1/2) 1.03 73.3±6.34µs ? ?/sec 1.00 71.3±5.30µs ? ?/sec
filter context fsb with value length 50 high selectivity (kept 1023/1024) 1.03 73.2±6.56µs ? ?/sec 1.00 71.4±5.19µs ? ?/sec
filter context fsb with value length 50 low selectivity (kept 1/1024) 1.03 73.3±6.58µs ? ?/sec 1.00 71.4±5.25µs ? ?/sec
filter context i32 (kept 1/2) 1.01 13.5±0.80µs ? ?/sec 1.00 13.4±0.79µs ? ?/sec
filter context i32 high selectivity (kept 1023/1024) 1.00 3.7±0.01µs ? ?/sec 1.00 3.7±0.00µs ? ?/sec
filter context i32 low selectivity (kept 1/1024) 1.00 145.5±2.19ns ? ?/sec 1.40 203.6±4.16ns ? ?/sec
filter context i32 w NULLs (kept 1/2) 1.02 85.8±6.56µs ? ?/sec 1.00 83.9±5.19µs ? ?/sec
filter context i32 w NULLs high selectivity (kept 1023/1024) 1.00 5.5±0.01µs ? ?/sec 1.01 5.6±0.01µs ? ?/sec
filter context i32 w NULLs low selectivity (kept 1/1024) 1.00 328.3±14.30ns ? ?/sec 1.30 426.4±9.55ns ? ?/sec
filter context mixed string view (kept 1/2) 1.00 92.4±5.29µs ? ?/sec 1.03 94.9±6.46µs ? ?/sec
filter context mixed string view high selectivity (kept 1023/1024) 1.00 21.1±0.20µs ? ?/sec 1.04 22.0±0.11µs ? ?/sec
filter context mixed string view low selectivity (kept 1/1024) 1.00 415.4±16.62ns ? ?/sec 1.44 597.7±11.42ns ? ?/sec
filter context short string view (kept 1/2) 1.00 93.9±5.36µs ? ?/sec 1.00 94.3±6.54µs ? ?/sec
filter context short string view high selectivity (kept 1023/1024) 1.09 22.7±0.15µs ? ?/sec 1.00 20.8±0.17µs ? ?/sec
filter context short string view low selectivity (kept 1/1024) 1.00 349.2±9.92ns ? ?/sec 1.36 474.1±11.37ns ? ?/sec
filter context string (kept 1/2) 1.01 421.9±5.27µs ? ?/sec 1.00 419.4±5.84µs ? ?/sec
filter context string dictionary (kept 1/2) 1.00 12.7±0.01µs ? ?/sec 1.14 14.5±0.30µs ? ?/sec
filter context string dictionary high selectivity (kept 1023/1024) 1.04 4.3±0.01µs ? ?/sec 1.00 4.1±0.01µs ? ?/sec
filter context string dictionary low selectivity (kept 1/1024) 1.00 520.1±0.86ns ? ?/sec 1.11 576.1±5.06ns ? ?/sec
filter context string dictionary w NULLs (kept 1/2) 1.04 87.9±6.53µs ? ?/sec 1.00 84.9±5.27µs ? ?/sec
filter context string dictionary w NULLs high selectivity (kept 1023/1024) 1.00 6.0±0.02µs ? ?/sec 1.00 6.0±0.02µs ? ?/sec
filter context string dictionary w NULLs low selectivity (kept 1/1024) 1.00 707.6±12.71ns ? ?/sec 1.14 805.1±9.35ns ? ?/sec
filter context string high selectivity (kept 1023/1024) 1.01 315.0±2.74µs ? ?/sec 1.00 313.0±2.44µs ? ?/sec
filter context string low selectivity (kept 1/1024) 1.00 758.2±10.14ns ? ?/sec 1.18 895.1±7.41ns ? ?/sec
filter context u8 (kept 1/2) 1.00 12.1±0.03µs ? ?/sec 1.01 12.1±0.03µs ? ?/sec
filter context u8 high selectivity (kept 1023/1024) 1.01 1098.3±5.06ns ? ?/sec 1.00 1090.6±3.43ns ? ?/sec
filter context u8 low selectivity (kept 1/1024) 1.00 128.5±1.37ns ? ?/sec 1.40 179.3±0.89ns ? ?/sec
filter context u8 w NULLs (kept 1/2) 1.03 87.1±6.65µs ? ?/sec 1.00 84.4±5.47µs ? ?/sec
filter context u8 w NULLs high selectivity (kept 1023/1024) 1.00 2.9±0.01µs ? ?/sec 1.03 3.0±0.01µs ? ?/sec
filter context u8 w NULLs low selectivity (kept 1/1024) 1.00 313.3±11.58ns ? ?/sec 1.28 399.7±10.34ns ? ?/sec
filter decimal128 (kept 1/2) 1.01 35.4±0.06µs ? ?/sec 1.00 35.0±0.07µs ? ?/sec
filter decimal128 high selectivity (kept 1023/1024) 1.04 19.8±0.44µs ? ?/sec 1.00 19.2±0.20µs ? ?/sec
filter decimal128 low selectivity (kept 1/1024) 1.00 1567.6±1.64ns ? ?/sec 1.03 1610.8±2.17ns ? ?/sec
filter f32 (kept 1/2) 1.00 106.6±0.41µs ? ?/sec 1.01 108.1±0.43µs ? ?/sec
filter fsb with value length 20 (kept 1/2) 1.03 80.2±0.05µs ? ?/sec 1.00 78.0±0.18µs ? ?/sec
filter fsb with value length 20 high selectivity (kept 1023/1024) 1.04 25.6±0.63µs ? ?/sec 1.00 24.6±0.87µs ? ?/sec
filter fsb with value length 20 low selectivity (kept 1/1024) 1.00 1620.4±2.48ns ? ?/sec 1.03 1665.2±4.96ns ? ?/sec
filter fsb with value length 5 (kept 1/2) 1.02 79.4±0.06µs ? ?/sec 1.00 78.0±0.19µs ? ?/sec
filter fsb with value length 5 high selectivity (kept 1023/1024) 1.02 6.2±0.05µs ? ?/sec 1.00 6.1±0.09µs ? ?/sec
filter fsb with value length 5 low selectivity (kept 1/1024) 1.00 1563.9±0.79ns ? ?/sec 1.03 1605.5±9.64ns ? ?/sec
filter fsb with value length 50 (kept 1/2) 1.06 128.5±0.56µs ? ?/sec 1.00 120.7±0.44µs ? ?/sec
filter fsb with value length 50 high selectivity (kept 1023/1024) 1.14 97.1±8.84µs ? ?/sec 1.00 85.1±6.79µs ? ?/sec
filter fsb with value length 50 low selectivity (kept 1/1024) 1.00 1626.5±2.14ns ? ?/sec 1.03 1678.6±3.49ns ? ?/sec
filter i32 (kept 1/2) 1.00 29.1±0.04µs ? ?/sec 1.01 29.4±0.05µs ? ?/sec
filter i32 high selectivity (kept 1023/1024) 1.00 4.8±0.07µs ? ?/sec 1.03 5.0±0.06µs ? ?/sec
filter i32 low selectivity (kept 1/1024) 1.00 1506.9±4.41ns ? ?/sec 1.04 1564.1±2.53ns ? ?/sec
filter optimize (kept 1/2) 1.00 28.0±0.02µs ? ?/sec 1.00 28.1±0.36µs ? ?/sec
filter optimize high selectivity (kept 1023/1024) 1.00 1330.7±1.19ns ? ?/sec 1.40 1866.6±1237.75ns ? ?/sec
filter optimize low selectivity (kept 1/1024) 1.00 1326.1±1.68ns ? ?/sec 1.00 1319.5±2.57ns ? ?/sec
filter run array (kept 1/2) 1.00 285.1±1.97µs ? ?/sec 1.00 284.5±2.85µs ? ?/sec
filter run array high selectivity (kept 1023/1024) 1.03 292.0±7.11µs ? ?/sec 1.00 284.7±5.38µs ? ?/sec
filter run array low selectivity (kept 1/1024) 1.00 236.5±1.01µs ? ?/sec 1.00 235.9±1.20µs ? ?/sec
filter single record batch 1.00 29.6±0.06µs ? ?/sec 1.00 29.6±0.09µs ? ?/sec
filter u8 (kept 1/2) 1.01 29.6±0.05µs ? ?/sec 1.00 29.3±0.09µs ? ?/sec
filter u8 high selectivity (kept 1023/1024) 1.00 2.2±0.03µs ? ?/sec 1.03 2.3±0.04µs ? ?/sec
filter u8 low selectivity (kept 1/1024) 1.00 1461.0±5.05ns ? ?/sec 1.04 1523.2±2.16ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 655.1s |
| Peak memory | 3.1 GiB |
| Avg memory | 3.0 GiB |
| CPU user | 651.8s |
| CPU sys | 0.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 675.2s |
| Peak memory | 3.0 GiB |
| Avg memory | 3.0 GiB |
| CPU user | 671.4s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4529771683
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9986#issuecomment-4529714415)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group issue-9298-repalce-array-data-arrow-select main
----- ------------------------------------------ ----
filter context decimal128 (kept 1/2) 1.07 21.2±0.09µs ? ?/sec 1.00 19.7±0.09µs ? ?/sec
filter context decimal128 high selectivity (kept 1023/1024) 1.02 19.6±0.11µs ? ?/sec 1.00 19.3±0.21µs ? ?/sec
filter context decimal128 low selectivity (kept 1/1024) 1.00 148.3±0.87ns ? ?/sec 1.32 195.8±0.85ns ? ?/sec
filter context f32 (kept 1/2) 1.00 84.9±5.46µs ? ?/sec 1.01 85.5±6.69µs ? ?/sec
filter context f32 high selectivity (kept 1023/1024) 1.00 5.5±0.01µs ? ?/sec 1.02 5.6±0.01µs ? ?/sec
filter context f32 low selectivity (kept 1/1024) 1.00 322.0±12.05ns ? ?/sec 1.34 431.0±13.40ns ? ?/sec
filter context fsb with value length 20 (kept 1/2) 1.00 71.2±5.11µs ? ?/sec 1.03 73.3±6.47µs ? ?/sec
filter context fsb with value length 20 high selectivity (kept 1023/1024) 1.00 71.3±5.25µs ? ?/sec 1.03 73.3±6.50µs ? ?/sec
filter context fsb with value length 20 low selectivity (kept 1/1024) 1.00 71.2±5.12µs ? ?/sec 1.03 73.4±6.43µs ? ?/sec
filter context fsb with value length 5 (kept 1/2) 1.00 71.4±5.26µs ? ?/sec 1.03 73.5±6.45µs ? ?/sec
filter context fsb with value length 5 high selectivity (kept 1023/1024) 1.00 71.3±5.33µs ? ?/sec 1.03 73.4±6.46µs ? ?/sec
filter context fsb with value length 5 low selectivity (kept 1/1024) 1.00 71.2±5.11µs ? ?/sec 1.03 73.4±6.43µs ? ?/sec
filter context fsb with value length 50 (kept 1/2) 1.00 71.2±5.12µs ? ?/sec 1.03 73.2±6.50µs ? ?/sec
filter context fsb with value length 50 high selectivity (kept 1023/1024) 1.00 71.1±5.30µs ? ?/sec 1.03 73.4±6.47µs ? ?/sec
filter context fsb with value length 50 low selectivity (kept 1/1024) 1.00 71.3±5.28µs ? ?/sec 1.03 73.4±6.46µs ? ?/sec
filter context i32 (kept 1/2) 1.00 12.3±0.02µs ? ?/sec 1.15 14.2±0.10µs ? ?/sec
filter context i32 high selectivity (kept 1023/1024) 1.00 3.7±0.00µs ? ?/sec 1.01 3.7±0.00µs ? ?/sec
filter context i32 low selectivity (kept 1/1024) 1.00 144.2±1.86ns ? ?/sec 1.39 200.3±0.67ns ? ?/sec
filter context i32 w NULLs (kept 1/2) 1.00 83.8±5.23µs ? ?/sec 1.03 86.0±6.54µs ? ?/sec
filter context i32 w NULLs high selectivity (kept 1023/1024) 1.00 5.5±0.01µs ? ?/sec 1.02 5.6±0.01µs ? ?/sec
filter context i32 w NULLs low selectivity (kept 1/1024) 1.00 321.8±11.13ns ? ?/sec 1.34 431.4±14.15ns ? ?/sec
filter context mixed string view (kept 1/2) 1.01 93.4±6.59µs ? ?/sec 1.00 92.2±5.47µs ? ?/sec
filter context mixed string view high selectivity (kept 1023/1024) 1.02 21.1±0.16µs ? ?/sec 1.00 20.7±0.14µs ? ?/sec
filter context mixed string view low selectivity (kept 1/1024) 1.00 417.4±18.71ns ? ?/sec 1.43 597.5±9.49ns ? ?/sec
filter context short string view (kept 1/2) 1.01 93.6±6.61µs ? ?/sec 1.00 92.2±5.49µs ? ?/sec
filter context short string view high selectivity (kept 1023/1024) 1.01 21.7±0.28µs ? ?/sec 1.00 21.5±0.13µs ? ?/sec
filter context short string view low selectivity (kept 1/1024) 1.00 353.5±13.00ns ? ?/sec 1.33 468.9±9.61ns ? ?/sec
filter context string (kept 1/2) 1.01 435.0±6.93µs ? ?/sec 1.00 429.7±7.94µs ? ?/sec
filter context string dictionary (kept 1/2) 1.00 12.7±0.03µs ? ?/sec 1.03 13.1±0.64µs ? ?/sec
filter context string dictionary high selectivity (kept 1023/1024) 1.00 4.1±0.01µs ? ?/sec 1.00 4.1±0.00µs ? ?/sec
filter context string dictionary low selectivity (kept 1/1024) 1.00 518.8±0.93ns ? ?/sec 1.11 575.7±1.81ns ? ?/sec
filter context string dictionary w NULLs (kept 1/2) 1.00 85.2±5.36µs ? ?/sec 1.03 87.7±6.60µs ? ?/sec
filter context string dictionary w NULLs high selectivity (kept 1023/1024) 1.00 6.1±0.01µs ? ?/sec 1.00 6.0±0.01µs ? ?/sec
filter context string dictionary w NULLs low selectivity (kept 1/1024) 1.00 713.8±10.43ns ? ?/sec 1.14 815.3±13.85ns ? ?/sec
filter context string high selectivity (kept 1023/1024) 1.08 349.8±16.35µs ? ?/sec 1.00 324.5±8.23µs ? ?/sec
filter context string low selectivity (kept 1/1024) 1.00 764.1±12.07ns ? ?/sec 1.16 884.8±11.06ns ? ?/sec
filter context u8 (kept 1/2) 1.00 12.1±0.01µs ? ?/sec 1.16 14.1±0.23µs ? ?/sec
filter context u8 high selectivity (kept 1023/1024) 1.00 1098.1±3.35ns ? ?/sec 1.00 1099.9±4.25ns ? ?/sec
filter context u8 low selectivity (kept 1/1024) 1.00 128.0±0.85ns ? ?/sec 1.42 181.7±0.95ns ? ?/sec
filter context u8 w NULLs (kept 1/2) 1.00 83.9±5.49µs ? ?/sec 1.02 85.6±6.51µs ? ?/sec
filter context u8 w NULLs high selectivity (kept 1023/1024) 1.00 2.9±0.01µs ? ?/sec 1.03 2.9±0.01µs ? ?/sec
filter context u8 w NULLs low selectivity (kept 1/1024) 1.00 311.3±10.19ns ? ?/sec 1.31 407.2±12.79ns ? ?/sec
filter decimal128 (kept 1/2) 1.00 35.6±0.07µs ? ?/sec 1.00 35.7±0.07µs ? ?/sec
filter decimal128 high selectivity (kept 1023/1024) 1.06 19.5±0.15µs ? ?/sec 1.00 18.5±0.08µs ? ?/sec
filter decimal128 low selectivity (kept 1/1024) 1.00 1568.4±1.55ns ? ?/sec 1.03 1614.4±2.06ns ? ?/sec
filter f32 (kept 1/2) 1.02 108.0±1.17µs ? ?/sec 1.00 105.9±0.50µs ? ?/sec
filter fsb with value length 20 (kept 1/2) 1.03 80.1±0.11µs ? ?/sec 1.00 78.1±0.23µs ? ?/sec
filter fsb with value length 20 high selectivity (kept 1023/1024) 1.01 24.8±0.65µs ? ?/sec 1.00 24.5±0.79µs ? ?/sec
filter fsb with value length 20 low selectivity (kept 1/1024) 1.00 1621.4±1.82ns ? ?/sec 1.03 1670.3±5.51ns ? ?/sec
filter fsb with value length 5 (kept 1/2) 1.02 79.5±0.14µs ? ?/sec 1.00 78.2±0.28µs ? ?/sec
filter fsb with value length 5 high selectivity (kept 1023/1024) 1.00 6.0±0.06µs ? ?/sec 1.04 6.2±0.09µs ? ?/sec
filter fsb with value length 5 low selectivity (kept 1/1024) 1.00 1567.2±1.70ns ? ?/sec 1.03 1619.3±13.89ns ? ?/sec
filter fsb with value length 50 (kept 1/2) 1.00 119.4±0.70µs ? ?/sec 1.01 120.5±0.50µs ? ?/sec
filter fsb with value length 50 high selectivity (kept 1023/1024) 1.00 86.3±6.28µs ? ?/sec 1.01 87.3±7.07µs ? ?/sec
filter fsb with value length 50 low selectivity (kept 1/1024) 1.00 1629.1±3.29ns ? ?/sec 1.03 1672.2±5.19ns ? ?/sec
filter i32 (kept 1/2) 1.00 29.4±0.06µs ? ?/sec 1.01 29.7±0.05µs ? ?/sec
filter i32 high selectivity (kept 1023/1024) 1.00 4.9±0.07µs ? ?/sec 1.00 4.9±0.06µs ? ?/sec
filter i32 low selectivity (kept 1/1024) 1.01 1581.3±5.56ns ? ?/sec 1.00 1564.1±3.16ns ? ?/sec
filter optimize (kept 1/2) 1.00 28.2±0.12µs ? ?/sec 1.05 29.8±0.05µs ? ?/sec
filter optimize high selectivity (kept 1023/1024) 1.00 1328.2±1.89ns ? ?/sec 1.00 1326.9±3.00ns ? ?/sec
filter optimize low selectivity (kept 1/1024) 1.00 1325.5±1.80ns ? ?/sec 1.01 1332.4±4.45ns ? ?/sec
filter run array (kept 1/2) 1.00 287.9±3.19µs ? ?/sec 1.00 288.0±3.34µs ? ?/sec
filter run array high selectivity (kept 1023/1024) 1.02 295.8±2.85µs ? ?/sec 1.00 290.9±9.44µs ? ?/sec
filter run array low selectivity (kept 1/1024) 1.00 236.2±0.81µs ? ?/sec 1.00 236.1±0.87µs ? ?/sec
filter single record batch 1.00 29.5±0.03µs ? ?/sec 1.00 29.4±0.03µs ? ?/sec
filter u8 (kept 1/2) 1.01 29.4±0.03µs ? ?/sec 1.00 29.1±0.06µs ? ?/sec
filter u8 high selectivity (kept 1023/1024) 1.00 2.2±0.04µs ? ?/sec 1.03 2.3±0.03µs ? ?/sec
filter u8 low selectivity (kept 1/1024) 1.00 1464.3±4.83ns ? ?/sec 1.05 1533.0±2.75ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 670.2s |
| Peak memory | 3.1 GiB |
| Avg memory | 3.0 GiB |
| CPU user | 666.6s |
| CPU sys | 0.9s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 665.2s |
| Peak memory | 3.0 GiB |
| Avg memory | 3.0 GiB |
| CPU user | 661.4s |
| CPU sys | 0.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "liamzwbao (via GitHub)" <gi...@apache.org>.
liamzwbao commented on code in PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#discussion_r3295115446
##########
arrow-select/src/filter.rs:
##########
@@ -843,17 +837,11 @@ fn filter_byte_view<T: ByteViewType>(
predicate: &FilterPredicate,
) -> GenericByteViewArray<T> {
let new_view_buffer = filter_native(array.views(), predicate);
+ let views = ScalarBuffer::new(new_view_buffer, 0, predicate.count);
Review Comment:
I looked into `ScalarBuffer::new_unchecked`, the docstring says "Only use for testing purpose," and the only caller is an FFI test that needs to bypass the alignment assertion to repro a dangling-pointer case. Consider the check is relatively cheap, maybe we could leave it as-is unless we see it show up in a benchmark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501463694
run benchmark filter_kernels
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501755955
🤔 hmm that run https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501666195 looks very good. Will run another
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [arrow-select] Replace `ArrayData` with direct `Array` construction in filter kernels [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9986:
URL: https://github.com/apache/arrow-rs/pull/9986#issuecomment-4501756916
run benchmark filter_kernels
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org