You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "coderfender (via GitHub)" <gi...@apache.org> on 2026/04/11 23:30:35 UTC
[PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
coderfender opened a new pull request, #21561:
URL: https://github.com/apache/datafusion/pull/21561
## Which issue does this PR close?
Evaluate perf with group accumulators for count distinct
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4262974208
run benchmark clickbench_partitioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3098481887
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ ArrayRef, AsArray, BooleanArray, Int64Array, ListArray, PrimitiveArray,
+};
+use arrow::buffer::OffsetBuffer;
+use arrow::datatypes::{ArrowPrimitiveType, Field};
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+use crate::aggregate::groups_accumulator::accumulate::accumulate;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ seen: HashSet<(usize, T::Native), RandomState>,
+ counts: Vec<i64>,
+}
+
+impl<T: ArrowPrimitiveType> PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ pub fn new() -> Self {
+ Self {
+ seen: HashSet::default(),
+ counts: Vec::new(),
+ }
+ }
+}
+
+impl<T: ArrowPrimitiveType> Default for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ArrowPrimitiveType + Send + std::fmt::Debug> GroupsAccumulator
+ for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn update_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> datafusion_common::Result<()> {
+ debug_assert_eq!(values.len(), 1);
+ self.counts.resize(total_num_groups, 0);
+ let arr = values[0].as_primitive::<T>();
+ accumulate(group_indices, arr, opt_filter, |group_idx, value| {
+ if self.seen.insert((group_idx, value)) {
+ self.counts[group_idx] += 1;
+ }
+ });
+ Ok(())
+ }
+
+ fn evaluate(&mut self, emit_to: EmitTo) -> datafusion_common::Result<ArrayRef> {
+ let counts = emit_to.take_needed(&mut self.counts);
+
+ match emit_to {
+ EmitTo::All => {
+ self.seen.clear();
+ }
+ EmitTo::First(n) => {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx >= n {
+ remaining.insert((group_idx - n, value));
+ }
+ }
+ self.seen = remaining;
+ }
+ }
+
+ Ok(Arc::new(Int64Array::from(counts)))
+ }
+
+ fn state(&mut self, emit_to: EmitTo) -> datafusion_common::Result<Vec<ArrayRef>> {
+ let num_emitted = match emit_to {
+ EmitTo::All => self.counts.len(),
+ EmitTo::First(n) => n,
+ };
+
+ let mut group_values: Vec<Vec<T::Native>> = vec![Vec::new(); num_emitted];
+
+ if matches!(emit_to, EmitTo::All) {
+ for (group_idx, value) in self.seen.drain() {
+ group_values[group_idx].push(value);
+ }
+ self.counts.clear();
+ } else {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx < num_emitted {
+ group_values[group_idx].push(value);
+ } else {
+ remaining.insert((group_idx - num_emitted, value));
+ }
+ }
+ self.seen = remaining;
+ let _ = emit_to.take_needed(&mut self.counts);
+ }
+
+ let mut offsets = vec![0i32];
+ let mut all_values = Vec::new();
+ for values in &group_values {
+ all_values.extend(values.iter().copied());
+ offsets.push(all_values.len() as i32);
+ }
+
+ let values_array = Arc::new(PrimitiveArray::<T>::from_iter_values(all_values));
Review Comment:
```suggestion
let values_array = Arc::new(PrimitiveArray::<T>::from_iter_values(all_values));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4241914660
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4241897983)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4241897983-1202-tn5gl 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (8d60834b3b09bad5093b74d0a32799e3dc98dd0a) to a13c23d (merge-base) [diff](https://github.com/apache/datafusion/compare/a13c23df67c218e45342b80238cae490609d8f3d..8d60834b3b09bad5093b74d0a32799e3dc98dd0a) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3077686167
##########
datafusion/functions-aggregate/src/count.rs:
##########
@@ -336,18 +337,70 @@ impl AggregateUDFImpl for Count {
}
fn groups_accumulator_supported(&self, args: AccumulatorArgs) -> bool {
- // groups accumulator only supports `COUNT(c1)`, not
- // `COUNT(c1, c2)`, etc
- if args.is_distinct {
+ if args.exprs.len() != 1 {
return false;
}
- args.exprs.len() == 1
+ if args.is_distinct {
+ // Only support primitive integer types for now
+ matches!(
+ args.expr_fields[0].data_type(),
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::UInt8
+ | DataType::UInt16
+ | DataType::UInt32
+ | DataType::UInt64
+ )
+ } else {
+ true
+ }
}
fn create_groups_accumulator(
&self,
- _args: AccumulatorArgs,
+ args: AccumulatorArgs,
) -> Result<Box<dyn GroupsAccumulator>> {
+ if args.is_distinct {
+ let data_type = args.expr_fields[0].data_type();
+ return match data_type {
+ DataType::Int8 => Ok(Box::new(
Review Comment:
Perhaps we can avoid using a hash set as well here like https://github.com/apache/datafusion/pulls?q=is%3Apr+is%3Aopen+hashset ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4231185703
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4231151449)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 6.56 / 7.04 ยฑ0.81 / 8.66 ms โ 6.75 / 7.24 ยฑ0.73 / 8.68 ms โ no change โ
โ QQuery 2 โ 146.79 / 147.78 ยฑ0.80 / 148.83 ms โ 145.30 / 146.68 ยฑ1.17 / 148.38 ms โ no change โ
โ QQuery 3 โ 112.83 / 113.80 ยฑ0.64 / 114.70 ms โ 114.32 / 115.24 ยฑ0.88 / 116.68 ms โ no change โ
โ QQuery 4 โ 1355.07 / 1386.38 ยฑ22.59 / 1418.44 ms โ 1387.67 / 1407.95 ยฑ14.22 / 1427.10 ms โ no change โ
โ QQuery 5 โ 171.43 / 172.73 ยฑ1.42 / 175.49 ms โ 173.06 / 175.02 ยฑ2.22 / 178.69 ms โ no change โ
โ QQuery 6 โ 822.98 / 861.15 ยฑ24.96 / 892.09 ms โ 855.97 / 879.77 ยฑ17.26 / 902.44 ms โ no change โ
โ QQuery 7 โ 336.33 / 344.51 ยฑ4.31 / 348.02 ms โ 343.05 / 345.13 ยฑ2.14 / 348.89 ms โ no change โ
โ QQuery 8 โ 115.20 / 116.50 ยฑ1.08 / 118.21 ms โ 118.51 / 119.15 ยฑ0.50 / 119.82 ms โ no change โ
โ QQuery 9 โ 101.46 / 110.46 ยฑ9.91 / 129.78 ms โ 100.83 / 103.72 ยฑ3.07 / 107.51 ms โ +1.06x faster โ
โ QQuery 10 โ 107.15 / 108.22 ยฑ0.81 / 109.14 ms โ 106.52 / 108.09 ยฑ1.01 / 109.41 ms โ no change โ
โ QQuery 11 โ 963.92 / 976.73 ยฑ10.67 / 995.75 ms โ 982.23 / 992.76 ยฑ8.92 / 1005.70 ms โ no change โ
โ QQuery 12 โ 44.90 / 46.66 ยฑ1.40 / 48.80 ms โ 46.89 / 47.49 ยฑ0.56 / 48.51 ms โ no change โ
โ QQuery 13 โ 402.02 / 405.58 ยฑ2.08 / 407.31 ms โ 402.47 / 404.05 ยฑ1.32 / 406.28 ms โ no change โ
โ QQuery 14 โ 1018.49 / 1020.69 ยฑ2.09 / 1024.38 ms โ 1003.19 / 1012.03 ยฑ6.41 / 1020.19 ms โ no change โ
โ QQuery 15 โ 15.95 / 16.64 ยฑ0.56 / 17.40 ms โ 16.40 / 17.23 ยฑ0.81 / 18.62 ms โ no change โ
โ QQuery 16 โ 7.20 / 7.54 ยฑ0.25 / 7.87 ms โ 7.66 / 8.22 ยฑ0.53 / 9.20 ms โ 1.09x slower โ
โ QQuery 17 โ 229.34 / 247.33 ยฑ34.46 / 316.25 ms โ 226.53 / 229.77 ยฑ3.28 / 235.79 ms โ +1.08x faster โ
โ QQuery 18 โ 126.95 / 129.19 ยฑ1.47 / 131.24 ms โ 126.98 / 128.99 ยฑ1.21 / 130.34 ms โ no change โ
โ QQuery 19 โ 154.37 / 157.10 ยฑ1.67 / 158.90 ms โ 156.33 / 158.27 ยฑ1.72 / 161.28 ms โ no change โ
โ QQuery 20 โ 14.11 / 14.64 ยฑ0.39 / 15.30 ms โ 14.05 / 14.66 ยฑ0.37 / 15.10 ms โ no change โ
โ QQuery 21 โ 19.34 / 19.69 ยฑ0.27 / 20.04 ms โ 19.88 / 20.40 ยฑ0.34 / 20.88 ms โ no change โ
โ QQuery 22 โ 486.43 / 491.90 ยฑ3.12 / 495.01 ms โ 489.28 / 495.17 ยฑ4.93 / 501.65 ms โ no change โ
โ QQuery 23 โ 890.26 / 901.68 ยฑ7.33 / 910.76 ms โ 905.74 / 918.25 ยฑ10.42 / 933.82 ms โ no change โ
โ QQuery 24 โ 385.36 / 386.52 ยฑ0.84 / 387.33 ms โ 391.48 / 392.49 ยฑ1.08 / 394.38 ms โ no change โ
โ QQuery 25 โ 341.38 / 344.73 ยฑ2.30 / 348.60 ms โ 347.63 / 350.84 ยฑ2.54 / 354.31 ms โ no change โ
โ QQuery 26 โ 80.94 / 81.96 ยฑ0.70 / 83.07 ms โ 81.84 / 84.34 ยฑ2.04 / 87.71 ms โ no change โ
โ QQuery 27 โ 7.39 / 8.00 ยฑ0.70 / 9.33 ms โ 7.34 / 8.02 ยฑ0.67 / 9.06 ms โ no change โ
โ QQuery 28 โ 147.68 / 149.53 ยฑ1.24 / 151.11 ms โ 152.99 / 154.48 ยฑ0.94 / 155.83 ms โ no change โ
โ QQuery 29 โ 282.96 / 284.16 ยฑ0.89 / 285.42 ms โ 287.39 / 291.18 ยฑ2.42 / 294.05 ms โ no change โ
โ QQuery 30 โ 43.79 / 46.13 ยฑ1.77 / 48.27 ms โ 44.52 / 46.55 ยฑ2.30 / 50.98 ms โ no change โ
โ QQuery 31 โ 173.05 / 174.28 ยฑ1.25 / 176.62 ms โ 172.16 / 174.64 ยฑ1.25 / 175.40 ms โ no change โ
โ QQuery 32 โ 57.09 / 58.61 ยฑ0.86 / 59.67 ms โ 58.03 / 58.87 ยฑ1.03 / 60.89 ms โ no change โ
โ QQuery 33 โ 140.94 / 141.80 ยฑ0.82 / 143.04 ms โ 140.84 / 143.32 ยฑ1.59 / 145.86 ms โ no change โ
โ QQuery 34 โ 6.94 / 7.11 ยฑ0.23 / 7.57 ms โ 7.29 / 7.50 ยฑ0.22 / 7.90 ms โ 1.06x slower โ
โ QQuery 35 โ 109.60 / 110.36 ยฑ0.97 / 112.18 ms โ 108.19 / 109.71 ยฑ0.88 / 110.93 ms โ no change โ
โ QQuery 36 โ 6.73 / 6.91 ยฑ0.19 / 7.22 ms โ 6.81 / 7.03 ยฑ0.14 / 7.23 ms โ no change โ
โ QQuery 37 โ 8.54 / 8.85 ยฑ0.28 / 9.29 ms โ 8.69 / 9.22 ยฑ0.55 / 10.21 ms โ no change โ
โ QQuery 38 โ 83.80 / 87.89 ยฑ4.64 / 95.78 ms โ 86.78 / 90.82 ยฑ2.86 / 95.20 ms โ no change โ
โ QQuery 39 โ 126.74 / 129.75 ยฑ1.91 / 132.09 ms โ 130.69 / 132.81 ยฑ1.20 / 134.36 ms โ no change โ
โ QQuery 40 โ 109.11 / 115.65 ยฑ8.54 / 132.38 ms โ 110.99 / 118.36 ยฑ7.56 / 132.66 ms โ no change โ
โ QQuery 41 โ 14.74 / 16.11 ยฑ0.92 / 16.98 ms โ 14.72 / 15.91 ยฑ0.81 / 17.09 ms โ no change โ
โ QQuery 42 โ 107.00 / 109.18 ยฑ1.35 / 110.73 ms โ 109.50 / 111.60 ยฑ1.10 / 112.60 ms โ no change โ
โ QQuery 43 โ 6.02 / 6.16 ยฑ0.14 / 6.42 ms โ 6.19 / 6.81 ยฑ0.74 / 8.23 ms โ 1.11x slower โ
โ QQuery 44 โ 11.86 / 12.01 ยฑ0.09 / 12.15 ms โ 11.83 / 12.06 ยฑ0.18 / 12.38 ms โ no change โ
โ QQuery 45 โ 50.30 / 51.50 ยฑ1.12 / 53.40 ms โ 51.51 / 52.30 ยฑ0.45 / 52.75 ms โ no change โ
โ QQuery 46 โ 8.56 / 9.40 ยฑ1.01 / 10.97 ms โ 9.00 / 9.33 ยฑ0.28 / 9.67 ms โ no change โ
โ QQuery 47 โ 768.50 / 773.33 ยฑ2.80 / 777.07 ms โ 752.92 / 775.27 ยฑ19.36 / 800.53 ms โ no change โ
โ QQuery 48 โ 293.06 / 297.40 ยฑ2.65 / 300.94 ms โ 287.55 / 292.91 ยฑ4.07 / 297.77 ms โ no change โ
โ QQuery 49 โ 253.31 / 256.07 ยฑ2.92 / 260.77 ms โ 250.51 / 253.13 ยฑ1.95 / 255.39 ms โ no change โ
โ QQuery 50 โ 222.76 / 230.07 ยฑ4.14 / 234.72 ms โ 222.90 / 227.82 ยฑ5.25 / 235.67 ms โ no change โ
โ QQuery 51 โ 179.81 / 183.93 ยฑ4.16 / 191.83 ms โ 182.58 / 184.11 ยฑ1.73 / 187.33 ms โ no change โ
โ QQuery 52 โ 107.73 / 109.29 ยฑ1.45 / 112.01 ms โ 108.63 / 109.37 ยฑ0.54 / 110.01 ms โ no change โ
โ QQuery 53 โ 102.19 / 102.78 ยฑ0.49 / 103.41 ms โ 103.18 / 104.74 ยฑ1.18 / 106.24 ms โ no change โ
โ QQuery 54 โ 147.94 / 149.30 ยฑ1.45 / 152.08 ms โ 148.08 / 149.37 ยฑ1.28 / 151.10 ms โ no change โ
โ QQuery 55 โ 107.05 / 108.83 ยฑ1.65 / 111.72 ms โ 108.14 / 108.85 ยฑ0.48 / 109.46 ms โ no change โ
โ QQuery 56 โ 141.63 / 143.61 ยฑ1.64 / 145.88 ms โ 141.42 / 144.51 ยฑ1.73 / 146.48 ms โ no change โ
โ QQuery 57 โ 177.53 / 179.24 ยฑ1.83 / 182.61 ms โ 177.62 / 179.04 ยฑ1.01 / 180.34 ms โ no change โ
โ QQuery 58 โ 290.03 / 305.28 ยฑ11.56 / 324.22 ms โ 305.71 / 308.03 ยฑ3.06 / 313.94 ms โ no change โ
โ QQuery 59 โ 201.59 / 204.43 ยฑ2.33 / 208.66 ms โ 199.37 / 202.00 ยฑ1.96 / 204.23 ms โ no change โ
โ QQuery 60 โ 143.53 / 146.09 ยฑ1.72 / 148.19 ms โ 143.10 / 145.99 ยฑ1.67 / 148.14 ms โ no change โ
โ QQuery 61 โ 13.41 / 13.61 ยฑ0.23 / 14.05 ms โ 13.40 / 13.59 ยฑ0.13 / 13.79 ms โ no change โ
โ QQuery 62 โ 903.50 / 953.44 ยฑ49.24 / 1047.66 ms โ 914.91 / 927.28 ยฑ12.48 / 950.56 ms โ no change โ
โ QQuery 63 โ 102.20 / 104.71 ยฑ1.95 / 107.82 ms โ 103.30 / 105.81 ยฑ2.47 / 109.89 ms โ no change โ
โ QQuery 64 โ 705.67 / 712.59 ยฑ4.83 / 720.36 ms โ 683.32 / 688.30 ยฑ3.81 / 693.97 ms โ no change โ
โ QQuery 65 โ 267.45 / 273.67 ยฑ4.93 / 279.94 ms โ 252.66 / 259.09 ยฑ3.85 / 263.97 ms โ +1.06x faster โ
โ QQuery 66 โ 259.19 / 268.79 ยฑ7.35 / 281.14 ms โ 246.04 / 254.69 ยฑ5.76 / 262.51 ms โ +1.06x faster โ
โ QQuery 67 โ 327.73 / 332.44 ยฑ5.09 / 342.22 ms โ 313.20 / 323.17 ยฑ7.81 / 336.86 ms โ no change โ
โ QQuery 68 โ 9.24 / 10.87 ยฑ2.07 / 14.67 ms โ 9.22 / 10.46 ยฑ0.96 / 11.85 ms โ no change โ
โ QQuery 69 โ 105.20 / 107.13 ยฑ2.23 / 110.42 ms โ 103.26 / 104.67 ยฑ1.03 / 105.87 ms โ no change โ
โ QQuery 70 โ 348.44 / 356.46 ยฑ8.54 / 371.33 ms โ 337.04 / 349.46 ยฑ10.31 / 367.83 ms โ no change โ
โ QQuery 71 โ 137.25 / 140.12 ยฑ2.22 / 142.78 ms โ 135.23 / 139.57 ยฑ4.90 / 148.72 ms โ no change โ
โ QQuery 72 โ 623.65 / 639.42 ยฑ12.11 / 659.85 ms โ 613.78 / 624.57 ยฑ8.53 / 637.95 ms โ no change โ
โ QQuery 73 โ 7.63 / 8.96 ยฑ1.58 / 11.99 ms โ 7.66 / 8.64 ยฑ0.83 / 9.89 ms โ no change โ
โ QQuery 74 โ 656.28 / 660.71 ยฑ4.14 / 667.54 ms โ 613.47 / 618.70 ยฑ3.59 / 623.77 ms โ +1.07x faster โ
โ QQuery 75 โ 277.46 / 281.99 ยฑ3.21 / 286.41 ms โ 275.09 / 278.62 ยฑ3.47 / 284.19 ms โ no change โ
โ QQuery 76 โ 133.88 / 135.43 ยฑ1.20 / 137.16 ms โ 131.75 / 133.00 ยฑ1.04 / 134.15 ms โ no change โ
โ QQuery 77 โ 189.77 / 191.55 ยฑ1.07 / 192.67 ms โ 188.97 / 190.48 ยฑ1.55 / 192.93 ms โ no change โ
โ QQuery 78 โ 349.76 / 353.37 ยฑ2.26 / 356.73 ms โ 340.58 / 343.64 ยฑ2.40 / 346.68 ms โ no change โ
โ QQuery 79 โ 244.40 / 250.25 ยฑ3.09 / 253.65 ms โ 236.43 / 240.86 ยฑ3.40 / 245.58 ms โ no change โ
โ QQuery 80 โ 320.86 / 324.67 ยฑ2.69 / 328.74 ms โ 320.20 / 321.85 ยฑ1.09 / 323.32 ms โ no change โ
โ QQuery 81 โ 26.94 / 28.46 ยฑ1.73 / 31.70 ms โ 26.34 / 27.17 ยฑ0.80 / 28.21 ms โ no change โ
โ QQuery 82 โ 197.05 / 200.03 ยฑ2.61 / 204.23 ms โ 198.51 / 201.49 ยฑ2.08 / 203.59 ms โ no change โ
โ QQuery 83 โ 39.34 / 40.54 ยฑ1.04 / 42.09 ms โ 38.29 / 40.61 ยฑ1.22 / 41.59 ms โ no change โ
โ QQuery 84 โ 49.78 / 50.49 ยฑ0.40 / 51.01 ms โ 48.06 / 49.10 ยฑ0.79 / 50.41 ms โ no change โ
โ QQuery 85 โ 148.88 / 151.06 ยฑ1.42 / 153.22 ms โ 146.31 / 148.69 ยฑ1.41 / 150.57 ms โ no change โ
โ QQuery 86 โ 40.10 / 40.54 ยฑ0.46 / 41.27 ms โ 39.31 / 40.77 ยฑ0.98 / 42.31 ms โ no change โ
โ QQuery 87 โ 83.85 / 89.44 ยฑ3.64 / 95.11 ms โ 87.71 / 92.19 ยฑ3.33 / 96.11 ms โ no change โ
โ QQuery 88 โ 101.60 / 103.05 ยฑ1.18 / 104.85 ms โ 101.14 / 102.34 ยฑ0.82 / 103.52 ms โ no change โ
โ QQuery 89 โ 119.34 / 121.14 ยฑ1.24 / 123.05 ms โ 119.26 / 120.85 ยฑ1.38 / 122.75 ms โ no change โ
โ QQuery 90 โ 23.87 / 24.33 ยฑ0.35 / 24.70 ms โ 23.74 / 24.48 ยฑ0.73 / 25.71 ms โ no change โ
โ QQuery 91 โ 63.81 / 64.62 ยฑ0.70 / 65.92 ms โ 63.43 / 64.76 ยฑ0.97 / 66.28 ms โ no change โ
โ QQuery 92 โ 58.08 / 59.78 ยฑ1.20 / 60.84 ms โ 57.97 / 59.04 ยฑ0.72 / 60.04 ms โ no change โ
โ QQuery 93 โ 190.00 / 191.89 ยฑ1.35 / 193.80 ms โ 187.83 / 188.57 ยฑ0.72 / 189.83 ms โ no change โ
โ QQuery 94 โ 62.48 / 62.95 ยฑ0.37 / 63.46 ms โ 61.97 / 62.46 ยฑ0.39 / 63.09 ms โ no change โ
โ QQuery 95 โ 129.61 / 130.74 ยฑ1.06 / 132.73 ms โ 129.73 / 131.13 ยฑ1.48 / 133.76 ms โ no change โ
โ QQuery 96 โ 74.50 / 75.63 ยฑ0.83 / 76.32 ms โ 75.08 / 75.89 ยฑ0.82 / 77.45 ms โ no change โ
โ QQuery 97 โ 126.31 / 129.58 ยฑ1.75 / 131.47 ms โ 126.86 / 128.48 ยฑ1.37 / 130.20 ms โ no change โ
โ QQuery 98 โ 155.17 / 159.14 ยฑ2.16 / 161.08 ms โ 155.06 / 156.75 ยฑ1.81 / 159.68 ms โ no change โ
โ QQuery 99 โ 10825.07 / 10905.21 ยฑ44.63 / 10953.01 ms โ 10740.37 / 10800.10 ยฑ38.46 / 10842.77 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 32088.92ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 31879.92ms โ
โ Average Time (HEAD) โ 324.13ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 322.02ms โ
โ Queries Faster โ 5 โ
โ Queries Slower โ 3 โ
โ Queries with No Change โ 91 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.8s |
| Peak memory | 5.6 GiB |
| Avg memory | 4.6 GiB |
| CPU user | 266.1s |
| CPU sys | 17.6s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 159.7s |
| Peak memory | 5.6 GiB |
| Avg memory | 4.7 GiB |
| CPU user | 265.0s |
| CPU sys | 17.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4256808728
@Dandandan , there is a consistently higher memory usage with no proportional speedups.The current approach packs group and index into a single `u128` but the
> > @Dandandan , I am trying to a Vector of pairs (group ID , value) approach to see if SIMD (sort and return group counts only during `evaluate` and `state` would cost lesser than computing hashes in `update_batch` which is more frequent) . Results were promising on my local but I would be happy if you could run a benchmarks on GH runners for more accuracy. I also moved distinct accumulators to a separate cold path (which were probably causing other count queries' minor slowness)
> > Could you please trigger CI benchmarks to see if this is adding value over hashtable approach ?
>
> It seems to me the problem would be memory use (whenever it is non-unique)?
Yup ! It seems like the memory spikes are not worth the minor speed ups (perhaps even noise)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266033487
Thank you for the approval @Dandandan . but it doesn't seem like the benchmarks ran correctly. Could you please trigger it again and perhaps even toch tpcds and other relevant queries please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4293184486
Benchmark for [this request](https://github.com/apache/datafusion/pull/21561#issuecomment-4292778636) hit the 7200s job deadline before finishing.
Benchmarks requested: `tpch`
<details><summary>Kubernetes message</summary>
```
Job was active longer than specified deadline
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4292777695
Thanks @coderfender
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4251570677
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4251531266)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4251531266-1281-ptjf2 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (28c607aad4745f0b021b60bf42e1f3b7d49c0971) to 2818abb (merge-base) [diff](https://github.com/apache/datafusion/compare/2818abb8a10bf8b001669c15cf597f9ddfb86ed1..28c607aad4745f0b021b60bf42e1f3b7d49c0971) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4231155640
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4231151449)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4231151449-1104-k4vbw 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (ee0c8650aacfb92d852062a1c0bcfb5465c93482) to 16e578d (merge-base) [diff](https://github.com/apache/datafusion/compare/16e578dfdd28c8bffc6a21f54164b889e79bf75c..ee0c8650aacfb92d852062a1c0bcfb5465c93482) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266254412
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266245112)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4266245112-1411-dc9j2 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (617e79032b8930cd89a52ec0f27e761d5b71c5c5) to 5a427cb (merge-base) [diff](https://github.com/apache/datafusion/compare/5a427cb55d8630b51b9590a25b369b434611975e..617e79032b8930cd89a52ec0f27e761d5b71c5c5) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266345513
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266245112)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.22 / 4.52 ยฑ6.45 / 17.41 ms โ 1.22 / 4.53 ยฑ6.46 / 17.46 ms โ no change โ
โ QQuery 1 โ 14.28 / 14.72 ยฑ0.24 / 14.94 ms โ 14.37 / 14.88 ยฑ0.28 / 15.19 ms โ no change โ
โ QQuery 2 โ 43.97 / 44.33 ยฑ0.26 / 44.66 ms โ 43.99 / 44.97 ยฑ1.09 / 46.96 ms โ no change โ
โ QQuery 3 โ 41.28 / 41.96 ยฑ0.69 / 43.22 ms โ 40.31 / 41.19 ยฑ0.81 / 42.41 ms โ no change โ
โ QQuery 4 โ 288.85 / 300.45 ยฑ7.50 / 311.81 ms โ 291.50 / 297.36 ยฑ6.94 / 310.69 ms โ no change โ
โ QQuery 5 โ 346.81 / 351.54 ยฑ3.28 / 356.39 ms โ 348.49 / 353.87 ยฑ2.98 / 357.52 ms โ no change โ
โ QQuery 6 โ 5.64 / 9.22 ยฑ5.23 / 19.62 ms โ 5.67 / 6.78 ยฑ1.09 / 8.60 ms โ +1.36x faster โ
โ QQuery 7 โ 16.91 / 17.65 ยฑ1.16 / 19.96 ms โ 16.74 / 17.57 ยฑ0.88 / 19.26 ms โ no change โ
โ QQuery 8 โ 412.57 / 427.83 ยฑ14.92 / 453.59 ms โ 435.39 / 443.46 ยฑ6.02 / 453.70 ms โ no change โ
โ QQuery 9 โ 663.06 / 666.99 ยฑ4.45 / 674.48 ms โ 583.10 / 592.30 ยฑ5.29 / 598.80 ms โ +1.13x faster โ
โ QQuery 10 โ 91.33 / 94.31 ยฑ3.75 / 101.58 ms โ 91.55 / 93.52 ยฑ2.85 / 99.17 ms โ no change โ
โ QQuery 11 โ 103.78 / 104.83 ยฑ0.73 / 105.62 ms โ 104.35 / 105.07 ยฑ0.56 / 105.99 ms โ no change โ
โ QQuery 12 โ 349.30 / 352.39 ยฑ3.34 / 358.82 ms โ 343.44 / 354.87 ยฑ8.98 / 370.91 ms โ no change โ
โ QQuery 13 โ 451.40 / 465.29 ยฑ11.08 / 478.92 ms โ 461.70 / 486.69 ยฑ23.85 / 526.97 ms โ no change โ
โ QQuery 14 โ 348.55 / 351.79 ยฑ2.77 / 355.68 ms โ 350.89 / 354.81 ยฑ2.94 / 357.77 ms โ no change โ
โ QQuery 15 โ 355.13 / 363.95 ยฑ8.41 / 374.60 ms โ 367.01 / 398.46 ยฑ27.70 / 442.09 ms โ 1.09x slower โ
โ QQuery 16 โ 718.71 / 725.58 ยฑ5.14 / 733.10 ms โ 728.63 / 736.72 ยฑ6.27 / 745.83 ms โ no change โ
โ QQuery 17 โ 720.74 / 728.70 ยฑ4.71 / 733.72 ms โ 725.92 / 737.97 ยฑ6.49 / 744.77 ms โ no change โ
โ QQuery 18 โ 1448.55 / 1515.43 ยฑ45.89 / 1585.57 ms โ 1437.70 / 1497.07 ยฑ46.42 / 1561.89 ms โ no change โ
โ QQuery 19 โ 37.56 / 45.84 ยฑ12.96 / 71.64 ms โ 35.05 / 59.34 ยฑ30.11 / 114.56 ms โ 1.29x slower โ
โ QQuery 20 โ 716.67 / 738.26 ยฑ18.65 / 764.91 ms โ 717.89 / 742.93 ยฑ31.63 / 804.00 ms โ no change โ
โ QQuery 21 โ 761.37 / 764.95 ยฑ2.66 / 769.52 ms โ 771.50 / 774.99 ยฑ3.25 / 780.67 ms โ no change โ
โ QQuery 22 โ 1144.95 / 1149.97 ยฑ6.86 / 1163.42 ms โ 1154.91 / 1157.29 ยฑ1.62 / 1159.30 ms โ no change โ
โ QQuery 23 โ 3108.72 / 3137.42 ยฑ34.20 / 3197.49 ms โ 3139.28 / 3181.86 ยฑ30.69 / 3233.23 ms โ no change โ
โ QQuery 24 โ 100.72 / 104.92 ยฑ3.44 / 109.85 ms โ 101.15 / 103.08 ยฑ1.40 / 105.18 ms โ no change โ
โ QQuery 25 โ 139.35 / 140.50 ยฑ0.63 / 141.26 ms โ 138.45 / 142.21 ยฑ3.12 / 146.25 ms โ no change โ
โ QQuery 26 โ 99.39 / 101.58 ยฑ1.18 / 102.93 ms โ 99.75 / 103.99 ยฑ2.64 / 106.49 ms โ no change โ
โ QQuery 27 โ 852.63 / 857.74 ยฑ6.40 / 870.26 ms โ 852.53 / 869.14 ยฑ16.30 / 890.74 ms โ no change โ
โ QQuery 28 โ 3287.53 / 3301.89 ยฑ10.02 / 3317.03 ms โ 3245.44 / 3269.27 ยฑ12.92 / 3283.76 ms โ no change โ
โ QQuery 29 โ 50.82 / 54.79 ยฑ4.34 / 61.77 ms โ 51.11 / 57.03 ยฑ6.46 / 68.88 ms โ no change โ
โ QQuery 30 โ 362.92 / 373.95 ยฑ6.42 / 382.07 ms โ 355.40 / 367.63 ยฑ6.77 / 374.51 ms โ no change โ
โ QQuery 31 โ 367.94 / 382.55 ยฑ8.95 / 394.68 ms โ 379.53 / 392.14 ยฑ11.39 / 413.30 ms โ no change โ
โ QQuery 32 โ 1037.65 / 1060.61 ยฑ22.83 / 1104.11 ms โ 1166.88 / 1221.85 ยฑ48.94 / 1306.17 ms โ 1.15x slower โ
โ QQuery 33 โ 1476.74 / 1498.72 ยฑ15.62 / 1520.51 ms โ 1523.30 / 1615.78 ยฑ61.39 / 1688.76 ms โ 1.08x slower โ
โ QQuery 34 โ 1507.46 / 1538.47 ยฑ20.44 / 1565.67 ms โ 1514.70 / 1560.53 ยฑ58.84 / 1676.83 ms โ no change โ
โ QQuery 35 โ 400.00 / 408.74 ยฑ9.98 / 427.54 ms โ 396.96 / 403.37 ยฑ5.61 / 412.59 ms โ no change โ
โ QQuery 36 โ 123.25 / 125.55 ยฑ1.42 / 127.50 ms โ 114.59 / 121.59 ยฑ4.30 / 126.58 ms โ no change โ
โ QQuery 37 โ 48.55 / 50.04 ยฑ1.19 / 52.15 ms โ 47.50 / 49.89 ยฑ1.91 / 53.12 ms โ no change โ
โ QQuery 38 โ 74.92 / 76.43 ยฑ2.40 / 81.22 ms โ 75.17 / 77.26 ยฑ1.16 / 78.50 ms โ no change โ
โ QQuery 39 โ 217.94 / 224.91 ยฑ4.58 / 231.46 ms โ 205.20 / 215.16 ยฑ8.29 / 225.82 ms โ no change โ
โ QQuery 40 โ 25.23 / 29.36 ยฑ2.24 / 31.65 ms โ 24.38 / 25.78 ยฑ1.23 / 27.39 ms โ +1.14x faster โ
โ QQuery 41 โ 21.54 / 23.09 ยฑ1.28 / 24.60 ms โ 20.84 / 24.19 ยฑ1.77 / 26.14 ms โ no change โ
โ QQuery 42 โ 19.78 / 20.14 ยฑ0.37 / 20.81 ms โ 20.13 / 22.55 ยฑ3.33 / 29.13 ms โ 1.12x slower โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 22791.92ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 23140.92ms โ
โ Average Time (HEAD) โ 530.04ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 538.16ms โ
โ Queries Faster โ 3 โ
โ Queries Slower โ 5 โ
โ Queries with No Change โ 35 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 114.9s |
| Peak memory | 37.6 GiB |
| Avg memory | 27.4 GiB |
| CPU user | 1089.1s |
| CPU sys | 85.9s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 116.6s |
| Peak memory | 34.4 GiB |
| Avg memory | 24.5 GiB |
| CPU user | 1083.8s |
| CPU sys | 102.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4263004259
Benchmark for [this request](https://github.com/apache/datafusion/pull/21561#issuecomment-4262974208) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
|
::: /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-58.1.0/src/types.rs:90:9
|
90 | pub struct $name {}
| ---------------- similarly named struct `UInt32Type` defined here
|
help: a struct with a similar name exists
|
469 - UInt16Type,
469 + UInt32Type,
|
help: consider importing this struct
|
18 + use arrow::datatypes::UInt16Type;
|
For more information about this error, try `rustc --explain E0425`.
warning: `datafusion-functions-aggregate` (lib) generated 1 warning
error: could not compile `datafusion-functions-aggregate` (lib) due to 4 previous errors; 1 warning emitted
warning: build failed, waiting for other jobs to finish...
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4265729704
run benchmark clickbench_partitioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4247074370
Seems like merge history is messed up when I cherry picked a commit . I will clean this up to have 2 commits 1.hashtable approach and the current one : Vector of group , count tuples to measure performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4256932880
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4256919523)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4256919523-1335-b6q2h 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (b9dcf85869706b90288153c487aaebd186cf30c8) to 2818abb (merge-base) [diff](https://github.com/apache/datafusion/compare/2818abb8a10bf8b001669c15cf597f9ddfb86ed1..b9dcf85869706b90288153c487aaebd186cf30c8) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4256932874
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4256919523)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4256919523-1334-2b22b 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (b9dcf85869706b90288153c487aaebd186cf30c8) to 2818abb (merge-base) [diff](https://github.com/apache/datafusion/compare/2818abb8a10bf8b001669c15cf597f9ddfb86ed1..b9dcf85869706b90288153c487aaebd186cf30c8) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4257015180
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4256919523)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 7.07 / 7.48 ยฑ0.71 / 8.91 ms โ 6.74 / 7.25 ยฑ0.83 / 8.90 ms โ no change โ
โ QQuery 2 โ 144.82 / 145.72 ยฑ0.68 / 146.45 ms โ 145.19 / 146.21 ยฑ0.86 / 147.53 ms โ no change โ
โ QQuery 3 โ 114.45 / 115.22 ยฑ0.63 / 116.20 ms โ 113.60 / 114.54 ยฑ0.62 / 115.34 ms โ no change โ
โ QQuery 4 โ 1385.34 / 1404.74 ยฑ15.64 / 1423.53 ms โ 1362.26 / 1387.17 ยฑ15.54 / 1401.48 ms โ no change โ
โ QQuery 5 โ 171.86 / 175.09 ยฑ2.43 / 179.15 ms โ 172.57 / 173.75 ยฑ1.32 / 176.29 ms โ no change โ
โ QQuery 6 โ 841.88 / 861.27 ยฑ21.44 / 896.24 ms โ 844.84 / 875.10 ยฑ22.50 / 913.60 ms โ no change โ
โ QQuery 7 โ 343.79 / 346.22 ยฑ2.07 / 350.05 ms โ 342.11 / 346.77 ยฑ4.59 / 355.59 ms โ no change โ
โ QQuery 8 โ 116.71 / 118.15 ยฑ1.17 / 120.00 ms โ 115.23 / 117.08 ยฑ1.04 / 118.15 ms โ no change โ
โ QQuery 9 โ 101.56 / 106.99 ยฑ5.19 / 115.59 ms โ 102.16 / 109.33 ยฑ10.59 / 130.08 ms โ no change โ
โ QQuery 10 โ 105.74 / 107.51 ยฑ1.17 / 109.37 ms โ 107.08 / 108.15 ยฑ1.10 / 110.23 ms โ no change โ
โ QQuery 11 โ 966.74 / 985.74 ยฑ13.99 / 1007.11 ms โ 972.77 / 975.84 ยฑ1.86 / 978.38 ms โ no change โ
โ QQuery 12 โ 45.83 / 46.73 ยฑ0.67 / 47.53 ms โ 44.39 / 45.39 ยฑ0.90 / 46.67 ms โ no change โ
โ QQuery 13 โ 402.43 / 407.22 ยฑ3.19 / 410.86 ms โ 402.11 / 404.43 ยฑ2.04 / 407.02 ms โ no change โ
โ QQuery 14 โ 1008.87 / 1012.44 ยฑ2.50 / 1015.49 ms โ 1000.35 / 1015.00 ยฑ9.67 / 1026.12 ms โ no change โ
โ QQuery 15 โ 15.49 / 16.96 ยฑ1.66 / 19.51 ms โ 16.39 / 17.68 ยฑ1.57 / 20.69 ms โ no change โ
โ QQuery 16 โ 7.32 / 7.97 ยฑ0.58 / 9.06 ms โ 7.36 / 7.91 ยฑ0.41 / 8.50 ms โ no change โ
โ QQuery 17 โ 226.96 / 230.27 ยฑ1.78 / 231.94 ms โ 231.95 / 233.49 ยฑ1.31 / 235.14 ms โ no change โ
โ QQuery 18 โ 128.28 / 129.67 ยฑ0.92 / 130.70 ms โ 127.48 / 130.08 ยฑ1.40 / 131.59 ms โ no change โ
โ QQuery 19 โ 157.59 / 160.15 ยฑ1.85 / 162.54 ms โ 158.53 / 159.68 ยฑ1.15 / 161.73 ms โ no change โ
โ QQuery 20 โ 14.16 / 14.57 ยฑ0.24 / 14.89 ms โ 14.45 / 15.10 ยฑ0.42 / 15.78 ms โ no change โ
โ QQuery 21 โ 20.07 / 20.57 ยฑ0.38 / 21.09 ms โ 19.82 / 20.50 ยฑ0.35 / 20.79 ms โ no change โ
โ QQuery 22 โ 500.02 / 503.55 ยฑ2.77 / 506.92 ms โ 488.61 / 494.89 ยฑ4.01 / 499.91 ms โ no change โ
โ QQuery 23 โ 906.41 / 913.58 ยฑ6.53 / 922.50 ms โ 884.12 / 907.50 ยฑ15.19 / 924.88 ms โ no change โ
โ QQuery 24 โ 384.98 / 391.69 ยฑ6.75 / 403.62 ms โ 387.88 / 389.98 ยฑ1.08 / 390.88 ms โ no change โ
โ QQuery 25 โ 342.68 / 344.89 ยฑ1.56 / 346.72 ms โ 343.45 / 346.56 ยฑ2.32 / 350.49 ms โ no change โ
โ QQuery 26 โ 80.73 / 81.93 ยฑ0.98 / 83.57 ms โ 82.21 / 84.47 ยฑ1.21 / 85.84 ms โ no change โ
โ QQuery 27 โ 7.01 / 7.76 ยฑ0.67 / 8.66 ms โ 7.10 / 7.59 ยฑ0.65 / 8.86 ms โ no change โ
โ QQuery 28 โ 149.71 / 150.52 ยฑ0.59 / 151.21 ms โ 150.29 / 151.58 ยฑ1.10 / 153.43 ms โ no change โ
โ QQuery 29 โ 283.96 / 285.46 ยฑ1.01 / 286.55 ms โ 284.44 / 285.79 ยฑ1.45 / 288.54 ms โ no change โ
โ QQuery 30 โ 44.94 / 46.34 ยฑ0.90 / 47.70 ms โ 43.93 / 45.85 ยฑ1.42 / 48.24 ms โ no change โ
โ QQuery 31 โ 169.85 / 172.94 ยฑ2.25 / 175.48 ms โ 170.96 / 172.27 ยฑ0.96 / 173.47 ms โ no change โ
โ QQuery 32 โ 56.74 / 58.45 ยฑ1.17 / 59.50 ms โ 57.40 / 58.69 ยฑ1.40 / 60.82 ms โ no change โ
โ QQuery 33 โ 140.41 / 141.67 ยฑ0.97 / 143.06 ms โ 140.61 / 142.44 ยฑ1.56 / 145.28 ms โ no change โ
โ QQuery 34 โ 6.79 / 7.12 ยฑ0.28 / 7.59 ms โ 7.45 / 7.65 ยฑ0.14 / 7.82 ms โ 1.07x slower โ
โ QQuery 35 โ 104.98 / 107.93 ยฑ2.43 / 110.84 ms โ 108.27 / 109.58 ยฑ0.84 / 110.54 ms โ no change โ
โ QQuery 36 โ 6.81 / 6.91 ยฑ0.07 / 7.02 ms โ 6.75 / 7.23 ยฑ0.51 / 8.19 ms โ no change โ
โ QQuery 37 โ 8.21 / 9.13 ยฑ0.89 / 10.77 ms โ 8.42 / 8.99 ยฑ0.46 / 9.59 ms โ no change โ
โ QQuery 38 โ 81.09 / 88.43 ยฑ4.28 / 92.23 ms โ 86.36 / 89.71 ยฑ3.35 / 96.08 ms โ no change โ
โ QQuery 39 โ 123.93 / 128.02 ยฑ3.92 / 134.97 ms โ 126.83 / 130.07 ยฑ2.15 / 133.26 ms โ no change โ
โ QQuery 40 โ 110.87 / 116.25 ยฑ4.74 / 124.93 ms โ 109.11 / 114.05 ยฑ4.68 / 122.85 ms โ no change โ
โ QQuery 41 โ 14.35 / 15.50 ยฑ0.77 / 16.32 ms โ 14.33 / 15.72 ยฑ0.94 / 16.67 ms โ no change โ
โ QQuery 42 โ 109.18 / 111.27 ยฑ1.07 / 112.22 ms โ 109.43 / 111.15 ยฑ1.02 / 112.30 ms โ no change โ
โ QQuery 43 โ 6.07 / 6.19 ยฑ0.20 / 6.59 ms โ 6.12 / 6.25 ยฑ0.18 / 6.60 ms โ no change โ
โ QQuery 44 โ 11.77 / 12.21 ยฑ0.50 / 13.18 ms โ 11.97 / 12.41 ยฑ0.48 / 13.28 ms โ no change โ
โ QQuery 45 โ 50.23 / 51.49 ยฑ1.40 / 54.19 ms โ 50.96 / 51.95 ยฑ1.05 / 53.82 ms โ no change โ
โ QQuery 46 โ 8.57 / 9.30 ยฑ0.72 / 10.54 ms โ 8.73 / 8.87 ยฑ0.10 / 9.02 ms โ no change โ
โ QQuery 47 โ 737.64 / 746.36 ยฑ5.72 / 754.67 ms โ 737.99 / 752.92 ยฑ9.72 / 765.84 ms โ no change โ
โ QQuery 48 โ 294.87 / 297.59 ยฑ2.36 / 300.57 ms โ 291.83 / 294.86 ยฑ2.51 / 299.06 ms โ no change โ
โ QQuery 49 โ 252.06 / 254.25 ยฑ2.33 / 258.50 ms โ 253.66 / 257.10 ยฑ2.21 / 259.72 ms โ no change โ
โ QQuery 50 โ 226.88 / 232.82 ยฑ4.08 / 236.87 ms โ 227.92 / 232.52 ยฑ2.72 / 235.75 ms โ no change โ
โ QQuery 51 โ 180.26 / 183.71 ยฑ2.50 / 187.54 ms โ 180.06 / 183.88 ยฑ3.64 / 190.08 ms โ no change โ
โ QQuery 52 โ 107.84 / 110.71 ยฑ1.64 / 112.84 ms โ 108.68 / 109.18 ยฑ0.45 / 109.98 ms โ no change โ
โ QQuery 53 โ 103.37 / 103.81 ยฑ0.36 / 104.30 ms โ 102.31 / 103.93 ยฑ1.16 / 105.91 ms โ no change โ
โ QQuery 54 โ 146.72 / 147.39 ยฑ0.70 / 148.74 ms โ 146.89 / 148.20 ยฑ0.89 / 149.04 ms โ no change โ
โ QQuery 55 โ 106.69 / 107.68 ยฑ1.23 / 110.04 ms โ 106.80 / 107.54 ยฑ0.61 / 108.27 ms โ no change โ
โ QQuery 56 โ 141.95 / 143.80 ยฑ1.28 / 145.76 ms โ 140.95 / 142.71 ยฑ0.92 / 143.60 ms โ no change โ
โ QQuery 57 โ 174.45 / 175.38 ยฑ0.79 / 176.60 ms โ 176.87 / 179.20 ยฑ1.51 / 180.84 ms โ no change โ
โ QQuery 58 โ 287.53 / 299.32 ยฑ6.58 / 306.63 ms โ 294.94 / 306.33 ยฑ7.96 / 316.41 ms โ no change โ
โ QQuery 59 โ 199.19 / 200.70 ยฑ0.82 / 201.54 ms โ 200.47 / 202.45 ยฑ1.34 / 204.13 ms โ no change โ
โ QQuery 60 โ 143.63 / 144.76 ยฑ0.91 / 146.40 ms โ 145.03 / 147.82 ยฑ2.12 / 150.35 ms โ no change โ
โ QQuery 61 โ 13.34 / 13.57 ยฑ0.18 / 13.75 ms โ 13.78 / 13.95 ยฑ0.16 / 14.16 ms โ no change โ
โ QQuery 62 โ 928.34 / 973.60 ยฑ49.05 / 1068.24 ms โ 914.90 / 972.40 ยฑ34.80 / 1023.42 ms โ no change โ
โ QQuery 63 โ 105.42 / 107.86 ยฑ3.00 / 113.56 ms โ 105.43 / 107.51 ยฑ1.73 / 110.04 ms โ no change โ
โ QQuery 64 โ 688.50 / 695.80 ยฑ4.58 / 701.31 ms โ 713.89 / 717.28 ยฑ2.54 / 721.30 ms โ no change โ
โ QQuery 65 โ 250.68 / 258.30 ยฑ3.87 / 261.37 ms โ 268.34 / 273.37 ยฑ4.62 / 279.75 ms โ 1.06x slower โ
โ QQuery 66 โ 245.86 / 267.22 ยฑ12.18 / 281.37 ms โ 264.11 / 269.23 ยฑ4.47 / 277.51 ms โ no change โ
โ QQuery 67 โ 318.04 / 321.69 ยฑ2.21 / 324.78 ms โ 327.49 / 331.27 ยฑ3.10 / 336.18 ms โ no change โ
โ QQuery 68 โ 9.37 / 10.74 ยฑ1.28 / 13.02 ms โ 10.42 / 12.06 ยฑ1.43 / 14.49 ms โ 1.12x slower โ
โ QQuery 69 โ 102.79 / 105.55 ยฑ2.35 / 109.86 ms โ 105.12 / 107.41 ยฑ1.80 / 110.45 ms โ no change โ
โ QQuery 70 โ 337.52 / 361.48 ยฑ13.38 / 377.29 ms โ 335.28 / 349.11 ยฑ13.02 / 370.77 ms โ no change โ
โ QQuery 71 โ 134.58 / 140.91 ยฑ6.99 / 154.22 ms โ 136.15 / 139.36 ยฑ3.69 / 146.41 ms โ no change โ
โ QQuery 72 โ 621.86 / 632.63 ยฑ8.34 / 642.44 ms โ 639.63 / 646.30 ยฑ6.88 / 656.19 ms โ no change โ
โ QQuery 73 โ 6.68 / 8.23 ยฑ1.40 / 10.58 ms โ 7.38 / 8.67 ยฑ1.40 / 11.18 ms โ 1.05x slower โ
โ QQuery 74 โ 599.87 / 609.30 ยฑ5.42 / 616.42 ms โ 650.87 / 656.76 ยฑ3.23 / 660.46 ms โ 1.08x slower โ
โ QQuery 75 โ 277.29 / 279.17 ยฑ1.37 / 281.52 ms โ 278.93 / 281.34 ยฑ1.84 / 284.49 ms โ no change โ
โ QQuery 76 โ 131.55 / 134.32 ยฑ1.76 / 137.03 ms โ 134.48 / 136.47 ยฑ1.83 / 139.24 ms โ no change โ
โ QQuery 77 โ 188.01 / 189.93 ยฑ1.47 / 191.76 ms โ 189.69 / 191.95 ยฑ1.38 / 193.64 ms โ no change โ
โ QQuery 78 โ 347.89 / 353.48 ยฑ4.94 / 361.13 ms โ 347.35 / 352.57 ยฑ4.63 / 360.06 ms โ no change โ
โ QQuery 79 โ 243.05 / 245.48 ยฑ3.30 / 251.97 ms โ 241.97 / 245.85 ยฑ2.59 / 249.36 ms โ no change โ
โ QQuery 80 โ 324.59 / 325.92 ยฑ1.33 / 328.35 ms โ 327.89 / 330.55 ยฑ2.63 / 335.16 ms โ no change โ
โ QQuery 81 โ 27.08 / 28.14 ยฑ0.75 / 29.36 ms โ 27.63 / 28.30 ยฑ1.07 / 30.42 ms โ no change โ
โ QQuery 82 โ 198.91 / 203.87 ยฑ3.26 / 207.82 ms โ 201.49 / 205.22 ยฑ2.44 / 208.88 ms โ no change โ
โ QQuery 83 โ 39.95 / 41.28 ยฑ1.34 / 43.54 ms โ 40.23 / 42.31 ยฑ1.39 / 43.55 ms โ no change โ
โ QQuery 84 โ 49.42 / 50.15 ยฑ0.51 / 50.80 ms โ 50.30 / 50.50 ยฑ0.19 / 50.78 ms โ no change โ
โ QQuery 85 โ 149.88 / 152.05 ยฑ1.21 / 153.25 ms โ 151.19 / 152.73 ยฑ0.92 / 153.68 ms โ no change โ
โ QQuery 86 โ 38.72 / 40.65 ยฑ1.22 / 42.02 ms โ 39.55 / 41.18 ยฑ1.43 / 43.86 ms โ no change โ
โ QQuery 87 โ 86.31 / 90.04 ยฑ3.88 / 97.45 ms โ 87.75 / 91.38 ยฑ2.85 / 96.47 ms โ no change โ
โ QQuery 88 โ 101.69 / 103.06 ยฑ1.61 / 106.10 ms โ 101.19 / 102.46 ยฑ1.13 / 104.14 ms โ no change โ
โ QQuery 89 โ 120.01 / 121.53 ยฑ1.30 / 123.08 ms โ 118.85 / 120.34 ยฑ0.95 / 121.33 ms โ no change โ
โ QQuery 90 โ 24.34 / 24.83 ยฑ0.30 / 25.16 ms โ 23.75 / 24.05 ยฑ0.27 / 24.41 ms โ no change โ
โ QQuery 91 โ 63.96 / 65.48 ยฑ1.28 / 67.08 ms โ 61.95 / 64.85 ยฑ1.85 / 67.75 ms โ no change โ
โ QQuery 92 โ 58.63 / 59.05 ยฑ0.34 / 59.59 ms โ 58.22 / 58.75 ยฑ0.59 / 59.65 ms โ no change โ
โ QQuery 93 โ 191.92 / 193.64 ยฑ1.39 / 195.99 ms โ 188.63 / 190.23 ยฑ1.26 / 192.15 ms โ no change โ
โ QQuery 94 โ 61.50 / 62.69 ยฑ1.10 / 64.43 ms โ 62.00 / 62.74 ยฑ0.67 / 63.84 ms โ no change โ
โ QQuery 95 โ 128.74 / 130.14 ยฑ1.01 / 131.55 ms โ 127.94 / 129.66 ยฑ1.25 / 131.13 ms โ no change โ
โ QQuery 96 โ 74.03 / 75.24 ยฑ1.17 / 77.04 ms โ 72.04 / 74.83 ยฑ1.46 / 75.98 ms โ no change โ
โ QQuery 97 โ 127.36 / 130.30 ยฑ2.35 / 133.73 ms โ 128.73 / 130.71 ยฑ1.58 / 132.49 ms โ no change โ
โ QQuery 98 โ 157.25 / 160.34 ยฑ1.64 / 161.72 ms โ 154.83 / 156.28 ยฑ1.34 / 157.92 ms โ no change โ
โ QQuery 99 โ 10799.62 / 10838.06 ยฑ36.29 / 10884.80 ms โ 10791.51 / 10835.29 ยฑ32.33 / 10874.14 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 31943.25ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 32045.61ms โ
โ Average Time (HEAD) โ 322.66ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 323.69ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 5 โ
โ Queries with No Change โ 94 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.1s |
| Peak memory | 5.4 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 263.4s |
| CPU sys | 17.9s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 160.5s |
| Peak memory | 5.7 GiB |
| Avg memory | 4.8 GiB |
| CPU user | 265.5s |
| CPU sys | 17.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4247315039
@Dandandan , I am trying to a Vector of pairs (group ID , value) approach to see if SIMD (sort and return group counts on ly during `evaluate` would cost lesser than computing hashes in `update_batch` method) . Results were promising on my local but I would be happy if you could run a benchmarks on GH runners for more accuracy. I also moved distinct accumulators to a separate cold path (which were probably causing other count queries' minor slowness)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3098475972
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ ArrayRef, AsArray, BooleanArray, Int64Array, ListArray, PrimitiveArray,
+};
+use arrow::buffer::OffsetBuffer;
+use arrow::datatypes::{ArrowPrimitiveType, Field};
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+use crate::aggregate::groups_accumulator::accumulate::accumulate;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ seen: HashSet<(usize, T::Native), RandomState>,
+ counts: Vec<i64>,
+}
+
+impl<T: ArrowPrimitiveType> PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ pub fn new() -> Self {
+ Self {
+ seen: HashSet::default(),
+ counts: Vec::new(),
+ }
+ }
+}
+
+impl<T: ArrowPrimitiveType> Default for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ArrowPrimitiveType + Send + std::fmt::Debug> GroupsAccumulator
+ for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn update_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> datafusion_common::Result<()> {
+ debug_assert_eq!(values.len(), 1);
+ self.counts.resize(total_num_groups, 0);
+ let arr = values[0].as_primitive::<T>();
+ accumulate(group_indices, arr, opt_filter, |group_idx, value| {
+ if self.seen.insert((group_idx, value)) {
+ self.counts[group_idx] += 1;
+ }
+ });
+ Ok(())
+ }
+
+ fn evaluate(&mut self, emit_to: EmitTo) -> datafusion_common::Result<ArrayRef> {
+ let counts = emit_to.take_needed(&mut self.counts);
+
+ match emit_to {
+ EmitTo::All => {
+ self.seen.clear();
+ }
+ EmitTo::First(n) => {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx >= n {
+ remaining.insert((group_idx - n, value));
+ }
+ }
+ self.seen = remaining;
+ }
+ }
+
+ Ok(Arc::new(Int64Array::from(counts)))
+ }
+
+ fn state(&mut self, emit_to: EmitTo) -> datafusion_common::Result<Vec<ArrayRef>> {
+ let num_emitted = match emit_to {
+ EmitTo::All => self.counts.len(),
+ EmitTo::First(n) => n,
+ };
+
+ let mut group_values: Vec<Vec<T::Native>> = vec![Vec::new(); num_emitted];
+
+ if matches!(emit_to, EmitTo::All) {
+ for (group_idx, value) in self.seen.drain() {
+ group_values[group_idx].push(value);
+ }
+ self.counts.clear();
+ } else {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx < num_emitted {
+ group_values[group_idx].push(value);
+ } else {
+ remaining.insert((group_idx - num_emitted, value));
+ }
+ }
+ self.seen = remaining;
+ let _ = emit_to.take_needed(&mut self.counts);
+ }
+
+ let mut offsets = vec![0i32];
+ let mut all_values = Vec::new();
+ for values in &group_values {
+ all_values.extend(values.iter().copied());
+ offsets.push(all_values.len() as i32);
+ }
+
+ let values_array = Arc::new(PrimitiveArray::<T>::from_iter_values(all_values));
+ let list_array = ListArray::new(
+ Arc::new(Field::new_list_field(T::DATA_TYPE, true)),
+ OffsetBuffer::new(offsets.into()),
+ values_array,
+ None,
+ );
+
+ Ok(vec![Arc::new(list_array)])
+ }
+
+ fn merge_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ _opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> datafusion_common::Result<()> {
+ debug_assert_eq!(values.len(), 1);
+ self.counts.resize(total_num_groups, 0);
+ let list_array = values[0].as_list::<i32>();
+
+ for (row_idx, &group_idx) in group_indices.iter().enumerate() {
+ let inner = list_array.value(row_idx);
Review Comment:
Something like?
```
let inner_values = inner.values();
let offsets = list_array.offsets();
for (row_idx, &group_idx) in group_indices.iter().enumerate() {
let start = offsets[row_idx] as usize;
let end = offsets[row_idx + 1] as usize;
for &value in &inner_values[start..end] {
if self.seen.insert((group_idx, value)) {
self.counts[group_idx] += 1;
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266117426
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266107489)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4266107489-1409-hxvgn 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (066504d269b8e402d6b732729ae5265a27ed8458) to 5a427cb (merge-base) [diff](https://github.com/apache/datafusion/compare/5a427cb55d8630b51b9590a25b369b434611975e..066504d269b8e402d6b732729ae5265a27ed8458) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3098632049
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ ArrayRef, AsArray, BooleanArray, Int64Array, ListArray, PrimitiveArray,
+};
+use arrow::buffer::OffsetBuffer;
+use arrow::datatypes::{ArrowPrimitiveType, Field};
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+use crate::aggregate::groups_accumulator::accumulate::accumulate;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ seen: HashSet<(usize, T::Native), RandomState>,
+ counts: Vec<i64>,
+}
+
+impl<T: ArrowPrimitiveType> PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ pub fn new() -> Self {
+ Self {
+ seen: HashSet::default(),
+ counts: Vec::new(),
+ }
+ }
+}
+
+impl<T: ArrowPrimitiveType> Default for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ArrowPrimitiveType + Send + std::fmt::Debug> GroupsAccumulator
+ for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn update_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> datafusion_common::Result<()> {
+ debug_assert_eq!(values.len(), 1);
+ self.counts.resize(total_num_groups, 0);
+ let arr = values[0].as_primitive::<T>();
+ accumulate(group_indices, arr, opt_filter, |group_idx, value| {
+ if self.seen.insert((group_idx, value)) {
+ self.counts[group_idx] += 1;
+ }
+ });
+ Ok(())
+ }
+
+ fn evaluate(&mut self, emit_to: EmitTo) -> datafusion_common::Result<ArrayRef> {
+ let counts = emit_to.take_needed(&mut self.counts);
+
+ match emit_to {
+ EmitTo::All => {
+ self.seen.clear();
+ }
+ EmitTo::First(n) => {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx >= n {
+ remaining.insert((group_idx - n, value));
+ }
+ }
+ self.seen = remaining;
+ }
+ }
+
+ Ok(Arc::new(Int64Array::from(counts)))
+ }
+
+ fn state(&mut self, emit_to: EmitTo) -> datafusion_common::Result<Vec<ArrayRef>> {
+ let num_emitted = match emit_to {
+ EmitTo::All => self.counts.len(),
+ EmitTo::First(n) => n,
+ };
+
+ let mut group_values: Vec<Vec<T::Native>> = vec![Vec::new(); num_emitted];
+
+ if matches!(emit_to, EmitTo::All) {
+ for (group_idx, value) in self.seen.drain() {
+ group_values[group_idx].push(value);
+ }
+ self.counts.clear();
+ } else {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx < num_emitted {
+ group_values[group_idx].push(value);
+ } else {
+ remaining.insert((group_idx - num_emitted, value));
+ }
+ }
+ self.seen = remaining;
+ let _ = emit_to.take_needed(&mut self.counts);
+ }
+
+ let mut offsets = vec![0i32];
+ let mut all_values = Vec::new();
+ for values in &group_values {
+ all_values.extend(values.iter().copied());
+ offsets.push(all_values.len() as i32);
+ }
+
+ let values_array = Arc::new(PrimitiveArray::<T>::from_iter_values(all_values));
Review Comment:
This is faster as well @coderfender (avoids a copy).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266245112
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3068983573
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, BooleanArray, Int64Array, ListBuilder, PrimitiveBuilder,
+};
+use arrow::datatypes::ArrowPrimitiveType;
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ /// Count distinct per group.
+ values: Vec<HashSet<T::Native, RandomState>>,
Review Comment:
It could be `Hashset<(T::Native, usize)>` to avoid individual hash sets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230897539
@Dandandan could you retrigger the benchmarks again please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4263055980
```โ QQuery 9 โ 640.75 / 661.55 ยฑ18.57 / 692.50 ms โ 591.43 / 597.45 ยฑ4.62 / 603.98 ms โ +1.11x faster```
This looks like a genuine improvement!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266120211
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266107489)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4266107489-1407-q98hw 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (066504d269b8e402d6b732729ae5265a27ed8458) to 5a427cb (merge-base) [diff](https://github.com/apache/datafusion/compare/5a427cb55d8630b51b9590a25b369b434611975e..066504d269b8e402d6b732729ae5265a27ed8458) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4263938330
@Dandandan true! I also benchmarked through `cargo bench` and the results are similar :
Benchmark PR is approved and ready to be merged to main for better comparison though
```
group groups_accumulator_hash_table main
----- ----------------------------- ----
count_distinct_groups g10000_high_skewed 1.00 50.0ยฑ0.69ยตs ? ?/sec 31.19 1558.7ยฑ19.53ยตs ? ?/sec
count_distinct_groups g10000_high_uniform 1.00 82.9ยฑ0.98ยตs ? ?/sec 19.69 1632.0ยฑ36.13ยตs ? ?/sec
count_distinct_groups g10000_low_skewed 1.00 48.5ยฑ0.48ยตs ? ?/sec 31.25 1514.7ยฑ31.43ยตs ? ?/sec
count_distinct_groups g10000_low_uniform 1.00 82.6ยฑ0.85ยตs ? ?/sec 19.34 1596.9ยฑ17.45ยตs ? ?/sec
count_distinct_groups g10000_mid_skewed 1.00 50.1ยฑ0.40ยตs ? ?/sec 30.94 1551.5ยฑ10.96ยตs ? ?/sec
count_distinct_groups g10000_mid_uniform 1.00 83.9ยฑ1.44ยตs ? ?/sec 19.10 1601.5ยฑ19.65ยตs ? ?/sec
count_distinct_groups g1000_high_skewed 1.00 48.6ยฑ0.86ยตs ? ?/sec 4.82 234.4ยฑ2.02ยตs ? ?/sec
count_distinct_groups g1000_high_uniform 1.00 80.7ยฑ0.72ยตs ? ?/sec 3.33 268.7ยฑ4.11ยตs ? ?/sec
count_distinct_groups g1000_low_skewed 1.00 32.4ยฑ0.22ยตs ? ?/sec 6.30 204.0ยฑ7.65ยตs ? ?/sec
count_distinct_groups g1000_low_uniform 1.00 80.5ยฑ0.86ยตs ? ?/sec 3.29 265.1ยฑ2.39ยตs ? ?/sec
count_distinct_groups g1000_mid_skewed 1.00 48.4ยฑ0.64ยตs ? ?/sec 4.75 229.6ยฑ1.71ยตs ? ?/sec
count_distinct_groups g1000_mid_uniform 1.00 80.7ยฑ0.61ยตs ? ?/sec 3.32 267.4ยฑ2.67ยตs ? ?/sec
count_distinct_groups g100_high_skewed 1.00 48.8ยฑ1.41ยตs ? ?/sec 1.80 87.8ยฑ1.53ยตs ? ?/sec
count_distinct_groups g100_high_uniform 1.00 80.7ยฑ0.88ยตs ? ?/sec 1.32 106.2ยฑ3.51ยตs ? ?/sec
count_distinct_groups g100_low_skewed 1.00 33.1ยฑ0.20ยตs ? ?/sec 1.65 54.7ยฑ0.52ยตs ? ?/sec
count_distinct_groups g100_low_uniform 1.00 81.0ยฑ1.66ยตs ? ?/sec 1.29 104.5ยฑ0.65ยตs ? ?/sec
count_distinct_groups g100_mid_skewed 1.00 48.0ยฑ0.64ยตs ? ?/sec 1.60 76.7ยฑ1.42ยตs ? ?/sec
count_distinct_groups g100_mid_uniform 1.00 80.2ยฑ0.64ยตs ? ?/sec 1.30 104.5ยฑ1.71ยตs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4270345808
Results seem to be faster now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4262001319
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4261984688)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4261984688-1389-lthd7 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (ba140a00af402b7f8e6ff3470d1f50b369766d69) to cc4717a (merge-base) [diff](https://github.com/apache/datafusion/compare/cc4717a14452e65a4d99c4b180d62c2793b713bd..ba140a00af402b7f8e6ff3470d1f50b369766d69) using: clickbench_extended
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4265806757
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4265729704)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.21 / 4.40 ยฑ6.32 / 17.04 ms โ 1.18 / 4.39 ยฑ6.34 / 17.07 ms โ no change โ
โ QQuery 1 โ 15.10 / 15.42 ยฑ0.30 / 15.98 ms โ 14.11 / 14.45 ยฑ0.18 / 14.60 ms โ +1.07x faster โ
โ QQuery 2 โ 43.56 / 43.90 ยฑ0.29 / 44.24 ms โ 43.72 / 43.93 ยฑ0.17 / 44.23 ms โ no change โ
โ QQuery 3 โ 40.09 / 42.73 ยฑ2.06 / 46.38 ms โ 39.61 / 40.25 ยฑ0.40 / 40.86 ms โ +1.06x faster โ
โ QQuery 4 โ 283.72 / 305.21 ยฑ12.09 / 318.27 ms โ 294.34 / 301.81 ยฑ8.59 / 315.63 ms โ no change โ
โ QQuery 5 โ 346.94 / 361.74 ยฑ11.06 / 376.16 ms โ 336.55 / 354.99 ยฑ12.43 / 372.60 ms โ no change โ
โ QQuery 6 โ 5.56 / 5.87 ยฑ0.28 / 6.31 ms โ 5.90 / 13.18 ยฑ13.88 / 40.93 ms โ 2.24x slower โ
โ QQuery 7 โ 17.29 / 18.39 ยฑ0.96 / 19.98 ms โ 17.16 / 17.56 ยฑ0.38 / 18.19 ms โ no change โ
โ QQuery 8 โ 406.59 / 424.81 ยฑ14.13 / 444.51 ms โ 423.60 / 444.90 ยฑ16.74 / 462.50 ms โ no change โ
โ QQuery 9 โ 643.47 / 664.98 ยฑ19.80 / 699.21 ms โ 572.54 / 600.58 ยฑ17.34 / 620.08 ms โ +1.11x faster โ
โ QQuery 10 โ 94.86 / 96.33 ยฑ1.70 / 99.55 ms โ 91.70 / 93.46 ยฑ1.09 / 94.87 ms โ no change โ
โ QQuery 11 โ 104.78 / 105.87 ยฑ1.08 / 107.35 ms โ 99.59 / 101.51 ยฑ1.75 / 104.81 ms โ no change โ
โ QQuery 12 โ 343.17 / 355.49 ยฑ13.19 / 378.60 ms โ 348.23 / 356.14 ยฑ8.40 / 370.78 ms โ no change โ
โ QQuery 13 โ 476.36 / 493.82 ยฑ11.94 / 507.90 ms โ 450.33 / 462.49 ยฑ10.24 / 479.11 ms โ +1.07x faster โ
โ QQuery 14 โ 343.21 / 357.37 ยฑ11.99 / 372.93 ms โ 355.74 / 362.13 ยฑ6.63 / 374.07 ms โ no change โ
โ QQuery 15 โ 373.00 / 398.36 ยฑ26.84 / 444.97 ms โ 345.35 / 371.01 ยฑ16.93 / 395.35 ms โ +1.07x faster โ
โ QQuery 16 โ 710.37 / 732.18 ยฑ18.67 / 762.82 ms โ 714.62 / 743.34 ยฑ19.09 / 765.64 ms โ no change โ
โ QQuery 17 โ 727.83 / 744.11 ยฑ11.86 / 762.28 ms โ 697.43 / 729.72 ยฑ19.15 / 753.88 ms โ no change โ
โ QQuery 18 โ 1485.32 / 1547.62 ยฑ41.99 / 1593.21 ms โ 1513.58 / 1552.24 ยฑ35.01 / 1610.33 ms โ no change โ
โ QQuery 19 โ 37.02 / 57.65 ยฑ20.11 / 89.52 ms โ 37.64 / 49.58 ยฑ10.20 / 66.03 ms โ +1.16x faster โ
โ QQuery 20 โ 729.38 / 746.88 ยฑ18.39 / 771.52 ms โ 716.11 / 740.49 ยฑ24.92 / 788.05 ms โ no change โ
โ QQuery 21 โ 776.02 / 779.38 ยฑ4.00 / 786.93 ms โ 759.65 / 768.34 ยฑ6.17 / 776.61 ms โ no change โ
โ QQuery 22 โ 1136.31 / 1155.49 ยฑ11.56 / 1169.50 ms โ 1141.62 / 1146.49 ยฑ4.56 / 1153.49 ms โ no change โ
โ QQuery 23 โ 3154.25 / 3186.84 ยฑ40.10 / 3260.45 ms โ 3113.22 / 3148.87 ยฑ22.08 / 3178.58 ms โ no change โ
โ QQuery 24 โ 103.19 / 106.88 ยฑ3.07 / 111.49 ms โ 99.64 / 104.28 ยฑ3.66 / 109.56 ms โ no change โ
โ QQuery 25 โ 138.02 / 142.64 ยฑ2.49 / 144.86 ms โ 138.80 / 141.12 ยฑ1.33 / 142.46 ms โ no change โ
โ QQuery 26 โ 98.55 / 101.19 ยฑ1.54 / 103.31 ms โ 103.04 / 104.90 ยฑ1.74 / 107.99 ms โ no change โ
โ QQuery 27 โ 849.47 / 854.67 ยฑ4.02 / 859.30 ms โ 851.63 / 860.49 ยฑ9.88 / 876.35 ms โ no change โ
โ QQuery 28 โ 3271.14 / 3322.96 ยฑ40.84 / 3391.77 ms โ 3263.40 / 3290.05 ยฑ17.34 / 3308.86 ms โ no change โ
โ QQuery 29 โ 51.53 / 77.45 ยฑ42.56 / 162.10 ms โ 51.91 / 55.58 ยฑ5.57 / 66.60 ms โ +1.39x faster โ
โ QQuery 30 โ 384.22 / 395.64 ยฑ10.52 / 414.48 ms โ 383.83 / 393.66 ยฑ6.64 / 404.07 ms โ no change โ
โ QQuery 31 โ 360.43 / 396.43 ยฑ19.65 / 412.94 ms โ 397.01 / 412.70 ยฑ11.98 / 433.36 ms โ no change โ
โ QQuery 32 โ 1234.76 / 1270.41 ยฑ28.08 / 1299.19 ms โ 1067.50 / 1098.30 ยฑ30.37 / 1151.51 ms โ +1.16x faster โ
โ QQuery 33 โ 1479.26 / 1513.83 ยฑ39.06 / 1590.18 ms โ 1480.07 / 1523.44 ยฑ28.58 / 1562.33 ms โ no change โ
โ QQuery 34 โ 1511.41 / 1550.49 ยฑ27.47 / 1582.09 ms โ 1511.20 / 1550.03 ยฑ25.84 / 1589.75 ms โ no change โ
โ QQuery 35 โ 415.09 / 430.83 ยฑ14.96 / 457.33 ms โ 399.67 / 415.64 ยฑ8.46 / 423.64 ms โ no change โ
โ QQuery 36 โ 122.19 / 128.23 ยฑ3.83 / 133.58 ms โ 124.24 / 126.30 ยฑ1.82 / 128.87 ms โ no change โ
โ QQuery 37 โ 50.44 / 52.54 ยฑ1.59 / 54.62 ms โ 49.91 / 51.72 ยฑ1.69 / 54.64 ms โ no change โ
โ QQuery 38 โ 76.37 / 78.89 ยฑ1.95 / 81.77 ms โ 76.28 / 78.93 ยฑ2.03 / 81.82 ms โ no change โ
โ QQuery 39 โ 208.86 / 225.86 ยฑ10.08 / 240.52 ms โ 228.58 / 231.43 ยฑ3.65 / 238.21 ms โ no change โ
โ QQuery 40 โ 25.72 / 28.02 ยฑ2.71 / 33.24 ms โ 26.00 / 28.04 ยฑ2.46 / 32.83 ms โ no change โ
โ QQuery 41 โ 21.86 / 23.73 ยฑ1.40 / 25.80 ms โ 19.95 / 22.60 ยฑ2.05 / 25.01 ms โ no change โ
โ QQuery 42 โ 20.97 / 21.11 ยฑ0.20 / 21.49 ms โ 20.73 / 21.01 ยฑ0.33 / 21.45 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 23366.61ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 22972.12ms โ
โ Average Time (HEAD) โ 543.41ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 534.24ms โ
โ Queries Faster โ 8 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 34 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 117.8s |
| Peak memory | 36.2 GiB |
| Avg memory | 26.2 GiB |
| CPU user | 1098.4s |
| CPU sys | 100.0s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 115.9s |
| Peak memory | 37.4 GiB |
| Avg memory | 28.8 GiB |
| CPU user | 1090.6s |
| CPU sys | 90.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4231186587
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4231151449)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.18 / 4.43 ยฑ6.40 / 17.24 ms โ 1.22 / 4.42 ยฑ6.33 / 17.08 ms โ no change โ
โ QQuery 1 โ 14.14 / 14.35 ยฑ0.12 / 14.46 ms โ 14.19 / 14.40 ยฑ0.14 / 14.60 ms โ no change โ
โ QQuery 2 โ 42.96 / 43.33 ยฑ0.26 / 43.76 ms โ 43.29 / 43.61 ยฑ0.19 / 43.86 ms โ no change โ
โ QQuery 3 โ 43.20 / 46.58 ยฑ2.17 / 49.34 ms โ 43.61 / 45.22 ยฑ0.84 / 45.94 ms โ no change โ
โ QQuery 4 โ 279.62 / 285.18 ยฑ3.85 / 291.38 ms โ 282.80 / 286.99 ยฑ3.55 / 293.04 ms โ no change โ
โ QQuery 5 โ 335.55 / 339.28 ยฑ3.26 / 343.68 ms โ 339.51 / 342.90 ยฑ3.60 / 347.58 ms โ no change โ
โ QQuery 6 โ 5.41 / 5.84 ยฑ0.32 / 6.15 ms โ 5.35 / 7.00 ยฑ2.10 / 11.10 ms โ 1.20x slower โ
โ QQuery 7 โ 16.62 / 16.75 ยฑ0.11 / 16.89 ms โ 16.31 / 18.35 ยฑ2.80 / 23.79 ms โ 1.10x slower โ
โ QQuery 8 โ 404.05 / 415.63 ยฑ8.26 / 423.57 ms โ 408.26 / 415.35 ยฑ6.32 / 426.82 ms โ no change โ
โ QQuery 9 โ 619.95 / 637.18 ยฑ12.54 / 654.86 ms โ 589.90 / 596.08 ยฑ3.37 / 599.61 ms โ +1.07x faster โ
โ QQuery 10 โ 91.77 / 94.44 ยฑ2.32 / 98.04 ms โ 90.59 / 92.41 ยฑ2.02 / 96.15 ms โ no change โ
โ QQuery 11 โ 103.91 / 105.27 ยฑ1.43 / 107.90 ms โ 101.47 / 102.56 ยฑ0.71 / 103.60 ms โ no change โ
โ QQuery 12 โ 336.25 / 340.52 ยฑ3.39 / 346.20 ms โ 327.77 / 345.68 ยฑ19.23 / 381.56 ms โ no change โ
โ QQuery 13 โ 442.42 / 458.16 ยฑ8.64 / 466.84 ms โ 448.85 / 473.02 ยฑ20.35 / 508.42 ms โ no change โ
โ QQuery 14 โ 333.66 / 342.27 ยฑ4.75 / 348.25 ms โ 341.73 / 347.10 ยฑ4.12 / 353.42 ms โ no change โ
โ QQuery 15 โ 343.71 / 355.88 ยฑ8.85 / 366.32 ms โ 349.27 / 359.30 ยฑ13.24 / 382.96 ms โ no change โ
โ QQuery 16 โ 720.11 / 730.36 ยฑ8.27 / 743.79 ms โ 728.67 / 747.74 ยฑ14.48 / 772.58 ms โ no change โ
โ QQuery 17 โ 706.95 / 715.76 ยฑ7.32 / 728.39 ms โ 695.30 / 727.76 ยฑ23.16 / 750.61 ms โ no change โ
โ QQuery 18 โ 1437.57 / 1488.96 ยฑ29.05 / 1521.91 ms โ 1338.52 / 1352.15 ยฑ11.03 / 1367.01 ms โ +1.10x faster โ
โ QQuery 19 โ 35.00 / 37.63 ยฑ2.10 / 41.46 ms โ 35.29 / 37.06 ยฑ1.37 / 39.40 ms โ no change โ
โ QQuery 20 โ 718.48 / 735.78 ยฑ17.63 / 757.90 ms โ 704.57 / 713.90 ยฑ13.53 / 740.83 ms โ no change โ
โ QQuery 21 โ 759.40 / 766.12 ยฑ4.15 / 772.20 ms โ 752.14 / 759.23 ยฑ4.94 / 765.71 ms โ no change โ
โ QQuery 22 โ 1134.45 / 1140.74 ยฑ6.13 / 1150.62 ms โ 1123.10 / 1130.92 ยฑ4.50 / 1135.36 ms โ no change โ
โ QQuery 23 โ 3057.42 / 3077.54 ยฑ16.97 / 3102.84 ms โ 3050.07 / 3067.41 ยฑ11.77 / 3083.38 ms โ no change โ
โ QQuery 24 โ 100.04 / 102.33 ยฑ1.21 / 103.48 ms โ 98.14 / 101.89 ยฑ3.20 / 106.95 ms โ no change โ
โ QQuery 25 โ 137.55 / 138.63 ยฑ1.09 / 139.99 ms โ 137.75 / 139.73 ยฑ1.63 / 142.16 ms โ no change โ
โ QQuery 26 โ 100.87 / 101.88 ยฑ0.98 / 103.75 ms โ 99.43 / 101.34 ยฑ1.42 / 102.80 ms โ no change โ
โ QQuery 27 โ 846.69 / 852.87 ยฑ4.71 / 860.44 ms โ 846.08 / 850.91 ยฑ5.36 / 860.84 ms โ no change โ
โ QQuery 28 โ 3262.91 / 3290.00 ยฑ19.98 / 3321.98 ms โ 3241.62 / 3279.85 ยฑ20.94 / 3302.28 ms โ no change โ
โ QQuery 29 โ 50.34 / 61.98 ยฑ16.80 / 95.30 ms โ 49.28 / 52.83 ยฑ2.41 / 55.41 ms โ +1.17x faster โ
โ QQuery 30 โ 350.63 / 362.40 ยฑ8.87 / 372.66 ms โ 349.23 / 357.21 ยฑ4.84 / 361.45 ms โ no change โ
โ QQuery 31 โ 352.81 / 365.75 ยฑ9.01 / 379.52 ms โ 359.74 / 373.17 ยฑ10.94 / 390.17 ms โ no change โ
โ QQuery 32 โ 1021.09 / 1134.61 ยฑ114.10 / 1326.90 ms โ 1020.55 / 1035.70 ยฑ15.61 / 1059.72 ms โ +1.10x faster โ
โ QQuery 33 โ 1439.08 / 1446.58 ยฑ5.57 / 1453.43 ms โ 1417.77 / 1444.08 ยฑ17.16 / 1469.70 ms โ no change โ
โ QQuery 34 โ 1452.12 / 1467.95 ยฑ10.61 / 1483.12 ms โ 1422.70 / 1446.76 ยฑ25.07 / 1492.31 ms โ no change โ
โ QQuery 35 โ 381.35 / 383.43 ยฑ1.57 / 385.86 ms โ 373.26 / 385.90 ยฑ11.36 / 400.74 ms โ no change โ
โ QQuery 36 โ 116.10 / 121.74 ยฑ3.65 / 125.44 ms โ 109.99 / 117.32 ยฑ5.10 / 123.26 ms โ no change โ
โ QQuery 37 โ 46.62 / 49.31 ยฑ1.68 / 51.76 ms โ 47.54 / 49.77 ยฑ1.61 / 51.77 ms โ no change โ
โ QQuery 38 โ 74.88 / 76.25 ยฑ0.93 / 77.18 ms โ 73.80 / 76.49 ยฑ1.92 / 79.73 ms โ no change โ
โ QQuery 39 โ 207.95 / 218.31 ยฑ5.25 / 222.54 ms โ 209.61 / 214.34 ยฑ3.98 / 218.99 ms โ no change โ
โ QQuery 40 โ 24.19 / 25.49 ยฑ1.30 / 27.77 ms โ 21.12 / 24.09 ยฑ1.88 / 26.86 ms โ +1.06x faster โ
โ QQuery 41 โ 18.86 / 20.76 ยฑ1.18 / 21.93 ms โ 18.84 / 20.39 ยฑ1.19 / 22.51 ms โ no change โ
โ QQuery 42 โ 19.18 / 19.56 ยฑ0.31 / 20.02 ms โ 19.29 / 19.53 ยฑ0.18 / 19.82 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 22437.80ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 22121.88ms โ
โ Average Time (HEAD) โ 521.81ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 514.46ms โ
โ Queries Faster โ 5 โ
โ Queries Slower โ 2 โ
โ Queries with No Change โ 36 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 113.3s |
| Peak memory | 40.5 GiB |
| Avg memory | 30.1 GiB |
| CPU user | 1068.2s |
| CPU sys | 84.2s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 111.7s |
| Peak memory | 41.3 GiB |
| Avg memory | 33.0 GiB |
| CPU user | 1056.9s |
| CPU sys | 78.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4231190853
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4231151449)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 821.46 / 832.96 ยฑ11.19 / 852.43 ms โ 818.15 / 832.59 ยฑ10.24 / 844.30 ms โ no change โ
โ QQuery 1 โ 210.26 / 210.95 ยฑ0.86 / 212.60 ms โ 207.38 / 209.25 ยฑ1.01 / 210.20 ms โ no change โ
โ QQuery 2 โ 499.88 / 502.56 ยฑ1.92 / 505.88 ms โ 499.74 / 502.04 ยฑ1.98 / 505.60 ms โ no change โ
โ QQuery 3 โ 314.20 / 316.27 ยฑ1.34 / 318.27 ms โ 315.85 / 317.00 ยฑ0.69 / 317.96 ms โ no change โ
โ QQuery 4 โ 659.30 / 669.99 ยฑ10.42 / 688.02 ms โ 649.63 / 662.11 ยฑ9.56 / 672.86 ms โ no change โ
โ QQuery 5 โ 9837.93 / 9974.65 ยฑ119.24 / 10120.51 ms โ 9623.91 / 9747.29 ยฑ91.15 / 9882.31 ms โ no change โ
โ QQuery 6 โ 995.83 / 1010.24 ยฑ18.78 / 1047.21 ms โ 992.44 / 1001.14 ยฑ7.30 / 1012.59 ms โ no change โ
โ QQuery 7 โ 752.69 / 766.98 ยฑ11.39 / 784.94 ms โ 828.62 / 901.53 ยฑ42.20 / 955.19 ms โ 1.18x slower โ
โ QQuery 8 โ 397.40 / 403.44 ยฑ6.58 / 415.95 ms โ 403.08 / 410.36 ยฑ3.96 / 415.19 ms โ no change โ
โ QQuery 9 โ 2780.05 / 2825.71 ยฑ41.79 / 2903.89 ms โ 2809.19 / 2836.50 ยฑ25.10 / 2881.82 ms โ no change โ
โ QQuery 10 โ 639.23 / 649.24 ยฑ10.65 / 666.89 ms โ 630.96 / 640.78 ยฑ6.27 / 650.18 ms โ no change โ
โ QQuery 11 โ 2039.54 / 2059.48 ยฑ13.79 / 2077.65 ms โ 2056.40 / 2068.64 ยฑ8.85 / 2079.13 ms โ no change โ
โ QQuery 12 โ 197.12 / 203.80 ยฑ6.72 / 215.86 ms โ 196.73 / 199.63 ยฑ2.32 / 203.35 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 20426.28ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 20328.87ms โ
โ Average Time (HEAD) โ 1571.25ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 1563.76ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 12 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 103.0s |
| Peak memory | 33.3 GiB |
| Avg memory | 29.6 GiB |
| CPU user | 995.5s |
| CPU sys | 45.8s |
| Peak spill | 0 B |
**clickbench_extended โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 102.4s |
| Peak memory | 32.0 GiB |
| Avg memory | 27.6 GiB |
| CPU user | 984.1s |
| CPU sys | 53.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4251531266
run benchmark tpch tpcds clickbench_partitioned clickbench_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4251569514
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4251531266)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4251531266-1282-q8kjb 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (28c607aad4745f0b021b60bf42e1f3b7d49c0971) to 2818abb (merge-base) [diff](https://github.com/apache/datafusion/compare/2818abb8a10bf8b001669c15cf597f9ddfb86ed1..28c607aad4745f0b021b60bf42e1f3b7d49c0971) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4257028698
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4256919523)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 765.54 / 778.90 ยฑ7.47 / 786.50 ms โ 780.31 / 783.19 ยฑ4.16 / 791.30 ms โ no change โ
โ QQuery 1 โ 208.07 / 208.78 ยฑ0.50 / 209.57 ms โ 208.57 / 209.26 ยฑ0.62 / 210.05 ms โ no change โ
โ QQuery 2 โ 491.61 / 495.76 ยฑ2.74 / 499.37 ms โ 492.00 / 498.58 ยฑ3.61 / 501.90 ms โ no change โ
โ QQuery 3 โ 313.45 / 314.24 ยฑ0.76 / 315.21 ms โ 315.40 / 315.72 ยฑ0.22 / 316.03 ms โ no change โ
โ QQuery 4 โ 646.33 / 662.06 ยฑ11.50 / 675.84 ms โ 648.72 / 667.05 ยฑ12.87 / 688.06 ms โ no change โ
โ QQuery 5 โ 9678.11 / 10036.16 ยฑ217.72 / 10346.29 ms โ 9641.57 / 9973.15 ยฑ199.50 / 10183.19 ms โ no change โ
โ QQuery 6 โ 995.68 / 1010.51 ยฑ15.64 / 1039.11 ms โ 992.12 / 997.62 ยฑ7.39 / 1012.21 ms โ no change โ
โ QQuery 7 โ 769.24 / 775.50 ยฑ5.78 / 783.45 ms โ 776.25 / 857.44 ยฑ49.32 / 907.29 ms โ 1.11x slower โ
โ QQuery 8 โ 390.73 / 396.23 ยฑ5.45 / 406.42 ms โ 394.77 / 395.92 ยฑ1.40 / 397.80 ms โ no change โ
โ QQuery 9 โ 2730.48 / 2807.24 ยฑ44.64 / 2870.59 ms โ 2740.76 / 2801.73 ยฑ33.18 / 2832.42 ms โ no change โ
โ QQuery 10 โ 629.73 / 638.29 ยฑ6.42 / 649.18 ms โ 629.55 / 638.90 ยฑ11.53 / 660.39 ms โ no change โ
โ QQuery 11 โ 2068.28 / 2076.28 ยฑ5.87 / 2086.44 ms โ 2188.26 / 2222.48 ยฑ18.74 / 2244.64 ms โ 1.07x slower โ
โ QQuery 12 โ 197.75 / 202.90 ยฑ5.59 / 212.44 ms โ 192.22 / 197.97 ยฑ3.30 / 201.23 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 20402.84ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 20559.00ms โ
โ Average Time (HEAD) โ 1569.45ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 1581.46ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 2 โ
โ Queries with No Change โ 11 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 102.9s |
| Peak memory | 33.2 GiB |
| Avg memory | 29.1 GiB |
| CPU user | 983.4s |
| CPU sys | 45.6s |
| Peak spill | 0 B |
**clickbench_extended โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 103.6s |
| Peak memory | 32.1 GiB |
| Avg memory | 27.4 GiB |
| CPU user | 996.8s |
| CPU sys | 51.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4251678327
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4251531266)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 6.92 / 7.37 ยฑ0.72 / 8.80 ms โ 6.82 / 7.29 ยฑ0.75 / 8.77 ms โ no change โ
โ QQuery 2 โ 144.13 / 145.74 ยฑ0.86 / 146.60 ms โ 145.23 / 146.13 ยฑ0.76 / 147.27 ms โ no change โ
โ QQuery 3 โ 113.89 / 114.60 ยฑ0.41 / 115.05 ms โ 114.22 / 114.56 ยฑ0.27 / 114.99 ms โ no change โ
โ QQuery 4 โ 1416.66 / 1431.69 ยฑ8.47 / 1441.08 ms โ 1373.83 / 1409.25 ยฑ18.78 / 1423.61 ms โ no change โ
โ QQuery 5 โ 173.02 / 174.73 ยฑ1.39 / 176.61 ms โ 174.47 / 175.66 ยฑ1.19 / 177.49 ms โ no change โ
โ QQuery 6 โ 841.11 / 876.54 ยฑ24.75 / 918.24 ms โ 845.68 / 871.68 ยฑ17.09 / 890.19 ms โ no change โ
โ QQuery 7 โ 343.68 / 347.94 ยฑ2.50 / 350.95 ms โ 344.16 / 347.28 ยฑ2.82 / 352.17 ms โ no change โ
โ QQuery 8 โ 115.56 / 117.34 ยฑ0.95 / 118.33 ms โ 117.02 / 118.22 ยฑ1.11 / 119.60 ms โ no change โ
โ QQuery 9 โ 101.55 / 106.13 ยฑ5.60 / 116.58 ms โ 101.68 / 109.34 ยฑ6.33 / 120.99 ms โ no change โ
โ QQuery 10 โ 106.59 / 107.41 ยฑ0.79 / 108.38 ms โ 107.21 / 109.12 ยฑ1.43 / 110.72 ms โ no change โ
โ QQuery 11 โ 977.07 / 989.94 ยฑ7.59 / 999.67 ms โ 971.94 / 977.27 ยฑ4.53 / 983.42 ms โ no change โ
โ QQuery 12 โ 45.90 / 47.43 ยฑ1.17 / 49.21 ms โ 45.19 / 46.05 ยฑ0.72 / 47.06 ms โ no change โ
โ QQuery 13 โ 402.22 / 405.58 ยฑ2.70 / 410.31 ms โ 405.04 / 406.55 ยฑ1.17 / 408.02 ms โ no change โ
โ QQuery 14 โ 994.99 / 1007.70 ยฑ7.74 / 1017.24 ms โ 1012.44 / 1015.34 ยฑ2.29 / 1018.04 ms โ no change โ
โ QQuery 15 โ 15.74 / 16.84 ยฑ1.06 / 18.49 ms โ 16.67 / 18.30 ยฑ1.28 / 20.55 ms โ 1.09x slower โ
โ QQuery 16 โ 7.19 / 8.76 ยฑ1.19 / 10.39 ms โ 7.32 / 8.29 ยฑ1.13 / 10.51 ms โ +1.06x faster โ
โ QQuery 17 โ 230.88 / 233.47 ยฑ3.00 / 239.03 ms โ 228.11 / 232.03 ยฑ2.72 / 236.34 ms โ no change โ
โ QQuery 18 โ 128.17 / 129.27 ยฑ0.96 / 130.90 ms โ 130.10 / 131.06 ยฑ0.72 / 132.29 ms โ no change โ
โ QQuery 19 โ 158.66 / 159.82 ยฑ1.25 / 162.16 ms โ 156.51 / 157.81 ยฑ1.58 / 160.84 ms โ no change โ
โ QQuery 20 โ 14.02 / 14.50 ยฑ0.38 / 14.95 ms โ 14.63 / 15.11 ยฑ0.30 / 15.38 ms โ no change โ
โ QQuery 21 โ 19.80 / 20.12 ยฑ0.23 / 20.50 ms โ 19.50 / 20.26 ยฑ0.82 / 21.71 ms โ no change โ
โ QQuery 22 โ 499.21 / 501.74 ยฑ2.08 / 504.38 ms โ 488.14 / 491.29 ยฑ1.96 / 493.53 ms โ no change โ
โ QQuery 23 โ 896.12 / 906.04 ยฑ7.58 / 917.04 ms โ 880.38 / 894.65 ยฑ9.74 / 910.34 ms โ no change โ
โ QQuery 24 โ 389.59 / 393.04 ยฑ3.03 / 396.81 ms โ 384.52 / 388.22 ยฑ2.20 / 391.16 ms โ no change โ
โ QQuery 25 โ 341.09 / 343.83 ยฑ1.79 / 346.72 ms โ 342.00 / 343.83 ยฑ1.01 / 344.74 ms โ no change โ
โ QQuery 26 โ 81.83 / 83.62 ยฑ1.64 / 86.69 ms โ 80.75 / 83.11 ยฑ1.40 / 84.63 ms โ no change โ
โ QQuery 27 โ 7.14 / 7.38 ยฑ0.17 / 7.62 ms โ 7.52 / 7.81 ยฑ0.27 / 8.26 ms โ 1.06x slower โ
โ QQuery 28 โ 149.43 / 151.27 ยฑ1.50 / 153.46 ms โ 149.28 / 151.24 ยฑ1.85 / 154.70 ms โ no change โ
โ QQuery 29 โ 281.45 / 284.39 ยฑ1.95 / 287.00 ms โ 283.20 / 286.02 ยฑ2.17 / 288.56 ms โ no change โ
โ QQuery 30 โ 44.34 / 45.52 ยฑ0.77 / 46.50 ms โ 45.25 / 46.63 ยฑ1.44 / 49.25 ms โ no change โ
โ QQuery 31 โ 170.94 / 173.29 ยฑ1.70 / 175.59 ms โ 170.92 / 173.11 ยฑ2.00 / 176.49 ms โ no change โ
โ QQuery 32 โ 57.34 / 59.11 ยฑ1.58 / 61.47 ms โ 57.52 / 57.93 ยฑ0.29 / 58.26 ms โ no change โ
โ QQuery 33 โ 139.96 / 141.60 ยฑ1.24 / 142.95 ms โ 141.75 / 143.46 ยฑ1.38 / 145.41 ms โ no change โ
โ QQuery 34 โ 7.08 / 7.52 ยฑ0.34 / 7.95 ms โ 7.06 / 7.29 ยฑ0.28 / 7.77 ms โ no change โ
โ QQuery 35 โ 108.05 / 108.96 ยฑ0.74 / 110.14 ms โ 108.85 / 110.31 ยฑ1.21 / 112.19 ms โ no change โ
โ QQuery 36 โ 6.58 / 6.85 ยฑ0.26 / 7.27 ms โ 6.68 / 7.28 ยฑ0.64 / 8.51 ms โ 1.06x slower โ
โ QQuery 37 โ 8.43 / 8.84 ยฑ0.30 / 9.27 ms โ 8.35 / 8.78 ยฑ0.31 / 9.16 ms โ no change โ
โ QQuery 38 โ 82.84 / 88.42 ยฑ4.99 / 97.48 ms โ 85.86 / 87.88 ยฑ2.45 / 92.53 ms โ no change โ
โ QQuery 39 โ 125.83 / 129.92 ยฑ2.49 / 133.35 ms โ 127.50 / 129.48 ยฑ2.01 / 133.21 ms โ no change โ
โ QQuery 40 โ 113.15 / 117.68 ยฑ6.33 / 130.18 ms โ 111.18 / 115.19 ยฑ4.87 / 124.78 ms โ no change โ
โ QQuery 41 โ 14.79 / 16.81 ยฑ1.53 / 18.54 ms โ 15.37 / 16.86 ยฑ0.98 / 18.22 ms โ no change โ
โ QQuery 42 โ 107.28 / 109.43 ยฑ1.83 / 111.70 ms โ 107.90 / 110.09 ยฑ1.90 / 113.27 ms โ no change โ
โ QQuery 43 โ 6.33 / 6.62 ยฑ0.29 / 7.14 ms โ 6.03 / 6.20 ยฑ0.16 / 6.49 ms โ +1.07x faster โ
โ QQuery 44 โ 12.28 / 12.71 ยฑ0.71 / 14.13 ms โ 11.89 / 12.00 ยฑ0.07 / 12.08 ms โ +1.06x faster โ
โ QQuery 45 โ 51.00 / 52.39 ยฑ1.21 / 54.09 ms โ 51.17 / 51.77 ยฑ0.58 / 52.52 ms โ no change โ
โ QQuery 46 โ 8.72 / 9.00 ยฑ0.36 / 9.70 ms โ 8.42 / 8.91 ยฑ0.30 / 9.23 ms โ no change โ
โ QQuery 47 โ 742.26 / 758.26 ยฑ9.03 / 768.00 ms โ 736.36 / 739.50 ยฑ2.85 / 742.97 ms โ no change โ
โ QQuery 48 โ 286.20 / 295.17 ยฑ6.46 / 304.30 ms โ 294.46 / 298.52 ยฑ3.37 / 303.21 ms โ no change โ
โ QQuery 49 โ 252.49 / 255.57 ยฑ2.19 / 258.02 ms โ 252.40 / 255.58 ยฑ3.18 / 261.01 ms โ no change โ
โ QQuery 50 โ 220.08 / 227.22 ยฑ6.29 / 237.90 ms โ 224.18 / 228.93 ยฑ4.49 / 236.45 ms โ no change โ
โ QQuery 51 โ 180.50 / 184.69 ยฑ2.21 / 186.95 ms โ 182.56 / 184.51 ยฑ1.41 / 186.02 ms โ no change โ
โ QQuery 52 โ 107.44 / 108.63 ยฑ0.74 / 109.48 ms โ 106.75 / 108.28 ยฑ1.25 / 110.29 ms โ no change โ
โ QQuery 53 โ 103.70 / 104.51 ยฑ0.91 / 106.23 ms โ 103.42 / 104.21 ยฑ0.85 / 105.85 ms โ no change โ
โ QQuery 54 โ 148.46 / 149.54 ยฑ0.97 / 150.97 ms โ 145.04 / 147.97 ยฑ1.92 / 150.69 ms โ no change โ
โ QQuery 55 โ 106.97 / 109.30 ยฑ1.73 / 111.94 ms โ 107.58 / 108.69 ยฑ1.32 / 111.13 ms โ no change โ
โ QQuery 56 โ 142.17 / 143.91 ยฑ1.42 / 146.48 ms โ 142.06 / 143.46 ยฑ1.12 / 145.30 ms โ no change โ
โ QQuery 57 โ 174.03 / 175.98 ยฑ1.97 / 178.66 ms โ 174.14 / 176.40 ยฑ1.27 / 177.55 ms โ no change โ
โ QQuery 58 โ 290.80 / 297.69 ยฑ5.66 / 305.30 ms โ 287.33 / 295.78 ยฑ7.53 / 309.91 ms โ no change โ
โ QQuery 59 โ 199.34 / 200.73 ยฑ1.02 / 201.81 ms โ 198.61 / 200.65 ยฑ1.61 / 203.18 ms โ no change โ
โ QQuery 60 โ 143.16 / 144.22 ยฑ0.85 / 145.24 ms โ 145.24 / 147.14 ยฑ0.98 / 148.05 ms โ no change โ
โ QQuery 61 โ 13.61 / 13.75 ยฑ0.07 / 13.82 ms โ 13.38 / 13.61 ยฑ0.24 / 14.00 ms โ no change โ
โ QQuery 62 โ 930.91 / 947.14 ยฑ20.00 / 983.77 ms โ 904.60 / 928.63 ยฑ19.27 / 959.77 ms โ no change โ
โ QQuery 63 โ 103.75 / 104.94 ยฑ0.98 / 106.30 ms โ 105.00 / 108.43 ยฑ3.93 / 115.88 ms โ no change โ
โ QQuery 64 โ 695.36 / 701.95 ยฑ4.24 / 706.16 ms โ 690.81 / 697.14 ยฑ3.57 / 701.07 ms โ no change โ
โ QQuery 65 โ 256.00 / 260.69 ยฑ2.79 / 264.26 ms โ 258.47 / 261.05 ยฑ3.53 / 267.67 ms โ no change โ
โ QQuery 66 โ 247.70 / 258.32 ยฑ11.00 / 278.46 ms โ 244.24 / 259.43 ยฑ8.71 / 268.12 ms โ no change โ
โ QQuery 67 โ 310.23 / 319.45 ยฑ8.60 / 335.26 ms โ 314.49 / 318.12 ยฑ4.36 / 326.61 ms โ no change โ
โ QQuery 68 โ 8.98 / 10.17 ยฑ0.74 / 10.96 ms โ 8.83 / 9.27 ยฑ0.51 / 10.24 ms โ +1.10x faster โ
โ QQuery 69 โ 100.45 / 102.66 ยฑ1.54 / 104.20 ms โ 103.57 / 104.72 ยฑ1.23 / 106.84 ms โ no change โ
โ QQuery 70 โ 349.90 / 355.71 ยฑ5.41 / 365.55 ms โ 325.80 / 343.16 ยฑ10.46 / 357.17 ms โ no change โ
โ QQuery 71 โ 137.75 / 139.15 ยฑ2.02 / 143.10 ms โ 134.87 / 137.65 ยฑ1.48 / 139.24 ms โ no change โ
โ QQuery 72 โ 630.11 / 633.68 ยฑ4.45 / 641.45 ms โ 619.05 / 625.96 ยฑ5.15 / 632.12 ms โ no change โ
โ QQuery 73 โ 6.77 / 7.63 ยฑ0.56 / 8.38 ms โ 6.83 / 7.91 ยฑ0.64 / 8.78 ms โ no change โ
โ QQuery 74 โ 616.62 / 628.01 ยฑ7.53 / 639.68 ms โ 609.46 / 615.14 ยฑ5.29 / 624.48 ms โ no change โ
โ QQuery 75 โ 275.85 / 281.20 ยฑ3.17 / 285.70 ms โ 277.11 / 279.52 ยฑ1.61 / 282.16 ms โ no change โ
โ QQuery 76 โ 133.93 / 134.48 ยฑ0.60 / 135.21 ms โ 132.22 / 134.01 ยฑ1.43 / 135.70 ms โ no change โ
โ QQuery 77 โ 188.67 / 191.03 ยฑ1.57 / 193.59 ms โ 189.99 / 192.60 ยฑ1.45 / 194.24 ms โ no change โ
โ QQuery 78 โ 343.76 / 346.13 ยฑ2.30 / 349.64 ms โ 347.63 / 350.01 ยฑ2.12 / 353.52 ms โ no change โ
โ QQuery 79 โ 235.06 / 237.98 ยฑ1.84 / 240.34 ms โ 234.23 / 238.55 ยฑ2.86 / 242.31 ms โ no change โ
โ QQuery 80 โ 323.21 / 324.81 ยฑ1.83 / 328.02 ms โ 324.09 / 328.52 ยฑ2.25 / 329.93 ms โ no change โ
โ QQuery 81 โ 26.86 / 27.23 ยฑ0.55 / 28.33 ms โ 28.00 / 28.35 ยฑ0.27 / 28.81 ms โ no change โ
โ QQuery 82 โ 199.98 / 201.87 ยฑ1.65 / 204.61 ms โ 200.89 / 204.85 ยฑ2.60 / 207.42 ms โ no change โ
โ QQuery 83 โ 39.10 / 39.79 ยฑ0.91 / 41.52 ms โ 39.47 / 41.29 ยฑ1.37 / 43.64 ms โ no change โ
โ QQuery 84 โ 49.11 / 49.76 ยฑ0.51 / 50.60 ms โ 48.59 / 49.70 ยฑ1.09 / 51.75 ms โ no change โ
โ QQuery 85 โ 149.33 / 151.46 ยฑ1.62 / 153.73 ms โ 149.93 / 151.17 ยฑ0.73 / 152.22 ms โ no change โ
โ QQuery 86 โ 39.76 / 40.52 ยฑ1.07 / 42.63 ms โ 38.81 / 40.50 ยฑ1.39 / 42.95 ms โ no change โ
โ QQuery 87 โ 84.77 / 90.03 ยฑ4.07 / 97.26 ms โ 87.25 / 89.65 ยฑ3.53 / 96.65 ms โ no change โ
โ QQuery 88 โ 100.11 / 105.86 ยฑ9.26 / 124.21 ms โ 100.51 / 101.84 ยฑ1.08 / 103.68 ms โ no change โ
โ QQuery 89 โ 119.37 / 120.80 ยฑ1.06 / 122.23 ms โ 118.65 / 120.76 ยฑ1.49 / 122.49 ms โ no change โ
โ QQuery 90 โ 24.16 / 24.64 ยฑ0.33 / 25.18 ms โ 22.99 / 24.37 ยฑ0.95 / 25.89 ms โ no change โ
โ QQuery 91 โ 61.90 / 64.19 ยฑ1.25 / 65.40 ms โ 64.36 / 65.44 ยฑ0.84 / 66.65 ms โ no change โ
โ QQuery 92 โ 58.06 / 58.88 ยฑ0.58 / 59.87 ms โ 58.60 / 59.69 ยฑ0.77 / 61.00 ms โ no change โ
โ QQuery 93 โ 188.56 / 190.81 ยฑ1.58 / 193.27 ms โ 186.99 / 189.55 ยฑ1.54 / 191.64 ms โ no change โ
โ QQuery 94 โ 61.23 / 62.49 ยฑ0.72 / 63.21 ms โ 62.05 / 62.72 ยฑ0.50 / 63.34 ms โ no change โ
โ QQuery 95 โ 128.83 / 129.74 ยฑ1.02 / 131.67 ms โ 128.56 / 131.42 ยฑ1.79 / 134.05 ms โ no change โ
โ QQuery 96 โ 74.30 / 75.22 ยฑ0.84 / 76.72 ms โ 74.43 / 75.10 ยฑ0.36 / 75.40 ms โ no change โ
โ QQuery 97 โ 125.22 / 127.67 ยฑ1.91 / 129.89 ms โ 125.88 / 127.23 ยฑ0.74 / 127.96 ms โ no change โ
โ QQuery 98 โ 155.91 / 158.04 ยฑ1.86 / 160.71 ms โ 154.73 / 156.09 ยฑ1.11 / 158.10 ms โ no change โ
โ QQuery 99 โ 10772.64 / 10832.68 ยฑ31.97 / 10865.66 ms โ 10761.14 / 10805.33 ยฑ37.50 / 10860.70 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 31934.95ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 31794.04ms โ
โ Average Time (HEAD) โ 322.58ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 321.15ms โ
โ Queries Faster โ 4 โ
โ Queries Slower โ 3 โ
โ Queries with No Change โ 92 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 5.2 GiB |
| Avg memory | 4.5 GiB |
| CPU user | 263.9s |
| CPU sys | 17.3s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 159.3s |
| Peak memory | 5.5 GiB |
| Avg memory | 4.5 GiB |
| CPU user | 263.0s |
| CPU sys | 17.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4292852354
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4292778636)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.23 / 4.40 ยฑ6.29 / 16.98 ms โ 1.19 / 4.34 ยฑ6.21 / 16.77 ms โ no change โ
โ QQuery 1 โ 14.40 / 14.52 ยฑ0.14 / 14.78 ms โ 14.35 / 14.77 ยฑ0.29 / 15.26 ms โ no change โ
โ QQuery 2 โ 43.84 / 43.92 ยฑ0.05 / 44.00 ms โ 43.94 / 44.17 ยฑ0.22 / 44.52 ms โ no change โ
โ QQuery 3 โ 39.83 / 41.60 ยฑ2.01 / 45.39 ms โ 40.75 / 41.25 ยฑ0.48 / 41.94 ms โ no change โ
โ QQuery 4 โ 279.94 / 285.22 ยฑ5.02 / 293.70 ms โ 282.29 / 290.34 ยฑ5.53 / 298.58 ms โ no change โ
โ QQuery 5 โ 329.48 / 332.50 ยฑ3.14 / 337.47 ms โ 334.94 / 337.55 ยฑ1.54 / 339.59 ms โ no change โ
โ QQuery 6 โ 5.68 / 6.92 ยฑ1.08 / 8.83 ms โ 5.79 / 6.48 ยฑ0.65 / 7.35 ms โ +1.07x faster โ
โ QQuery 7 โ 16.36 / 17.17 ยฑ0.50 / 17.68 ms โ 17.04 / 17.69 ยฑ0.96 / 19.59 ms โ no change โ
โ QQuery 8 โ 401.60 / 408.43 ยฑ4.40 / 414.49 ms โ 404.90 / 412.07 ยฑ7.42 / 422.87 ms โ no change โ
โ QQuery 9 โ 612.77 / 633.33 ยฑ10.76 / 642.39 ms โ 555.26 / 564.38 ยฑ6.23 / 573.07 ms โ +1.12x faster โ
โ QQuery 10 โ 91.14 / 94.42 ยฑ3.48 / 101.12 ms โ 89.80 / 91.90 ยฑ1.27 / 93.64 ms โ no change โ
โ QQuery 11 โ 104.08 / 105.11 ยฑ0.82 / 106.37 ms โ 100.48 / 101.85 ยฑ0.84 / 103.02 ms โ no change โ
โ QQuery 12 โ 329.51 / 334.02 ยฑ6.47 / 346.89 ms โ 333.08 / 341.72 ยฑ9.79 / 354.05 ms โ no change โ
โ QQuery 13 โ 459.23 / 467.30 ยฑ9.05 / 484.29 ms โ 445.23 / 458.43 ยฑ8.95 / 470.00 ms โ no change โ
โ QQuery 14 โ 348.15 / 350.83 ยฑ1.76 / 352.82 ms โ 337.75 / 344.48 ยฑ3.64 / 348.09 ms โ no change โ
โ QQuery 15 โ 354.63 / 369.53 ยฑ12.50 / 384.20 ms โ 349.72 / 365.85 ยฑ18.39 / 401.20 ms โ no change โ
โ QQuery 16 โ 711.07 / 715.96 ยฑ2.85 / 719.83 ms โ 699.55 / 705.52 ยฑ4.15 / 712.43 ms โ no change โ
โ QQuery 17 โ 705.77 / 713.39 ยฑ5.58 / 720.82 ms โ 698.90 / 705.44 ยฑ5.48 / 714.35 ms โ no change โ
โ QQuery 18 โ 1415.26 / 1445.89 ยฑ21.95 / 1467.37 ms โ 1404.39 / 1458.54 ยฑ39.78 / 1522.83 ms โ no change โ
โ QQuery 19 โ 36.41 / 59.41 ยฑ26.71 / 97.77 ms โ 36.72 / 38.08 ยฑ1.21 / 39.86 ms โ +1.56x faster โ
โ QQuery 20 โ 717.91 / 745.69 ยฑ33.42 / 803.96 ms โ 716.59 / 738.84 ยฑ23.44 / 781.78 ms โ no change โ
โ QQuery 21 โ 761.82 / 765.99 ยฑ4.12 / 773.91 ms โ 764.30 / 769.65 ยฑ3.06 / 772.97 ms โ no change โ
โ QQuery 22 โ 1131.08 / 1138.12 ยฑ6.78 / 1150.95 ms โ 1138.49 / 1140.37 ยฑ2.21 / 1144.60 ms โ no change โ
โ QQuery 23 โ 3098.06 / 3133.77 ยฑ22.48 / 3164.92 ms โ 3069.51 / 3086.81 ยฑ10.69 / 3098.80 ms โ no change โ
โ QQuery 24 โ 100.78 / 104.13 ยฑ3.30 / 110.30 ms โ 99.93 / 103.32 ยฑ3.12 / 109.09 ms โ no change โ
โ QQuery 25 โ 139.17 / 142.76 ยฑ2.42 / 146.60 ms โ 139.45 / 156.99 ยฑ28.07 / 212.99 ms โ 1.10x slower โ
โ QQuery 26 โ 101.83 / 103.25 ยฑ1.17 / 104.67 ms โ 100.59 / 103.49 ยฑ1.66 / 105.31 ms โ no change โ
โ QQuery 27 โ 846.95 / 850.85 ยฑ3.43 / 855.79 ms โ 849.55 / 866.66 ยฑ23.90 / 913.48 ms โ no change โ
โ QQuery 28 โ 3219.71 / 3241.80 ยฑ11.99 / 3251.94 ms โ 3242.87 / 3259.45 ยฑ11.96 / 3276.70 ms โ no change โ
โ QQuery 29 โ 49.14 / 53.67 ยฑ3.65 / 59.31 ms โ 50.73 / 54.64 ยฑ2.98 / 58.51 ms โ no change โ
โ QQuery 30 โ 358.61 / 368.43 ยฑ5.42 / 373.87 ms โ 358.84 / 367.29 ยฑ7.41 / 380.90 ms โ no change โ
โ QQuery 31 โ 368.28 / 373.30 ยฑ3.77 / 380.00 ms โ 368.60 / 384.40 ยฑ11.96 / 398.75 ms โ no change โ
โ QQuery 32 โ 1136.02 / 1214.60 ยฑ50.73 / 1268.19 ms โ 1015.69 / 1038.43 ยฑ21.51 / 1079.05 ms โ +1.17x faster โ
โ QQuery 33 โ 1406.80 / 1437.86 ยฑ29.30 / 1493.26 ms โ 1420.88 / 1441.20 ยฑ17.72 / 1465.21 ms โ no change โ
โ QQuery 34 โ 1428.83 / 1444.18 ยฑ10.88 / 1460.68 ms โ 1440.13 / 1456.90 ยฑ13.73 / 1478.91 ms โ no change โ
โ QQuery 35 โ 378.11 / 386.37 ยฑ12.54 / 411.32 ms โ 379.41 / 388.70 ยฑ9.88 / 406.99 ms โ no change โ
โ QQuery 36 โ 109.78 / 119.91 ยฑ5.25 / 124.08 ms โ 115.02 / 121.88 ยฑ3.97 / 127.07 ms โ no change โ
โ QQuery 37 โ 48.00 / 49.61 ยฑ1.16 / 50.81 ms โ 48.46 / 50.24 ยฑ1.67 / 52.94 ms โ no change โ
โ QQuery 38 โ 74.67 / 77.26 ยฑ3.55 / 84.18 ms โ 75.72 / 76.71 ยฑ0.75 / 77.73 ms โ no change โ
โ QQuery 39 โ 208.21 / 213.85 ยฑ4.25 / 218.30 ms โ 209.96 / 217.15 ยฑ6.18 / 224.56 ms โ no change โ
โ QQuery 40 โ 23.84 / 24.80 ยฑ0.94 / 26.33 ms โ 20.55 / 24.05 ยฑ1.77 / 25.47 ms โ no change โ
โ QQuery 41 โ 19.49 / 22.34 ยฑ3.20 / 28.44 ms โ 18.73 / 21.02 ยฑ1.16 / 21.91 ms โ +1.06x faster โ
โ QQuery 42 โ 19.05 / 19.54 ยฑ0.44 / 20.30 ms โ 19.59 / 19.84 ยฑ0.25 / 20.29 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 22475.96ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 22232.90ms โ
โ Average Time (HEAD) โ 522.70ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 517.04ms โ
โ Queries Faster โ 5 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 37 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 115.0s |
| Peak memory | 37.1 GiB |
| Avg memory | 27.4 GiB |
| CPU user | 1059.3s |
| CPU sys | 95.4s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 115.0s |
| Peak memory | 38.4 GiB |
| Avg memory | 28.2 GiB |
| CPU user | 1051.6s |
| CPU sys | 89.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230786454
run benchmark tpch tpcds clickbench_partitiomed clickbemch_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230789957
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4230786454)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4230786454-1099-gqfv6 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (f0b2a4aab055f557acc50f4e919d92ab57f18c35) to 16e578d (merge-base) [diff](https://github.com/apache/datafusion/compare/16e578dfdd28c8bffc6a21f54164b889e79bf75c..f0b2a4aab055f557acc50f4e919d92ab57f18c35) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4265647324
@Dandandan , could you re run the benchmarks please ? I do see a consistent speedup with Q9 and seems like this PR is ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4242391331
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4242372936)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4242372936-1212-4fj8m 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (8d60834b3b09bad5093b74d0a32799e3dc98dd0a) to a13c23d (merge-base) [diff](https://github.com/apache/datafusion/compare/a13c23df67c218e45342b80238cae490609d8f3d..8d60834b3b09bad5093b74d0a32799e3dc98dd0a) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266255778
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266245112)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4266245112-1410-5h7df 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (617e79032b8930cd89a52ec0f27e761d5b71c5c5) to 5a427cb (merge-base) [diff](https://github.com/apache/datafusion/compare/5a427cb55d8630b51b9590a25b369b434611975e..617e79032b8930cd89a52ec0f27e761d5b71c5c5) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3098441567
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ ArrayRef, AsArray, BooleanArray, Int64Array, ListArray, PrimitiveArray,
+};
+use arrow::buffer::OffsetBuffer;
+use arrow::datatypes::{ArrowPrimitiveType, Field};
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+use crate::aggregate::groups_accumulator::accumulate::accumulate;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ seen: HashSet<(usize, T::Native), RandomState>,
+ counts: Vec<i64>,
+}
+
+impl<T: ArrowPrimitiveType> PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ pub fn new() -> Self {
+ Self {
+ seen: HashSet::default(),
+ counts: Vec::new(),
+ }
+ }
+}
+
+impl<T: ArrowPrimitiveType> Default for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ArrowPrimitiveType + Send + std::fmt::Debug> GroupsAccumulator
+ for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn update_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> datafusion_common::Result<()> {
+ debug_assert_eq!(values.len(), 1);
+ self.counts.resize(total_num_groups, 0);
+ let arr = values[0].as_primitive::<T>();
+ accumulate(group_indices, arr, opt_filter, |group_idx, value| {
+ if self.seen.insert((group_idx, value)) {
+ self.counts[group_idx] += 1;
+ }
+ });
+ Ok(())
+ }
+
+ fn evaluate(&mut self, emit_to: EmitTo) -> datafusion_common::Result<ArrayRef> {
+ let counts = emit_to.take_needed(&mut self.counts);
+
+ match emit_to {
+ EmitTo::All => {
+ self.seen.clear();
+ }
+ EmitTo::First(n) => {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx >= n {
+ remaining.insert((group_idx - n, value));
+ }
+ }
+ self.seen = remaining;
+ }
+ }
+
+ Ok(Arc::new(Int64Array::from(counts)))
+ }
+
+ fn state(&mut self, emit_to: EmitTo) -> datafusion_common::Result<Vec<ArrayRef>> {
+ let num_emitted = match emit_to {
+ EmitTo::All => self.counts.len(),
+ EmitTo::First(n) => n,
+ };
+
+ let mut group_values: Vec<Vec<T::Native>> = vec![Vec::new(); num_emitted];
Review Comment:
Something like this (Claude-generated) probably is faster?
```
// Prefix-sum counts[..num_emitted] into offsets
let mut offsets = Vec::with_capacity(num_emitted + 1);
offsets.push(0i32);
let mut total = 0i32;
for &c in &self.counts[..num_emitted] {
total += c as i32;
offsets.push(total);
}
let mut all_values = vec![T::Native::default(); total as usize];
let mut cursors: Vec<i32> = offsets[..num_emitted].to_vec();
if matches!(emit_to, EmitTo::All) {
for (group_idx, value) in self.seen.drain() {
let pos = cursors[group_idx] as usize;
all_values[pos] = value;
cursors[group_idx] += 1;
}
self.counts.clear();
} else {
let mut remaining = HashSet::default();
for (group_idx, value) in self.seen.drain() {
if group_idx < num_emitted {
let pos = cursors[group_idx] as usize;
all_values[pos] = value;
cursors[group_idx] += 1;
} else {
remaining.insert((group_idx - num_emitted, value));
}
}
self.seen = remaining;
let _ = emit_to.take_needed(&mut self.counts);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4257458310
@Dandandan , requesting benchmarks . I added `hashtable` approach and a vector to keep `evaluate` finishing in constant time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3098481887
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ ArrayRef, AsArray, BooleanArray, Int64Array, ListArray, PrimitiveArray,
+};
+use arrow::buffer::OffsetBuffer;
+use arrow::datatypes::{ArrowPrimitiveType, Field};
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+use crate::aggregate::groups_accumulator::accumulate::accumulate;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ seen: HashSet<(usize, T::Native), RandomState>,
+ counts: Vec<i64>,
+}
+
+impl<T: ArrowPrimitiveType> PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ pub fn new() -> Self {
+ Self {
+ seen: HashSet::default(),
+ counts: Vec::new(),
+ }
+ }
+}
+
+impl<T: ArrowPrimitiveType> Default for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ArrowPrimitiveType + Send + std::fmt::Debug> GroupsAccumulator
+ for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn update_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> datafusion_common::Result<()> {
+ debug_assert_eq!(values.len(), 1);
+ self.counts.resize(total_num_groups, 0);
+ let arr = values[0].as_primitive::<T>();
+ accumulate(group_indices, arr, opt_filter, |group_idx, value| {
+ if self.seen.insert((group_idx, value)) {
+ self.counts[group_idx] += 1;
+ }
+ });
+ Ok(())
+ }
+
+ fn evaluate(&mut self, emit_to: EmitTo) -> datafusion_common::Result<ArrayRef> {
+ let counts = emit_to.take_needed(&mut self.counts);
+
+ match emit_to {
+ EmitTo::All => {
+ self.seen.clear();
+ }
+ EmitTo::First(n) => {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx >= n {
+ remaining.insert((group_idx - n, value));
+ }
+ }
+ self.seen = remaining;
+ }
+ }
+
+ Ok(Arc::new(Int64Array::from(counts)))
+ }
+
+ fn state(&mut self, emit_to: EmitTo) -> datafusion_common::Result<Vec<ArrayRef>> {
+ let num_emitted = match emit_to {
+ EmitTo::All => self.counts.len(),
+ EmitTo::First(n) => n,
+ };
+
+ let mut group_values: Vec<Vec<T::Native>> = vec![Vec::new(); num_emitted];
+
+ if matches!(emit_to, EmitTo::All) {
+ for (group_idx, value) in self.seen.drain() {
+ group_values[group_idx].push(value);
+ }
+ self.counts.clear();
+ } else {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx < num_emitted {
+ group_values[group_idx].push(value);
+ } else {
+ remaining.insert((group_idx - num_emitted, value));
+ }
+ }
+ self.seen = remaining;
+ let _ = emit_to.take_needed(&mut self.counts);
+ }
+
+ let mut offsets = vec![0i32];
+ let mut all_values = Vec::new();
+ for values in &group_values {
+ all_values.extend(values.iter().copied());
+ offsets.push(all_values.len() as i32);
+ }
+
+ let values_array = Arc::new(PrimitiveArray::<T>::from_iter_values(all_values));
Review Comment:
```suggestion
let values_array = Arc::new(PrimitiveArray::<T>::new(ScalarBuffer::from(all_values), None));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4256932869
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4256919523)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4256919523-1333-vnj7z 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (b9dcf85869706b90288153c487aaebd186cf30c8) to 2818abb (merge-base) [diff](https://github.com/apache/datafusion/compare/2818abb8a10bf8b001669c15cf597f9ddfb86ed1..b9dcf85869706b90288153c487aaebd186cf30c8) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4292788900
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4292778636)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4292778636-1727-cll4s 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (54ba4a1026fb743fb5fd9198889aec3bf92ea80c) to 90a8117 (merge-base) [diff](https://github.com/apache/datafusion/compare/90a8117be6c54b2b188149e14ab339f2a9608dff..54ba4a1026fb743fb5fd9198889aec3bf92ea80c) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #21561:
URL: https://github.com/apache/datafusion/pull/21561
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4262117613
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4261984688)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.23 / 4.54 ยฑ6.43 / 17.40 ms โ 1.23 / 4.50 ยฑ6.40 / 17.29 ms โ no change โ
โ QQuery 1 โ 14.38 / 14.75 ยฑ0.27 / 15.21 ms โ 14.27 / 14.86 ยฑ0.33 / 15.21 ms โ no change โ
โ QQuery 2 โ 44.02 / 44.29 ยฑ0.25 / 44.73 ms โ 44.10 / 44.68 ยฑ0.29 / 44.90 ms โ no change โ
โ QQuery 3 โ 41.72 / 44.55 ยฑ3.12 / 49.52 ms โ 44.16 / 47.21 ยฑ2.26 / 50.45 ms โ 1.06x slower โ
โ QQuery 4 โ 293.41 / 299.06 ยฑ5.71 / 309.36 ms โ 299.18 / 307.50 ยฑ9.41 / 325.41 ms โ no change โ
โ QQuery 5 โ 350.31 / 354.48 ยฑ3.63 / 359.15 ms โ 349.63 / 354.16 ยฑ3.56 / 360.12 ms โ no change โ
โ QQuery 6 โ 5.34 / 6.66 ยฑ0.90 / 7.61 ms โ 5.45 / 6.30 ยฑ1.14 / 8.48 ms โ +1.06x faster โ
โ QQuery 7 โ 17.00 / 17.66 ยฑ0.59 / 18.47 ms โ 16.69 / 17.71 ยฑ0.80 / 18.81 ms โ no change โ
โ QQuery 8 โ 429.13 / 434.40 ยฑ3.47 / 439.23 ms โ 432.05 / 437.94 ยฑ4.88 / 446.16 ms โ no change โ
โ QQuery 9 โ 640.75 / 661.55 ยฑ18.57 / 692.50 ms โ 591.43 / 597.45 ยฑ4.62 / 603.98 ms โ +1.11x faster โ
โ QQuery 10 โ 92.78 / 94.89 ยฑ2.52 / 99.76 ms โ 90.94 / 93.64 ยฑ2.17 / 96.81 ms โ no change โ
โ QQuery 11 โ 104.43 / 106.18 ยฑ1.10 / 107.70 ms โ 103.96 / 106.35 ยฑ1.83 / 109.19 ms โ no change โ
โ QQuery 12 โ 341.29 / 345.49 ยฑ3.38 / 349.38 ms โ 351.17 / 358.96 ยฑ5.50 / 367.25 ms โ no change โ
โ QQuery 13 โ 476.48 / 491.15 ยฑ10.69 / 506.66 ms โ 470.25 / 479.46 ยฑ7.63 / 489.41 ms โ no change โ
โ QQuery 14 โ 347.84 / 355.56 ยฑ5.36 / 364.48 ms โ 344.35 / 351.08 ยฑ5.97 / 361.59 ms โ no change โ
โ QQuery 15 โ 355.38 / 369.89 ยฑ18.54 / 406.44 ms โ 350.14 / 369.18 ยฑ13.81 / 391.91 ms โ no change โ
โ QQuery 16 โ 720.47 / 757.69 ยฑ28.01 / 797.20 ms โ 720.81 / 727.14 ยฑ6.09 / 737.82 ms โ no change โ
โ QQuery 17 โ 724.08 / 732.29 ยฑ9.35 / 747.95 ms โ 710.56 / 725.53 ยฑ10.55 / 739.21 ms โ no change โ
โ QQuery 18 โ 1421.97 / 1477.62 ยฑ61.71 / 1594.09 ms โ 1460.63 / 1485.28 ยฑ19.66 / 1519.09 ms โ no change โ
โ QQuery 19 โ 37.06 / 47.35 ยฑ16.60 / 80.35 ms โ 38.20 / 39.56 ยฑ1.56 / 41.62 ms โ +1.20x faster โ
โ QQuery 20 โ 716.13 / 738.18 ยฑ22.34 / 773.91 ms โ 719.85 / 731.95 ยฑ18.25 / 767.46 ms โ no change โ
โ QQuery 21 โ 762.21 / 765.47 ยฑ3.11 / 770.39 ms โ 762.31 / 767.86 ยฑ4.26 / 773.98 ms โ no change โ
โ QQuery 22 โ 1134.64 / 1140.60 ยฑ3.75 / 1145.65 ms โ 1138.06 / 1141.38 ยฑ2.92 / 1145.50 ms โ no change โ
โ QQuery 23 โ 3141.67 / 3148.43 ยฑ5.36 / 3157.41 ms โ 3109.00 / 3127.34 ยฑ14.91 / 3154.34 ms โ no change โ
โ QQuery 24 โ 100.29 / 103.09 ยฑ2.45 / 107.03 ms โ 102.52 / 103.76 ยฑ0.72 / 104.63 ms โ no change โ
โ QQuery 25 โ 137.07 / 142.18 ยฑ2.83 / 145.33 ms โ 140.42 / 143.06 ยฑ2.05 / 145.96 ms โ no change โ
โ QQuery 26 โ 98.07 / 102.93 ยฑ2.83 / 106.53 ms โ 104.31 / 105.15 ยฑ0.84 / 106.66 ms โ no change โ
โ QQuery 27 โ 848.28 / 852.36 ยฑ3.00 / 857.63 ms โ 848.87 / 853.36 ยฑ4.42 / 861.09 ms โ no change โ
โ QQuery 28 โ 3267.27 / 3292.75 ยฑ16.78 / 3319.76 ms โ 3261.77 / 3286.93 ยฑ19.38 / 3317.34 ms โ no change โ
โ QQuery 29 โ 51.20 / 55.38 ยฑ4.36 / 61.83 ms โ 50.50 / 55.18 ยฑ5.34 / 62.72 ms โ no change โ
โ QQuery 30 โ 367.44 / 376.10 ยฑ5.88 / 385.67 ms โ 367.58 / 374.41 ยฑ3.60 / 377.63 ms โ no change โ
โ QQuery 31 โ 377.76 / 393.36 ยฑ15.21 / 420.83 ms โ 363.20 / 380.88 ยฑ15.22 / 408.04 ms โ no change โ
โ QQuery 32 โ 1068.53 / 1176.51 ยฑ111.51 / 1353.41 ms โ 1050.29 / 1060.57 ยฑ11.15 / 1080.16 ms โ +1.11x faster โ
โ QQuery 33 โ 1465.90 / 1480.38 ยฑ14.42 / 1507.33 ms โ 1486.73 / 1491.37 ยฑ4.48 / 1498.84 ms โ no change โ
โ QQuery 34 โ 1467.66 / 1482.19 ยฑ9.34 / 1494.00 ms โ 1515.84 / 1541.33 ยฑ17.23 / 1559.79 ms โ no change โ
โ QQuery 35 โ 398.01 / 401.08 ยฑ3.68 / 407.86 ms โ 396.47 / 400.08 ยฑ3.59 / 405.14 ms โ no change โ
โ QQuery 36 โ 119.72 / 122.50 ยฑ1.74 / 124.42 ms โ 121.19 / 122.81 ยฑ1.57 / 125.24 ms โ no change โ
โ QQuery 37 โ 48.21 / 50.56 ยฑ2.24 / 54.28 ms โ 49.19 / 50.59 ยฑ1.29 / 52.59 ms โ no change โ
โ QQuery 38 โ 74.15 / 77.32 ยฑ1.61 / 78.62 ms โ 75.35 / 78.13 ยฑ2.38 / 81.69 ms โ no change โ
โ QQuery 39 โ 209.83 / 215.29 ยฑ3.73 / 219.60 ms โ 212.00 / 218.30 ยฑ4.50 / 223.28 ms โ no change โ
โ QQuery 40 โ 26.49 / 27.41 ยฑ0.87 / 28.48 ms โ 23.41 / 25.49 ยฑ1.39 / 27.79 ms โ +1.08x faster โ
โ QQuery 41 โ 21.61 / 22.45 ยฑ0.66 / 23.45 ms โ 19.87 / 21.90 ยฑ1.98 / 25.70 ms โ no change โ
โ QQuery 42 โ 19.74 / 20.29 ยฑ0.52 / 21.13 ms โ 19.32 / 20.11 ยฑ0.42 / 20.54 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 22846.88ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 22670.45ms โ
โ Average Time (HEAD) โ 531.32ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 527.22ms โ
โ Queries Faster โ 5 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 37 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 115.4s |
| Peak memory | 37.9 GiB |
| Avg memory | 28.0 GiB |
| CPU user | 1083.9s |
| CPU sys | 88.7s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 114.5s |
| Peak memory | 39.0 GiB |
| Avg memory | 31.2 GiB |
| CPU user | 1078.7s |
| CPU sys | 89.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230810743
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4230786454)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 6.67 / 7.10 ยฑ0.72 / 8.54 ms โ 6.64 / 7.05 ยฑ0.79 / 8.62 ms โ no change โ
โ QQuery 2 โ 145.52 / 146.06 ยฑ0.55 / 146.84 ms โ 143.94 / 144.92 ยฑ0.69 / 145.77 ms โ no change โ
โ QQuery 3 โ 113.10 / 113.85 ยฑ0.57 / 114.51 ms โ 113.42 / 114.24 ยฑ0.68 / 115.10 ms โ no change โ
โ QQuery 4 โ 1319.23 / 1330.98 ยฑ11.07 / 1349.57 ms โ 1298.35 / 1327.61 ยฑ27.90 / 1378.68 ms โ no change โ
โ QQuery 5 โ 172.99 / 174.44 ยฑ1.15 / 176.35 ms โ 171.54 / 172.70 ยฑ0.95 / 174.34 ms โ no change โ
โ QQuery 6 โ 851.09 / 863.36 ยฑ6.48 / 870.22 ms โ 843.12 / 868.04 ยฑ22.23 / 909.25 ms โ no change โ
โ QQuery 7 โ 340.10 / 341.59 ยฑ1.36 / 344.13 ms โ 338.82 / 343.62 ยฑ2.91 / 347.73 ms โ no change โ
โ QQuery 8 โ 115.78 / 117.00 ยฑ0.99 / 117.90 ms โ 116.26 / 116.66 ยฑ0.46 / 117.52 ms โ no change โ
โ QQuery 9 โ 101.00 / 104.78 ยฑ3.16 / 108.12 ms โ 100.33 / 104.53 ยฑ3.23 / 107.54 ms โ no change โ
โ QQuery 10 โ 104.75 / 106.03 ยฑ0.71 / 106.81 ms โ 107.03 / 107.46 ยฑ0.41 / 108.16 ms โ no change โ
โ QQuery 11 โ 899.26 / 911.99 ยฑ7.96 / 921.52 ms โ 894.74 / 951.23 ยฑ30.46 / 987.02 ms โ no change โ
โ QQuery 12 โ 44.96 / 46.16 ยฑ0.96 / 47.38 ms โ 44.51 / 46.69 ยฑ1.53 / 48.92 ms โ no change โ
โ QQuery 13 โ 399.04 / 400.05 ยฑ0.87 / 401.68 ms โ 401.80 / 404.66 ยฑ2.68 / 409.31 ms โ no change โ
โ QQuery 14 โ 997.99 / 1001.65 ยฑ2.57 / 1004.77 ms โ 988.85 / 997.22 ยฑ7.25 / 1009.32 ms โ no change โ
โ QQuery 15 โ 15.36 / 16.46 ยฑ0.84 / 17.84 ms โ 15.01 / 16.80 ยฑ1.12 / 18.28 ms โ no change โ
โ QQuery 16 โ 7.17 / 7.56 ยฑ0.36 / 8.03 ms โ 7.24 / 7.58 ยฑ0.35 / 8.02 ms โ no change โ
โ QQuery 17 โ 227.77 / 229.20 ยฑ0.94 / 230.58 ms โ 226.27 / 228.60 ยฑ1.81 / 231.10 ms โ no change โ
โ QQuery 18 โ 126.99 / 127.64 ยฑ0.65 / 128.49 ms โ 128.14 / 129.22 ยฑ1.14 / 131.43 ms โ no change โ
โ QQuery 19 โ 155.20 / 156.58 ยฑ1.07 / 157.69 ms โ 155.13 / 157.46 ยฑ1.54 / 159.74 ms โ no change โ
โ QQuery 20 โ 13.85 / 14.29 ยฑ0.26 / 14.59 ms โ 13.96 / 14.35 ยฑ0.41 / 14.90 ms โ no change โ
โ QQuery 21 โ 19.10 / 19.48 ยฑ0.33 / 19.88 ms โ 19.26 / 19.91 ยฑ0.52 / 20.65 ms โ no change โ
โ QQuery 22 โ 484.06 / 487.99 ยฑ2.62 / 492.04 ms โ 483.72 / 489.34 ยฑ3.39 / 493.61 ms โ no change โ
โ QQuery 23 โ 852.27 / 864.37 ยฑ7.39 / 873.54 ms โ 864.44 / 869.10 ยฑ5.15 / 878.77 ms โ no change โ
โ QQuery 24 โ 374.54 / 378.04 ยฑ3.53 / 384.26 ms โ 380.07 / 382.78 ยฑ3.03 / 387.21 ms โ no change โ
โ QQuery 25 โ 337.29 / 340.29 ยฑ2.27 / 343.81 ms โ 335.49 / 340.57 ยฑ3.40 / 346.01 ms โ no change โ
โ QQuery 26 โ 80.06 / 81.81 ยฑ2.20 / 86.02 ms โ 80.93 / 82.63 ยฑ1.79 / 86.05 ms โ no change โ
โ QQuery 27 โ 7.00 / 7.08 ยฑ0.05 / 7.14 ms โ 6.78 / 7.25 ยฑ0.41 / 7.98 ms โ no change โ
โ QQuery 28 โ 147.46 / 149.05 ยฑ0.84 / 149.85 ms โ 149.12 / 150.47 ยฑ1.22 / 152.38 ms โ no change โ
โ QQuery 29 โ 278.04 / 280.35 ยฑ1.45 / 282.24 ms โ 278.34 / 281.08 ยฑ1.50 / 282.75 ms โ no change โ
โ QQuery 30 โ 42.76 / 45.29 ยฑ1.85 / 48.14 ms โ 43.36 / 44.93 ยฑ1.35 / 47.38 ms โ no change โ
โ QQuery 31 โ 170.17 / 171.77 ยฑ1.08 / 173.20 ms โ 169.10 / 171.27 ยฑ1.54 / 172.88 ms โ no change โ
โ QQuery 32 โ 57.04 / 58.66 ยฑ1.27 / 60.86 ms โ 56.69 / 57.35 ยฑ0.70 / 58.22 ms โ no change โ
โ QQuery 33 โ 139.04 / 141.18 ยฑ1.80 / 143.48 ms โ 139.62 / 141.68 ยฑ2.04 / 145.24 ms โ no change โ
โ QQuery 34 โ 6.92 / 7.17 ยฑ0.22 / 7.58 ms โ 6.88 / 7.13 ยฑ0.26 / 7.61 ms โ no change โ
โ QQuery 35 โ 108.29 / 108.76 ยฑ0.68 / 110.08 ms โ 102.84 / 106.97 ยฑ2.28 / 109.24 ms โ no change โ
โ QQuery 36 โ 6.39 / 6.65 ยฑ0.26 / 7.15 ms โ 6.36 / 6.57 ยฑ0.29 / 7.14 ms โ no change โ
โ QQuery 37 โ 8.11 / 8.42 ยฑ0.23 / 8.69 ms โ 8.26 / 8.39 ยฑ0.12 / 8.54 ms โ no change โ
โ QQuery 38 โ 82.84 / 86.11 ยฑ3.37 / 92.29 ms โ 83.54 / 86.17 ยฑ3.15 / 92.17 ms โ no change โ
โ QQuery 39 โ 122.37 / 124.22 ยฑ1.69 / 127.37 ms โ 123.58 / 126.42 ยฑ2.09 / 128.87 ms โ no change โ
โ QQuery 40 โ 108.73 / 113.81 ยฑ8.15 / 130.04 ms โ 108.47 / 117.69 ยฑ8.07 / 132.57 ms โ no change โ
โ QQuery 41 โ 14.86 / 15.71 ยฑ0.76 / 17.12 ms โ 14.81 / 15.91 ยฑ1.00 / 17.42 ms โ no change โ
โ QQuery 42 โ 107.91 / 109.56 ยฑ0.99 / 110.87 ms โ 106.94 / 108.93 ยฑ1.61 / 111.41 ms โ no change โ
โ QQuery 43 โ 5.85 / 6.03 ยฑ0.18 / 6.37 ms โ 6.18 / 6.44 ยฑ0.23 / 6.74 ms โ 1.07x slower โ
โ QQuery 44 โ 11.44 / 11.69 ยฑ0.16 / 11.86 ms โ 11.94 / 12.67 ยฑ0.99 / 14.54 ms โ 1.08x slower โ
โ QQuery 45 โ 49.27 / 51.05 ยฑ0.98 / 52.14 ms โ 49.28 / 51.37 ยฑ1.57 / 53.21 ms โ no change โ
โ QQuery 46 โ 8.45 / 8.86 ยฑ0.45 / 9.73 ms โ 8.16 / 8.74 ยฑ0.55 / 9.73 ms โ no change โ
โ QQuery 47 โ 703.68 / 717.37 ยฑ7.63 / 724.32 ms โ 688.31 / 702.25 ยฑ10.98 / 718.99 ms โ no change โ
โ QQuery 48 โ 283.48 / 290.31 ยฑ4.54 / 297.33 ms โ 291.43 / 295.22 ยฑ2.69 / 299.51 ms โ no change โ
โ QQuery 49 โ 247.34 / 251.04 ยฑ2.12 / 253.23 ms โ 248.25 / 250.65 ยฑ1.66 / 253.40 ms โ no change โ
โ QQuery 50 โ 221.95 / 225.40 ยฑ3.91 / 232.82 ms โ 217.23 / 222.68 ยฑ3.36 / 227.71 ms โ no change โ
โ QQuery 51 โ 179.02 / 180.58 ยฑ1.30 / 182.02 ms โ 180.58 / 183.96 ยฑ2.91 / 188.56 ms โ no change โ
โ QQuery 52 โ 108.24 / 109.22 ยฑ0.78 / 110.44 ms โ 108.12 / 109.01 ยฑ0.73 / 110.21 ms โ no change โ
โ QQuery 53 โ 102.42 / 103.80 ยฑ1.05 / 105.21 ms โ 101.58 / 103.28 ยฑ1.38 / 105.02 ms โ no change โ
โ QQuery 54 โ 145.77 / 146.83 ยฑ0.87 / 148.30 ms โ 145.73 / 147.44 ยฑ1.20 / 149.27 ms โ no change โ
โ QQuery 55 โ 107.32 / 108.82 ยฑ0.89 / 109.56 ms โ 106.60 / 107.46 ยฑ0.71 / 108.29 ms โ no change โ
โ QQuery 56 โ 140.13 / 141.85 ยฑ1.54 / 143.77 ms โ 138.89 / 141.09 ยฑ1.37 / 142.88 ms โ no change โ
โ QQuery 57 โ 176.23 / 177.21 ยฑ0.90 / 178.69 ms โ 174.08 / 175.85 ยฑ1.38 / 178.30 ms โ no change โ
โ QQuery 58 โ 291.30 / 301.87 ยฑ7.33 / 313.62 ms โ 300.86 / 303.47 ยฑ2.65 / 308.42 ms โ no change โ
โ QQuery 59 โ 198.28 / 199.39 ยฑ0.99 / 201.25 ms โ 200.47 / 204.11 ยฑ3.12 / 209.38 ms โ no change โ
โ QQuery 60 โ 143.19 / 144.22 ยฑ0.78 / 145.10 ms โ 143.39 / 144.69 ยฑ0.98 / 146.32 ms โ no change โ
โ QQuery 61 โ 12.93 / 13.13 ยฑ0.21 / 13.51 ms โ 13.23 / 13.58 ยฑ0.34 / 14.17 ms โ no change โ
โ QQuery 62 โ 924.37 / 946.25 ยฑ22.24 / 986.54 ms โ 890.22 / 919.10 ยฑ18.71 / 938.71 ms โ no change โ
โ QQuery 63 โ 104.45 / 106.49 ยฑ2.48 / 111.29 ms โ 102.29 / 103.86 ยฑ1.04 / 105.10 ms โ no change โ
โ QQuery 64 โ 672.07 / 678.36 ยฑ4.09 / 684.89 ms โ 679.64 / 685.80 ยฑ4.04 / 691.82 ms โ no change โ
โ QQuery 65 โ 242.44 / 250.34 ยฑ4.95 / 257.89 ms โ 248.45 / 251.95 ยฑ3.53 / 258.50 ms โ no change โ
โ QQuery 66 โ 240.40 / 248.97 ยฑ5.78 / 257.76 ms โ 241.48 / 254.93 ยฑ9.99 / 266.99 ms โ no change โ
โ QQuery 67 โ 305.04 / 311.65 ยฑ7.58 / 326.41 ms โ 312.94 / 321.02 ยฑ7.59 / 332.20 ms โ no change โ
โ QQuery 68 โ 10.19 / 11.21 ยฑ0.73 / 12.39 ms โ 8.68 / 10.29 ยฑ1.69 / 13.12 ms โ +1.09x faster โ
โ QQuery 69 โ 101.77 / 103.42 ยฑ1.75 / 106.28 ms โ 103.24 / 104.80 ยฑ0.87 / 105.72 ms โ no change โ
โ QQuery 70 โ 336.64 / 347.16 ยฑ11.48 / 368.52 ms โ 320.54 / 346.05 ยฑ15.58 / 367.90 ms โ no change โ
โ QQuery 71 โ 133.27 / 136.10 ยฑ1.84 / 138.54 ms โ 135.97 / 138.89 ยฑ3.04 / 144.38 ms โ no change โ
โ QQuery 72 โ 611.94 / 617.05 ยฑ4.48 / 624.93 ms โ 596.60 / 611.64 ยฑ8.52 / 622.63 ms โ no change โ
โ QQuery 73 โ 6.60 / 7.92 ยฑ0.72 / 8.74 ms โ 7.03 / 7.94 ยฑ0.87 / 9.27 ms โ no change โ
โ QQuery 74 โ 564.08 / 567.88 ยฑ4.30 / 576.11 ms โ 553.35 / 563.86 ยฑ9.53 / 579.02 ms โ no change โ
โ QQuery 75 โ 273.98 / 275.96 ยฑ1.76 / 279.23 ms โ 275.31 / 277.50 ยฑ1.68 / 279.83 ms โ no change โ
โ QQuery 76 โ 130.69 / 132.45 ยฑ1.59 / 135.38 ms โ 131.03 / 131.88 ยฑ0.65 / 132.88 ms โ no change โ
โ QQuery 77 โ 186.35 / 189.20 ยฑ2.45 / 192.41 ms โ 188.41 / 190.03 ยฑ0.84 / 190.80 ms โ no change โ
โ QQuery 78 โ 337.21 / 340.33 ยฑ2.13 / 342.86 ms โ 335.72 / 337.13 ยฑ1.56 / 339.94 ms โ no change โ
โ QQuery 79 โ 233.14 / 234.52 ยฑ1.12 / 236.01 ms โ 230.24 / 234.02 ยฑ1.99 / 235.88 ms โ no change โ
โ QQuery 80 โ 319.02 / 320.94 ยฑ1.60 / 323.86 ms โ 319.97 / 324.30 ยฑ3.86 / 329.96 ms โ no change โ
โ QQuery 81 โ 25.97 / 27.33 ยฑ1.22 / 29.27 ms โ 26.11 / 26.62 ยฑ0.42 / 27.30 ms โ no change โ
โ QQuery 82 โ 197.31 / 199.34 ยฑ2.21 / 203.33 ms โ 197.15 / 199.69 ยฑ2.47 / 204.18 ms โ no change โ
โ QQuery 83 โ 37.65 / 39.42 ยฑ1.19 / 40.92 ms โ 38.74 / 39.71 ยฑ0.86 / 41.22 ms โ no change โ
โ QQuery 84 โ 47.28 / 47.82 ยฑ0.50 / 48.67 ms โ 47.82 / 48.99 ยฑ0.79 / 50.20 ms โ no change โ
โ QQuery 85 โ 144.83 / 148.70 ยฑ2.09 / 151.12 ms โ 148.28 / 150.12 ยฑ1.76 / 153.48 ms โ no change โ
โ QQuery 86 โ 38.19 / 39.35 ยฑ0.75 / 40.18 ms โ 39.43 / 40.19 ยฑ0.48 / 40.64 ms โ no change โ
โ QQuery 87 โ 85.08 / 88.06 ยฑ3.53 / 94.97 ms โ 80.11 / 86.50 ยฑ4.50 / 93.88 ms โ no change โ
โ QQuery 88 โ 99.71 / 100.70 ยฑ0.69 / 101.64 ms โ 99.49 / 100.48 ยฑ1.16 / 102.71 ms โ no change โ
โ QQuery 89 โ 118.40 / 119.78 ยฑ1.15 / 121.33 ms โ 118.44 / 119.35 ยฑ0.94 / 120.99 ms โ no change โ
โ QQuery 90 โ 22.54 / 23.74 ยฑ1.11 / 25.78 ms โ 23.26 / 24.18 ยฑ0.68 / 25.36 ms โ no change โ
โ QQuery 91 โ 61.81 / 64.31 ยฑ1.34 / 65.62 ms โ 64.07 / 64.48 ยฑ0.39 / 64.96 ms โ no change โ
โ QQuery 92 โ 57.05 / 57.91 ยฑ0.99 / 59.64 ms โ 56.16 / 57.74 ยฑ1.00 / 59.07 ms โ no change โ
โ QQuery 93 โ 185.01 / 186.26 ยฑ0.77 / 187.21 ms โ 183.50 / 186.60 ยฑ1.61 / 188.15 ms โ no change โ
โ QQuery 94 โ 61.08 / 61.52 ยฑ0.38 / 62.17 ms โ 61.35 / 62.19 ยฑ0.77 / 63.53 ms โ no change โ
โ QQuery 95 โ 126.61 / 127.67 ยฑ0.63 / 128.42 ms โ 126.61 / 128.23 ยฑ1.52 / 131.03 ms โ no change โ
โ QQuery 96 โ 70.84 / 73.27 ยฑ1.35 / 74.72 ms โ 73.14 / 74.07 ยฑ1.03 / 75.65 ms โ no change โ
โ QQuery 97 โ 123.94 / 126.62 ยฑ1.80 / 128.70 ms โ 125.18 / 127.21 ยฑ1.56 / 129.10 ms โ no change โ
โ QQuery 98 โ 153.15 / 155.09 ยฑ1.28 / 156.86 ms โ 151.91 / 153.64 ยฑ1.54 / 156.30 ms โ no change โ
โ QQuery 99 โ 10769.94 / 10822.64 ยฑ39.76 / 10893.40 ms โ 10767.18 / 10783.97 ยฑ13.36 / 10799.71 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 31346.92ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 31356.03ms โ
โ Average Time (HEAD) โ 316.64ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 316.73ms โ
โ Queries Faster โ 1 โ
โ Queries Slower โ 2 โ
โ Queries with No Change โ 96 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 157.1s |
| Peak memory | 5.7 GiB |
| Avg memory | 4.7 GiB |
| CPU user | 258.2s |
| CPU sys | 16.8s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 157.0s |
| Peak memory | 5.4 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 258.4s |
| CPU sys | 17.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230369262
Requesting to run benchmarks TPCH , TPCDS , clickbench etc to see how groups accumulator impl performs in count(distinct) use case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4242360948
Nice, looks like a small improvement to clickbench (query 9) as well!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4242010077
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4241897983)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.19 / 4.41 ยฑ6.33 / 17.06 ms โ 1.21 / 4.49 ยฑ6.44 / 17.38 ms โ no change โ
โ QQuery 1 โ 14.18 / 14.64 ยฑ0.35 / 15.17 ms โ 14.30 / 14.94 ยฑ0.56 / 15.98 ms โ no change โ
โ QQuery 2 โ 43.38 / 44.18 ยฑ0.57 / 44.82 ms โ 43.66 / 43.92 ยฑ0.16 / 44.16 ms โ no change โ
โ QQuery 3 โ 41.71 / 45.14 ยฑ1.86 / 47.28 ms โ 41.49 / 43.29 ยฑ1.89 / 46.71 ms โ no change โ
โ QQuery 4 โ 282.10 / 286.88 ยฑ4.55 / 294.58 ms โ 285.00 / 290.98 ยฑ4.14 / 296.97 ms โ no change โ
โ QQuery 5 โ 336.33 / 342.51 ยฑ5.17 / 349.48 ms โ 338.76 / 344.24 ยฑ2.87 / 346.59 ms โ no change โ
โ QQuery 6 โ 5.52 / 6.47 ยฑ0.81 / 7.72 ms โ 5.57 / 6.83 ยฑ0.91 / 7.83 ms โ 1.06x slower โ
โ QQuery 7 โ 16.40 / 16.95 ยฑ0.54 / 17.95 ms โ 16.65 / 17.11 ยฑ0.52 / 18.13 ms โ no change โ
โ QQuery 8 โ 405.67 / 417.44 ยฑ9.19 / 433.98 ms โ 407.15 / 416.76 ยฑ5.67 / 423.09 ms โ no change โ
โ QQuery 9 โ 631.21 / 635.23 ยฑ3.90 / 642.45 ms โ 591.09 / 598.82 ยฑ5.31 / 607.48 ms โ +1.06x faster โ
โ QQuery 10 โ 89.78 / 92.67 ยฑ3.43 / 99.42 ms โ 90.82 / 93.43 ยฑ2.97 / 99.23 ms โ no change โ
โ QQuery 11 โ 102.72 / 104.68 ยฑ0.99 / 105.42 ms โ 103.80 / 105.35 ยฑ1.74 / 108.73 ms โ no change โ
โ QQuery 12 โ 331.35 / 341.36 ยฑ9.14 / 354.12 ms โ 337.22 / 340.25 ยฑ2.89 / 345.72 ms โ no change โ
โ QQuery 13 โ 461.69 / 474.65 ยฑ13.39 / 499.62 ms โ 444.98 / 458.89 ยฑ7.73 / 465.78 ms โ no change โ
โ QQuery 14 โ 342.10 / 346.82 ยฑ5.51 / 357.55 ms โ 345.96 / 348.19 ยฑ2.67 / 353.42 ms โ no change โ
โ QQuery 15 โ 347.39 / 360.93 ยฑ15.45 / 389.04 ms โ 342.65 / 362.14 ยฑ14.69 / 382.36 ms โ no change โ
โ QQuery 16 โ 700.00 / 712.46 ยฑ14.53 / 737.86 ms โ 711.34 / 735.70 ยฑ21.76 / 770.23 ms โ no change โ
โ QQuery 17 โ 694.46 / 705.26 ยฑ6.33 / 713.97 ms โ 704.26 / 712.72 ยฑ5.02 / 719.38 ms โ no change โ
โ QQuery 18 โ 1427.38 / 1456.77 ยฑ26.36 / 1506.41 ms โ 1426.47 / 1486.29 ยฑ39.05 / 1526.91 ms โ no change โ
โ QQuery 19 โ 35.67 / 49.28 ยฑ22.35 / 93.86 ms โ 35.55 / 38.20 ยฑ3.16 / 43.62 ms โ +1.29x faster โ
โ QQuery 20 โ 721.75 / 745.82 ยฑ19.92 / 774.33 ms โ 712.33 / 724.33 ยฑ17.99 / 760.08 ms โ no change โ
โ QQuery 21 โ 766.68 / 773.14 ยฑ8.18 / 789.17 ms โ 763.22 / 778.36 ยฑ13.07 / 794.51 ms โ no change โ
โ QQuery 22 โ 1126.23 / 1132.31 ยฑ4.49 / 1139.31 ms โ 1125.79 / 1127.63 ยฑ2.05 / 1131.31 ms โ no change โ
โ QQuery 23 โ 3065.58 / 3090.14 ยฑ30.47 / 3150.03 ms โ 3076.42 / 3092.64 ยฑ13.20 / 3108.30 ms โ no change โ
โ QQuery 24 โ 96.30 / 101.65 ยฑ4.07 / 108.29 ms โ 101.43 / 103.49 ยฑ1.67 / 105.76 ms โ no change โ
โ QQuery 25 โ 141.09 / 142.00 ยฑ0.82 / 143.40 ms โ 138.95 / 141.30 ยฑ1.62 / 143.71 ms โ no change โ
โ QQuery 26 โ 98.68 / 101.53 ยฑ2.24 / 104.74 ms โ 96.14 / 100.29 ยฑ3.13 / 105.05 ms โ no change โ
โ QQuery 27 โ 851.03 / 860.95 ยฑ7.43 / 869.99 ms โ 847.33 / 857.70 ยฑ9.12 / 870.60 ms โ no change โ
โ QQuery 28 โ 3242.79 / 3281.95 ยฑ22.50 / 3310.99 ms โ 3274.86 / 3295.39 ยฑ16.54 / 3325.61 ms โ no change โ
โ QQuery 29 โ 50.37 / 56.44 ยฑ5.42 / 66.65 ms โ 49.82 / 54.32 ยฑ3.55 / 60.48 ms โ no change โ
โ QQuery 30 โ 349.55 / 362.57 ยฑ8.99 / 373.72 ms โ 360.19 / 365.80 ยฑ4.85 / 373.71 ms โ no change โ
โ QQuery 31 โ 376.69 / 384.37 ยฑ7.36 / 395.07 ms โ 363.82 / 379.80 ยฑ9.14 / 388.84 ms โ no change โ
โ QQuery 32 โ 1102.89 / 1258.70 ยฑ174.01 / 1579.54 ms โ 1187.29 / 1244.00 ยฑ56.99 / 1332.52 ms โ no change โ
โ QQuery 33 โ 1456.47 / 1468.95 ยฑ11.44 / 1487.87 ms โ 1469.18 / 1534.56 ยฑ33.47 / 1561.28 ms โ no change โ
โ QQuery 34 โ 1447.96 / 1457.11 ยฑ7.53 / 1466.40 ms โ 1445.93 / 1473.91 ยฑ15.45 / 1488.42 ms โ no change โ
โ QQuery 35 โ 382.99 / 386.45 ยฑ3.30 / 392.30 ms โ 381.06 / 390.09 ยฑ9.78 / 407.42 ms โ no change โ
โ QQuery 36 โ 116.24 / 121.96 ยฑ3.34 / 124.91 ms โ 112.00 / 118.08 ยฑ5.00 / 125.18 ms โ no change โ
โ QQuery 37 โ 46.83 / 48.16 ยฑ0.88 / 49.23 ms โ 47.75 / 49.17 ยฑ1.23 / 51.48 ms โ no change โ
โ QQuery 38 โ 75.63 / 77.66 ยฑ1.52 / 80.00 ms โ 74.40 / 76.06 ยฑ1.42 / 77.84 ms โ no change โ
โ QQuery 39 โ 209.61 / 215.77 ยฑ4.97 / 221.96 ms โ 199.94 / 214.46 ยฑ7.56 / 221.43 ms โ no change โ
โ QQuery 40 โ 25.78 / 28.41 ยฑ2.12 / 31.63 ms โ 21.55 / 24.51 ยฑ3.05 / 29.98 ms โ +1.16x faster โ
โ QQuery 41 โ 20.11 / 22.38 ยฑ1.91 / 25.59 ms โ 20.98 / 22.19 ยฑ1.28 / 24.13 ms โ no change โ
โ QQuery 42 โ 19.77 / 20.20 ยฑ0.48 / 20.90 ms โ 18.98 / 19.52 ยฑ0.31 / 19.79 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 22597.34ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 22650.15ms โ
โ Average Time (HEAD) โ 525.52ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 526.75ms โ
โ Queries Faster โ 3 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 39 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 114.2s |
| Peak memory | 40.1 GiB |
| Avg memory | 30.4 GiB |
| CPU user | 1067.0s |
| CPU sys | 92.5s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 114.3s |
| Peak memory | 38.5 GiB |
| Avg memory | 26.7 GiB |
| CPU user | 1063.2s |
| CPU sys | 98.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4242030135
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4241897983)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 844.37 / 859.87 ยฑ8.55 / 868.08 ms โ 841.21 / 846.37 ยฑ5.07 / 855.93 ms โ no change โ
โ QQuery 1 โ 206.68 / 207.81 ยฑ1.00 / 209.65 ms โ 213.11 / 213.50 ยฑ0.21 / 213.71 ms โ no change โ
โ QQuery 2 โ 498.06 / 500.07 ยฑ1.66 / 502.66 ms โ 505.25 / 508.05 ยฑ2.26 / 510.92 ms โ no change โ
โ QQuery 3 โ 314.33 / 316.25 ยฑ1.11 / 317.76 ms โ 315.62 / 316.92 ยฑ0.85 / 317.85 ms โ no change โ
โ QQuery 4 โ 660.29 / 682.13 ยฑ23.21 / 724.62 ms โ 659.44 / 676.43 ยฑ12.80 / 698.31 ms โ no change โ
โ QQuery 5 โ 9755.51 / 9938.93 ยฑ126.19 / 10103.47 ms โ 9744.53 / 9919.22 ยฑ141.06 / 10123.02 ms โ no change โ
โ QQuery 6 โ 998.79 / 1014.25 ยฑ13.64 / 1039.62 ms โ 998.59 / 1006.75 ยฑ8.59 / 1023.05 ms โ no change โ
โ QQuery 7 โ 782.66 / 868.60 ยฑ135.43 / 1136.77 ms โ 766.38 / 778.31 ยฑ11.21 / 798.56 ms โ +1.12x faster โ
โ QQuery 8 โ 405.90 / 409.84 ยฑ3.14 / 415.16 ms โ 402.39 / 404.79 ยฑ2.61 / 409.53 ms โ no change โ
โ QQuery 9 โ 2957.26 / 2985.68 ยฑ23.96 / 3018.70 ms โ 2760.30 / 2816.69 ยฑ41.37 / 2868.33 ms โ +1.06x faster โ
โ QQuery 10 โ 629.71 / 643.15 ยฑ9.46 / 655.94 ms โ 629.12 / 641.83 ยฑ11.82 / 663.96 ms โ no change โ
โ QQuery 11 โ 2070.90 / 2082.00 ยฑ6.82 / 2089.67 ms โ 2210.72 / 2219.68 ยฑ9.43 / 2234.75 ms โ 1.07x slower โ
โ QQuery 12 โ 193.89 / 200.13 ยฑ5.40 / 209.82 ms โ 193.82 / 199.18 ยฑ4.28 / 204.46 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 20708.71ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 20547.73ms โ
โ Average Time (HEAD) โ 1592.98ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 1580.59ms โ
โ Queries Faster โ 2 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 10 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 104.4s |
| Peak memory | 32.0 GiB |
| Avg memory | 27.5 GiB |
| CPU user | 996.0s |
| CPU sys | 50.0s |
| Peak spill | 0 B |
**clickbench_extended โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 103.6s |
| Peak memory | 33.6 GiB |
| Avg memory | 29.8 GiB |
| CPU user | 995.0s |
| CPU sys | 45.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4261984688
run benchmark tpch tpcds clickbench_partitioned clickbench_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3098451712
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ ArrayRef, AsArray, BooleanArray, Int64Array, ListArray, PrimitiveArray,
+};
+use arrow::buffer::OffsetBuffer;
+use arrow::datatypes::{ArrowPrimitiveType, Field};
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+use crate::aggregate::groups_accumulator::accumulate::accumulate;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ seen: HashSet<(usize, T::Native), RandomState>,
+ counts: Vec<i64>,
+}
+
+impl<T: ArrowPrimitiveType> PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ pub fn new() -> Self {
+ Self {
+ seen: HashSet::default(),
+ counts: Vec::new(),
+ }
+ }
+}
+
+impl<T: ArrowPrimitiveType> Default for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ArrowPrimitiveType + Send + std::fmt::Debug> GroupsAccumulator
+ for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn update_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> datafusion_common::Result<()> {
+ debug_assert_eq!(values.len(), 1);
+ self.counts.resize(total_num_groups, 0);
+ let arr = values[0].as_primitive::<T>();
+ accumulate(group_indices, arr, opt_filter, |group_idx, value| {
+ if self.seen.insert((group_idx, value)) {
+ self.counts[group_idx] += 1;
+ }
+ });
+ Ok(())
+ }
+
+ fn evaluate(&mut self, emit_to: EmitTo) -> datafusion_common::Result<ArrayRef> {
+ let counts = emit_to.take_needed(&mut self.counts);
+
+ match emit_to {
+ EmitTo::All => {
+ self.seen.clear();
+ }
+ EmitTo::First(n) => {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx >= n {
+ remaining.insert((group_idx - n, value));
+ }
+ }
+ self.seen = remaining;
+ }
+ }
+
+ Ok(Arc::new(Int64Array::from(counts)))
+ }
+
+ fn state(&mut self, emit_to: EmitTo) -> datafusion_common::Result<Vec<ArrayRef>> {
+ let num_emitted = match emit_to {
+ EmitTo::All => self.counts.len(),
+ EmitTo::First(n) => n,
+ };
+
+ let mut group_values: Vec<Vec<T::Native>> = vec![Vec::new(); num_emitted];
Review Comment:
Ah yes . I tried having one vector but that my implementation more complex than I thought I could maintain . But let me give this a try and get back to you to see if that helps with benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4262001099
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4261984688)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4261984688-1387-wblcp 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (ba140a00af402b7f8e6ff3470d1f50b369766d69) to cc4717a (merge-base) [diff](https://github.com/apache/datafusion/compare/cc4717a14452e65a4d99c4b180d62c2793b713bd..ba140a00af402b7f8e6ff3470d1f50b369766d69) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4262001717
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4261984688)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4261984688-1386-jwgn7 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (ba140a00af402b7f8e6ff3470d1f50b369766d69) to cc4717a (merge-base) [diff](https://github.com/apache/datafusion/compare/cc4717a14452e65a4d99c4b180d62c2793b713bd..ba140a00af402b7f8e6ff3470d1f50b369766d69) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3098475972
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ ArrayRef, AsArray, BooleanArray, Int64Array, ListArray, PrimitiveArray,
+};
+use arrow::buffer::OffsetBuffer;
+use arrow::datatypes::{ArrowPrimitiveType, Field};
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+use crate::aggregate::groups_accumulator::accumulate::accumulate;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ seen: HashSet<(usize, T::Native), RandomState>,
+ counts: Vec<i64>,
+}
+
+impl<T: ArrowPrimitiveType> PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ pub fn new() -> Self {
+ Self {
+ seen: HashSet::default(),
+ counts: Vec::new(),
+ }
+ }
+}
+
+impl<T: ArrowPrimitiveType> Default for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ArrowPrimitiveType + Send + std::fmt::Debug> GroupsAccumulator
+ for PrimitiveDistinctCountGroupsAccumulator<T>
+where
+ T::Native: Eq + Hash,
+{
+ fn update_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> datafusion_common::Result<()> {
+ debug_assert_eq!(values.len(), 1);
+ self.counts.resize(total_num_groups, 0);
+ let arr = values[0].as_primitive::<T>();
+ accumulate(group_indices, arr, opt_filter, |group_idx, value| {
+ if self.seen.insert((group_idx, value)) {
+ self.counts[group_idx] += 1;
+ }
+ });
+ Ok(())
+ }
+
+ fn evaluate(&mut self, emit_to: EmitTo) -> datafusion_common::Result<ArrayRef> {
+ let counts = emit_to.take_needed(&mut self.counts);
+
+ match emit_to {
+ EmitTo::All => {
+ self.seen.clear();
+ }
+ EmitTo::First(n) => {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx >= n {
+ remaining.insert((group_idx - n, value));
+ }
+ }
+ self.seen = remaining;
+ }
+ }
+
+ Ok(Arc::new(Int64Array::from(counts)))
+ }
+
+ fn state(&mut self, emit_to: EmitTo) -> datafusion_common::Result<Vec<ArrayRef>> {
+ let num_emitted = match emit_to {
+ EmitTo::All => self.counts.len(),
+ EmitTo::First(n) => n,
+ };
+
+ let mut group_values: Vec<Vec<T::Native>> = vec![Vec::new(); num_emitted];
+
+ if matches!(emit_to, EmitTo::All) {
+ for (group_idx, value) in self.seen.drain() {
+ group_values[group_idx].push(value);
+ }
+ self.counts.clear();
+ } else {
+ let mut remaining = HashSet::default();
+ for (group_idx, value) in self.seen.drain() {
+ if group_idx < num_emitted {
+ group_values[group_idx].push(value);
+ } else {
+ remaining.insert((group_idx - num_emitted, value));
+ }
+ }
+ self.seen = remaining;
+ let _ = emit_to.take_needed(&mut self.counts);
+ }
+
+ let mut offsets = vec![0i32];
+ let mut all_values = Vec::new();
+ for values in &group_values {
+ all_values.extend(values.iter().copied());
+ offsets.push(all_values.len() as i32);
+ }
+
+ let values_array = Arc::new(PrimitiveArray::<T>::from_iter_values(all_values));
+ let list_array = ListArray::new(
+ Arc::new(Field::new_list_field(T::DATA_TYPE, true)),
+ OffsetBuffer::new(offsets.into()),
+ values_array,
+ None,
+ );
+
+ Ok(vec![Arc::new(list_array)])
+ }
+
+ fn merge_batch(
+ &mut self,
+ values: &[ArrayRef],
+ group_indices: &[usize],
+ _opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ ) -> datafusion_common::Result<()> {
+ debug_assert_eq!(values.len(), 1);
+ self.counts.resize(total_num_groups, 0);
+ let list_array = values[0].as_list::<i32>();
+
+ for (row_idx, &group_idx) in group_indices.iter().enumerate() {
+ let inner = list_array.value(row_idx);
Review Comment:
Something like this should be slightly faster:
```
let inner_values = inner.values();
let offsets = list_array.offsets();
for (row_idx, &group_idx) in group_indices.iter().enumerate() {
let start = offsets[row_idx] as usize;
let end = offsets[row_idx + 1] as usize;
for &value in &inner_values[start..end] {
if self.seen.insert((group_idx, value)) {
self.counts[group_idx] += 1;
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4241905992
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4241897983)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4241897983-1204-cdlkk 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (8d60834b3b09bad5093b74d0a32799e3dc98dd0a) to a13c23d (merge-base) [diff](https://github.com/apache/datafusion/compare/a13c23df67c218e45342b80238cae490609d8f3d..8d60834b3b09bad5093b74d0a32799e3dc98dd0a) using: clickbench_extended
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4241914601
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4241897983)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4241897983-1201-njt5c 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (8d60834b3b09bad5093b74d0a32799e3dc98dd0a) to a13c23d (merge-base) [diff](https://github.com/apache/datafusion/compare/a13c23df67c218e45342b80238cae490609d8f3d..8d60834b3b09bad5093b74d0a32799e3dc98dd0a) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3068983946
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, BooleanArray, Int64Array, ListBuilder, PrimitiveBuilder,
+};
+use arrow::datatypes::ArrowPrimitiveType;
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ /// Count distinct per group.
+ values: Vec<HashSet<T::Native, RandomState>>,
Review Comment:
Usize is the group index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4292788946
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4292778636)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4292778636-1726-x5krl 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (54ba4a1026fb743fb5fd9198889aec3bf92ea80c) to 90a8117 (merge-base) [diff](https://github.com/apache/datafusion/compare/90a8117be6c54b2b188149e14ab339f2a9608dff..54ba4a1026fb743fb5fd9198889aec3bf92ea80c) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4292855222
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4292778636)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 6.60 / 7.00 ยฑ0.74 / 8.48 ms โ 7.15 / 7.50 ยฑ0.63 / 8.76 ms โ 1.07x slower โ
โ QQuery 2 โ 146.84 / 148.71 ยฑ1.16 / 150.32 ms โ 147.17 / 147.70 ยฑ0.63 / 148.89 ms โ no change โ
โ QQuery 3 โ 113.73 / 114.50 ยฑ0.79 / 115.99 ms โ 115.45 / 116.16 ยฑ0.62 / 117.01 ms โ no change โ
โ QQuery 4 โ 1333.08 / 1361.72 ยฑ20.18 / 1393.07 ms โ 1358.45 / 1388.99 ยฑ18.95 / 1410.53 ms โ no change โ
โ QQuery 5 โ 172.78 / 176.23 ยฑ2.46 / 180.19 ms โ 171.80 / 175.92 ยฑ3.77 / 182.61 ms โ no change โ
โ QQuery 6 โ 833.99 / 865.41 ยฑ28.56 / 904.82 ms โ 850.65 / 861.96 ยฑ5.96 / 867.56 ms โ no change โ
โ QQuery 7 โ 343.24 / 349.26 ยฑ4.56 / 356.79 ms โ 343.81 / 347.43 ยฑ2.54 / 351.12 ms โ no change โ
โ QQuery 8 โ 116.41 / 118.32 ยฑ1.45 / 120.33 ms โ 119.41 / 119.75 ยฑ0.36 / 120.41 ms โ no change โ
โ QQuery 9 โ 101.49 / 106.37 ยฑ2.55 / 108.93 ms โ 103.45 / 111.44 ยฑ8.82 / 128.45 ms โ no change โ
โ QQuery 10 โ 106.98 / 109.55 ยฑ1.72 / 112.13 ms โ 108.39 / 109.45 ยฑ0.55 / 109.86 ms โ no change โ
โ QQuery 11 โ 963.99 / 983.20 ยฑ15.39 / 1008.65 ms โ 991.13 / 1003.08 ยฑ10.60 / 1022.68 ms โ no change โ
โ QQuery 12 โ 44.69 / 45.98 ยฑ1.19 / 47.95 ms โ 44.41 / 46.70 ยฑ1.56 / 48.93 ms โ no change โ
โ QQuery 13 โ 401.58 / 409.19 ยฑ5.14 / 415.70 ms โ 402.09 / 408.80 ยฑ5.09 / 415.17 ms โ no change โ
โ QQuery 14 โ 1009.95 / 1023.49 ยฑ9.14 / 1033.98 ms โ 1012.66 / 1020.21 ยฑ7.36 / 1034.13 ms โ no change โ
โ QQuery 15 โ 17.45 / 18.40 ยฑ0.93 / 19.98 ms โ 16.20 / 17.23 ยฑ0.68 / 18.21 ms โ +1.07x faster โ
โ QQuery 16 โ 7.91 / 8.04 ยฑ0.12 / 8.25 ms โ 7.17 / 7.51 ยฑ0.38 / 8.04 ms โ +1.07x faster โ
โ QQuery 17 โ 230.21 / 235.63 ยฑ4.34 / 240.62 ms โ 228.77 / 235.85 ยฑ6.85 / 246.88 ms โ no change โ
โ QQuery 18 โ 128.00 / 130.37 ยฑ2.22 / 134.17 ms โ 132.98 / 134.41 ยฑ1.73 / 137.66 ms โ no change โ
โ QQuery 19 โ 158.70 / 162.56 ยฑ2.14 / 165.08 ms โ 154.46 / 159.55 ยฑ3.81 / 164.34 ms โ no change โ
โ QQuery 20 โ 14.82 / 15.60 ยฑ0.42 / 16.01 ms โ 14.90 / 15.71 ยฑ0.65 / 16.70 ms โ no change โ
โ QQuery 21 โ 20.94 / 21.25 ยฑ0.31 / 21.63 ms โ 20.10 / 20.95 ยฑ0.80 / 22.10 ms โ no change โ
โ QQuery 22 โ 501.84 / 529.57 ยฑ17.93 / 557.22 ms โ 485.24 / 502.50 ยฑ14.28 / 518.14 ms โ +1.05x faster โ
โ QQuery 23 โ 919.25 / 938.09 ยฑ11.93 / 956.42 ms โ 901.52 / 913.07 ยฑ7.46 / 922.17 ms โ no change โ
โ QQuery 24 โ 387.46 / 393.83 ยฑ4.93 / 399.48 ms โ 383.92 / 391.20 ยฑ6.38 / 400.89 ms โ no change โ
โ QQuery 25 โ 340.50 / 348.47 ยฑ5.61 / 357.45 ms โ 344.10 / 346.96 ยฑ3.52 / 353.65 ms โ no change โ
โ QQuery 26 โ 82.22 / 84.54 ยฑ1.49 / 86.53 ms โ 81.85 / 85.03 ยฑ1.66 / 86.43 ms โ no change โ
โ QQuery 27 โ 7.52 / 8.36 ยฑ0.91 / 9.58 ms โ 7.11 / 7.69 ยฑ0.63 / 8.88 ms โ +1.09x faster โ
โ QQuery 28 โ 148.09 / 150.01 ยฑ2.35 / 154.56 ms โ 150.14 / 152.30 ยฑ2.62 / 156.69 ms โ no change โ
โ QQuery 29 โ 283.25 / 289.28 ยฑ5.71 / 298.36 ms โ 284.96 / 290.51 ยฑ3.17 / 293.81 ms โ no change โ
โ QQuery 30 โ 43.45 / 45.60 ยฑ1.50 / 47.24 ms โ 43.28 / 46.48 ยฑ1.66 / 47.79 ms โ no change โ
โ QQuery 31 โ 170.94 / 174.26 ยฑ3.58 / 179.27 ms โ 172.79 / 175.20 ยฑ2.09 / 178.77 ms โ no change โ
โ QQuery 32 โ 13.72 / 14.45 ยฑ0.72 / 15.74 ms โ 13.97 / 14.44 ยฑ0.34 / 14.96 ms โ no change โ
โ QQuery 33 โ 141.28 / 144.29 ยฑ2.16 / 147.44 ms โ 141.40 / 144.11 ยฑ2.00 / 147.59 ms โ no change โ
โ QQuery 34 โ 7.39 / 8.03 ยฑ0.67 / 9.27 ms โ 7.07 / 7.44 ยฑ0.31 / 7.83 ms โ +1.08x faster โ
โ QQuery 35 โ 108.95 / 109.71 ยฑ0.88 / 111.41 ms โ 107.57 / 109.71 ยฑ1.45 / 111.74 ms โ no change โ
โ QQuery 36 โ 6.68 / 6.88 ยฑ0.18 / 7.21 ms โ 6.46 / 6.74 ยฑ0.29 / 7.31 ms โ no change โ
โ QQuery 37 โ 8.33 / 8.56 ยฑ0.20 / 8.89 ms โ 8.42 / 9.50 ยฑ1.10 / 11.09 ms โ 1.11x slower โ
โ QQuery 38 โ 83.24 / 92.34 ยฑ6.13 / 100.22 ms โ 85.05 / 87.72 ยฑ2.39 / 91.84 ms โ +1.05x faster โ
โ QQuery 39 โ 125.60 / 126.79 ยฑ1.26 / 129.11 ms โ 129.13 / 136.71 ยฑ5.32 / 143.81 ms โ 1.08x slower โ
โ QQuery 40 โ 109.30 / 118.02 ยฑ10.46 / 138.41 ms โ 115.54 / 122.65 ยฑ7.09 / 136.19 ms โ no change โ
โ QQuery 41 โ 13.95 / 14.55 ยฑ0.39 / 14.97 ms โ 15.04 / 15.62 ยฑ0.31 / 15.90 ms โ 1.07x slower โ
โ QQuery 42 โ 107.85 / 110.06 ยฑ1.32 / 111.70 ms โ 109.07 / 111.23 ยฑ1.12 / 112.14 ms โ no change โ
โ QQuery 43 โ 6.16 / 6.34 ยฑ0.21 / 6.69 ms โ 5.79 / 5.92 ยฑ0.21 / 6.35 ms โ +1.07x faster โ
โ QQuery 44 โ 11.74 / 12.21 ยฑ0.63 / 13.44 ms โ 11.92 / 12.85 ยฑ1.09 / 14.91 ms โ 1.05x slower โ
โ QQuery 45 โ 50.79 / 51.64 ยฑ0.63 / 52.26 ms โ 51.57 / 52.38 ยฑ1.09 / 54.48 ms โ no change โ
โ QQuery 46 โ 8.20 / 8.63 ยฑ0.39 / 9.18 ms โ 8.84 / 9.06 ยฑ0.25 / 9.53 ms โ no change โ
โ QQuery 47 โ 727.71 / 737.39 ยฑ8.49 / 750.15 ms โ 828.67 / 848.81 ยฑ16.90 / 879.70 ms โ 1.15x slower โ
โ QQuery 48 โ 287.83 / 297.53 ยฑ6.77 / 303.90 ms โ 293.55 / 304.27 ยฑ5.86 / 309.90 ms โ no change โ
โ QQuery 49 โ 250.33 / 256.81 ยฑ3.57 / 260.07 ms โ 251.80 / 256.47 ยฑ3.76 / 262.79 ms โ no change โ
โ QQuery 50 โ 223.14 / 235.99 ยฑ7.85 / 246.33 ms โ 234.30 / 238.37 ยฑ4.73 / 247.04 ms โ no change โ
โ QQuery 51 โ 183.09 / 187.14 ยฑ3.35 / 192.41 ms โ 177.26 / 183.52 ยฑ3.53 / 186.94 ms โ no change โ
โ QQuery 52 โ 110.09 / 111.02 ยฑ0.98 / 112.67 ms โ 109.43 / 110.87 ยฑ1.21 / 112.46 ms โ no change โ
โ QQuery 53 โ 102.10 / 104.58 ยฑ1.78 / 106.85 ms โ 103.01 / 105.36 ยฑ1.63 / 107.82 ms โ no change โ
โ QQuery 54 โ 146.25 / 151.29 ยฑ3.29 / 154.37 ms โ 151.91 / 154.60 ยฑ2.13 / 157.98 ms โ no change โ
โ QQuery 55 โ 107.71 / 110.26 ยฑ1.71 / 113.03 ms โ 107.76 / 109.56 ยฑ1.07 / 110.69 ms โ no change โ
โ QQuery 56 โ 141.39 / 143.44 ยฑ2.69 / 148.60 ms โ 141.30 / 146.31 ยฑ3.65 / 151.40 ms โ no change โ
โ QQuery 57 โ 173.52 / 176.92 ยฑ3.13 / 181.46 ms โ 175.95 / 178.12 ยฑ2.63 / 183.12 ms โ no change โ
โ QQuery 58 โ 274.17 / 276.43 ยฑ2.37 / 280.92 ms โ 273.77 / 276.29 ยฑ2.87 / 281.88 ms โ no change โ
โ QQuery 59 โ 196.11 / 200.15 ยฑ4.66 / 209.14 ms โ 196.21 / 200.55 ยฑ3.77 / 206.91 ms โ no change โ
โ QQuery 60 โ 143.47 / 145.01 ยฑ1.32 / 147.41 ms โ 142.90 / 148.02 ยฑ2.63 / 150.33 ms โ no change โ
โ QQuery 61 โ 13.95 / 14.11 ยฑ0.16 / 14.39 ms โ 12.98 / 13.36 ยฑ0.36 / 13.90 ms โ +1.06x faster โ
โ QQuery 62 โ 918.46 / 966.78 ยฑ55.79 / 1068.18 ms โ 902.17 / 936.57 ยฑ31.13 / 980.30 ms โ no change โ
โ QQuery 63 โ 104.96 / 107.57 ยฑ2.12 / 109.43 ms โ 103.85 / 106.22 ยฑ2.32 / 110.28 ms โ no change โ
โ QQuery 64 โ 682.60 / 699.13 ยฑ10.19 / 712.62 ms โ 696.11 / 706.02 ยฑ11.70 / 728.62 ms โ no change โ
โ QQuery 65 โ 249.24 / 259.93 ยฑ12.89 / 284.53 ms โ 250.69 / 266.39 ยฑ10.53 / 280.43 ms โ no change โ
โ QQuery 66 โ 247.44 / 258.17 ยฑ8.69 / 273.22 ms โ 256.38 / 270.01 ยฑ10.54 / 284.02 ms โ no change โ
โ QQuery 67 โ 298.17 / 318.84 ยฑ14.40 / 336.36 ms โ 334.35 / 338.65 ยฑ4.76 / 345.45 ms โ 1.06x slower โ
โ QQuery 68 โ 8.83 / 11.01 ยฑ2.11 / 14.88 ms โ 8.49 / 10.29 ยฑ1.36 / 12.15 ms โ +1.07x faster โ
โ QQuery 69 โ 100.79 / 107.15 ยฑ5.04 / 113.20 ms โ 101.76 / 103.30 ยฑ1.65 / 106.28 ms โ no change โ
โ QQuery 70 โ 333.20 / 348.45 ยฑ14.17 / 366.61 ms โ 340.92 / 360.69 ยฑ15.68 / 383.69 ms โ no change โ
โ QQuery 71 โ 133.96 / 142.01 ยฑ4.51 / 146.87 ms โ 134.42 / 139.71 ยฑ3.52 / 145.13 ms โ no change โ
โ QQuery 72 โ 617.09 / 646.81 ยฑ17.17 / 669.43 ms โ 619.43 / 635.16 ยฑ10.92 / 649.41 ms โ no change โ
โ QQuery 73 โ 7.38 / 8.50 ยฑ0.88 / 9.84 ms โ 6.93 / 8.53 ยฑ1.01 / 9.43 ms โ no change โ
โ QQuery 74 โ 583.82 / 621.82 ยฑ40.43 / 675.18 ms โ 607.39 / 630.83 ยฑ26.41 / 678.37 ms โ no change โ
โ QQuery 75 โ 274.62 / 281.17 ยฑ4.92 / 289.03 ms โ 277.91 / 281.52 ยฑ3.40 / 287.11 ms โ no change โ
โ QQuery 76 โ 131.97 / 133.70 ยฑ1.43 / 136.26 ms โ 132.60 / 133.93 ยฑ0.97 / 135.21 ms โ no change โ
โ QQuery 77 โ 192.01 / 195.32 ยฑ2.60 / 198.64 ms โ 190.19 / 193.75 ยฑ1.96 / 195.44 ms โ no change โ
โ QQuery 78 โ 360.57 / 363.77 ยฑ2.50 / 367.35 ms โ 343.33 / 350.45 ยฑ7.57 / 364.26 ms โ no change โ
โ QQuery 79 โ 252.57 / 257.92 ยฑ3.55 / 262.00 ms โ 232.82 / 240.85 ยฑ9.20 / 258.78 ms โ +1.07x faster โ
โ QQuery 80 โ 329.80 / 332.67 ยฑ2.46 / 337.14 ms โ 327.30 / 330.53 ยฑ2.06 / 333.20 ms โ no change โ
โ QQuery 81 โ 27.78 / 28.31 ยฑ0.38 / 28.77 ms โ 26.00 / 26.83 ยฑ0.54 / 27.40 ms โ +1.05x faster โ
โ QQuery 82 โ 41.38 / 42.48 ยฑ0.72 / 43.56 ms โ 41.11 / 42.55 ยฑ2.31 / 47.15 ms โ no change โ
โ QQuery 83 โ 39.27 / 40.63 ยฑ1.43 / 43.21 ms โ 40.40 / 41.64 ยฑ1.26 / 43.90 ms โ no change โ
โ QQuery 84 โ 49.94 / 51.25 ยฑ1.36 / 53.88 ms โ 48.85 / 50.59 ยฑ1.04 / 51.95 ms โ no change โ
โ QQuery 85 โ 150.07 / 151.29 ยฑ1.28 / 153.21 ms โ 149.13 / 152.01 ยฑ2.63 / 155.27 ms โ no change โ
โ QQuery 86 โ 41.80 / 42.18 ยฑ0.30 / 42.55 ms โ 38.92 / 40.98 ยฑ1.41 / 43.28 ms โ no change โ
โ QQuery 87 โ 83.25 / 87.81 ยฑ6.29 / 100.20 ms โ 85.04 / 88.21 ยฑ3.32 / 94.32 ms โ no change โ
โ QQuery 88 โ 101.03 / 102.62 ยฑ1.66 / 104.69 ms โ 100.91 / 102.82 ยฑ1.57 / 104.74 ms โ no change โ
โ QQuery 89 โ 119.76 / 121.14 ยฑ1.13 / 123.21 ms โ 118.79 / 120.80 ยฑ1.20 / 121.96 ms โ no change โ
โ QQuery 90 โ 23.40 / 24.03 ยฑ0.67 / 25.25 ms โ 24.57 / 25.21 ยฑ0.66 / 26.48 ms โ no change โ
โ QQuery 91 โ 63.03 / 67.06 ยฑ2.25 / 69.75 ms โ 64.48 / 66.62 ยฑ1.22 / 67.81 ms โ no change โ
โ QQuery 92 โ 58.52 / 59.27 ยฑ0.67 / 60.50 ms โ 57.45 / 58.59 ยฑ0.79 / 59.73 ms โ no change โ
โ QQuery 93 โ 190.37 / 193.09 ยฑ2.90 / 198.40 ms โ 189.08 / 193.27 ยฑ4.30 / 199.77 ms โ no change โ
โ QQuery 94 โ 64.22 / 65.13 ยฑ0.96 / 66.71 ms โ 62.06 / 62.93 ยฑ0.99 / 64.82 ms โ no change โ
โ QQuery 95 โ 130.73 / 134.15 ยฑ3.44 / 138.80 ms โ 130.38 / 133.24 ยฑ2.08 / 135.63 ms โ no change โ
โ QQuery 96 โ 73.26 / 74.12 ยฑ0.52 / 74.63 ms โ 73.57 / 74.97 ยฑ1.09 / 76.65 ms โ no change โ
โ QQuery 97 โ 128.66 / 132.88 ยฑ4.59 / 139.04 ms โ 128.94 / 133.05 ยฑ3.02 / 135.87 ms โ no change โ
โ QQuery 98 โ 155.26 / 160.94 ยฑ3.95 / 167.53 ms โ 161.74 / 163.41 ยฑ1.75 / 166.72 ms โ no change โ
โ QQuery 99 โ 10808.75 / 10906.40 ยฑ66.79 / 11011.89 ms โ 10853.88 / 10921.15 ยฑ44.11 / 10986.73 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 31898.86ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 32030.10ms โ
โ Average Time (HEAD) โ 322.21ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 323.54ms โ
โ Queries Faster โ 11 โ
โ Queries Slower โ 7 โ
โ Queries with No Change โ 81 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 6.4 GiB |
| Avg memory | 5.4 GiB |
| CPU user | 265.6s |
| CPU sys | 17.6s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 165.0s |
| Peak memory | 6.2 GiB |
| Avg memory | 5.3 GiB |
| CPU user | 265.2s |
| CPU sys | 17.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4251549675
> @Dandandan , I am trying to a Vector of pairs (group ID , value) approach to see if SIMD (sort and return group counts only during `evaluate` and `state` would cost lesser than computing hashes in `update_batch` which is more frequent) . Results were promising on my local but I would be happy if you could run a benchmarks on GH runners for more accuracy. I also moved distinct accumulators to a separate cold path (which were probably causing other count queries' minor slowness)
>
> Could you please trigger CI benchmarks to see if this is adding value over hashtable approach ?
It seems to me the problem would be memory use (whenever it is non-unique)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4251696700
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4251531266)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 775.96 / 784.61 ยฑ9.85 / 799.94 ms โ 774.77 / 788.30 ยฑ13.30 / 808.25 ms โ no change โ
โ QQuery 1 โ 206.74 / 208.03 ยฑ0.66 / 208.60 ms โ 212.79 / 213.38 ยฑ0.51 / 214.01 ms โ no change โ
โ QQuery 2 โ 492.20 / 494.93 ยฑ1.53 / 496.87 ms โ 502.43 / 504.52 ยฑ1.76 / 506.69 ms โ no change โ
โ QQuery 3 โ 311.71 / 313.13 ยฑ0.89 / 314.41 ms โ 313.34 / 314.58 ยฑ0.98 / 315.54 ms โ no change โ
โ QQuery 4 โ 657.18 / 674.49 ยฑ11.15 / 688.53 ms โ 650.71 / 663.45 ยฑ10.97 / 682.87 ms โ no change โ
โ QQuery 5 โ 9816.25 / 9968.66 ยฑ124.31 / 10177.31 ms โ 9670.84 / 9840.15 ยฑ153.64 / 10127.34 ms โ no change โ
โ QQuery 6 โ 991.04 / 1000.88 ยฑ11.22 / 1022.56 ms โ 991.63 / 998.88 ยฑ5.56 / 1007.19 ms โ no change โ
โ QQuery 7 โ 758.05 / 767.59 ยฑ7.42 / 778.59 ms โ 761.76 / 768.09 ยฑ3.60 / 772.60 ms โ no change โ
โ QQuery 8 โ 387.31 / 393.66 ยฑ4.04 / 399.16 ms โ 390.93 / 394.80 ยฑ4.25 / 402.65 ms โ no change โ
โ QQuery 9 โ 2720.53 / 2778.04 ยฑ39.85 / 2823.47 ms โ 2744.53 / 2779.59 ยฑ25.87 / 2825.03 ms โ no change โ
โ QQuery 10 โ 627.94 / 638.44 ยฑ13.25 / 664.18 ms โ 634.48 / 641.38 ยฑ7.02 / 653.18 ms โ no change โ
โ QQuery 11 โ 2048.90 / 2068.56 ยฑ13.34 / 2085.92 ms โ 2020.73 / 2047.03 ยฑ17.07 / 2072.46 ms โ no change โ
โ QQuery 12 โ 192.89 / 197.65 ยฑ4.30 / 203.02 ms โ 193.19 / 200.42 ยฑ4.58 / 205.98 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 20288.67ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 20154.56ms โ
โ Average Time (HEAD) โ 1560.67ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 1550.35ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 13 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 102.2s |
| Peak memory | 33.9 GiB |
| Avg memory | 29.2 GiB |
| CPU user | 991.3s |
| CPU sys | 44.5s |
| Peak spill | 0 B |
**clickbench_extended โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 101.6s |
| Peak memory | 34.0 GiB |
| Avg memory | 29.7 GiB |
| CPU user | 986.4s |
| CPU sys | 45.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4256919523
run benchmark tpch tpcds clickbench_partitioned clickbench_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4262138119
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4261984688)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 768.50 / 785.49 ยฑ10.45 / 795.40 ms โ 779.06 / 786.53 ยฑ8.83 / 798.83 ms โ no change โ
โ QQuery 1 โ 207.41 / 208.76 ยฑ0.89 / 209.85 ms โ 207.86 / 209.12 ยฑ0.97 / 210.53 ms โ no change โ
โ QQuery 2 โ 494.12 / 497.21 ยฑ1.78 / 499.54 ms โ 491.83 / 498.03 ยฑ3.25 / 501.15 ms โ no change โ
โ QQuery 3 โ 314.77 / 316.23 ยฑ0.82 / 317.29 ms โ 313.88 / 314.87 ยฑ0.52 / 315.35 ms โ no change โ
โ QQuery 4 โ 647.77 / 666.53 ยฑ15.37 / 689.27 ms โ 659.05 / 667.19 ยฑ7.16 / 678.36 ms โ no change โ
โ QQuery 5 โ 9401.86 / 9929.45 ยฑ275.83 / 10177.19 ms โ 9614.50 / 9978.14 ยฑ329.12 / 10550.00 ms โ no change โ
โ QQuery 6 โ 990.59 / 1003.55 ยฑ15.08 / 1032.81 ms โ 995.77 / 1014.60 ยฑ27.16 / 1068.34 ms โ no change โ
โ QQuery 7 โ 752.06 / 789.56 ยฑ32.73 / 828.93 ms โ 778.91 / 827.81 ยฑ52.33 / 927.10 ms โ no change โ
โ QQuery 8 โ 386.99 / 392.94 ยฑ3.85 / 398.57 ms โ 396.90 / 403.85 ยฑ8.44 / 420.47 ms โ no change โ
โ QQuery 9 โ 2749.93 / 2765.48 ยฑ11.75 / 2784.61 ms โ 2985.60 / 3008.35 ยฑ25.80 / 3056.79 ms โ 1.09x slower โ
โ QQuery 10 โ 629.29 / 642.68 ยฑ11.74 / 663.69 ms โ 629.68 / 639.38 ยฑ6.93 / 647.74 ms โ no change โ
โ QQuery 11 โ 2024.45 / 2055.84 ยฑ24.35 / 2093.25 ms โ 2051.69 / 2062.80 ยฑ9.24 / 2074.57 ms โ no change โ
โ QQuery 12 โ 196.91 / 200.34 ยฑ2.71 / 204.97 ms โ 197.39 / 204.34 ยฑ5.83 / 214.47 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 20254.05ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 20615.03ms โ
โ Average Time (HEAD) โ 1558.00ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 1585.77ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 12 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 102.1s |
| Peak memory | 33.3 GiB |
| Avg memory | 28.0 GiB |
| CPU user | 992.5s |
| CPU sys | 46.7s |
| Peak spill | 0 B |
**clickbench_extended โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 103.9s |
| Peak memory | 31.9 GiB |
| Avg memory | 26.7 GiB |
| CPU user | 995.4s |
| CPU sys | 51.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4257015612
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4256919523)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.19 / 4.47 ยฑ6.39 / 17.26 ms โ 1.20 / 4.47 ยฑ6.38 / 17.22 ms โ no change โ
โ QQuery 1 โ 14.66 / 14.85 ยฑ0.11 / 14.96 ms โ 14.01 / 14.52 ยฑ0.26 / 14.72 ms โ no change โ
โ QQuery 2 โ 43.87 / 44.43 ยฑ0.55 / 45.49 ms โ 43.85 / 44.06 ยฑ0.24 / 44.50 ms โ no change โ
โ QQuery 3 โ 44.20 / 45.28 ยฑ1.07 / 47.25 ms โ 41.92 / 44.02 ยฑ1.74 / 46.75 ms โ no change โ
โ QQuery 4 โ 287.23 / 301.13 ยฑ9.05 / 313.40 ms โ 292.16 / 295.84 ยฑ5.89 / 307.52 ms โ no change โ
โ QQuery 5 โ 342.41 / 350.29 ยฑ4.20 / 354.91 ms โ 342.18 / 347.48 ยฑ3.79 / 353.88 ms โ no change โ
โ QQuery 6 โ 5.62 / 6.60 ยฑ1.41 / 9.39 ms โ 6.30 / 8.92 ยฑ2.97 / 14.47 ms โ 1.35x slower โ
โ QQuery 7 โ 17.01 / 17.27 ยฑ0.37 / 17.99 ms โ 16.70 / 16.95 ยฑ0.49 / 17.93 ms โ no change โ
โ QQuery 8 โ 409.75 / 423.90 ยฑ11.37 / 441.07 ms โ 417.89 / 427.37 ยฑ7.85 / 436.69 ms โ no change โ
โ QQuery 9 โ 636.24 / 642.22 ยฑ4.65 / 648.36 ms โ 569.47 / 577.71 ยฑ4.71 / 583.28 ms โ +1.11x faster โ
โ QQuery 10 โ 91.33 / 92.69 ยฑ1.88 / 96.35 ms โ 91.14 / 92.90 ยฑ2.50 / 97.69 ms โ no change โ
โ QQuery 11 โ 104.21 / 104.56 ยฑ0.37 / 105.06 ms โ 102.61 / 104.10 ยฑ1.20 / 106.25 ms โ no change โ
โ QQuery 12 โ 336.12 / 341.30 ยฑ3.28 / 345.46 ms โ 337.69 / 344.32 ยฑ3.39 / 347.28 ms โ no change โ
โ QQuery 13 โ 453.67 / 461.54 ยฑ4.78 / 466.44 ms โ 448.70 / 467.12 ยฑ15.33 / 485.30 ms โ no change โ
โ QQuery 14 โ 335.84 / 342.79 ยฑ5.20 / 350.12 ms โ 342.89 / 345.80 ยฑ2.48 / 349.25 ms โ no change โ
โ QQuery 15 โ 352.52 / 363.73 ยฑ13.08 / 387.15 ms โ 356.70 / 377.72 ยฑ20.23 / 410.80 ms โ no change โ
โ QQuery 16 โ 703.85 / 720.03 ยฑ18.35 / 751.30 ms โ 712.37 / 721.66 ยฑ12.53 / 746.37 ms โ no change โ
โ QQuery 17 โ 700.99 / 704.97 ยฑ4.27 / 712.93 ms โ 708.64 / 712.75 ยฑ3.87 / 717.86 ms โ no change โ
โ QQuery 18 โ 1424.33 / 1471.68 ยฑ31.27 / 1510.65 ms โ 1350.35 / 1368.19 ยฑ11.76 / 1383.87 ms โ +1.08x faster โ
โ QQuery 19 โ 35.39 / 38.04 ยฑ3.02 / 43.73 ms โ 34.45 / 35.31 ยฑ0.74 / 36.63 ms โ +1.08x faster โ
โ QQuery 20 โ 715.60 / 743.43 ยฑ28.90 / 779.75 ms โ 712.85 / 720.27 ยฑ9.00 / 737.64 ms โ no change โ
โ QQuery 21 โ 764.37 / 769.58 ยฑ5.11 / 779.13 ms โ 760.91 / 765.16 ยฑ2.42 / 767.72 ms โ no change โ
โ QQuery 22 โ 1131.17 / 1137.60 ยฑ6.55 / 1147.98 ms โ 1124.95 / 1134.03 ยฑ6.62 / 1144.72 ms โ no change โ
โ QQuery 23 โ 3078.34 / 3096.80 ยฑ26.58 / 3149.64 ms โ 3064.01 / 3083.60 ยฑ15.03 / 3109.65 ms โ no change โ
โ QQuery 24 โ 98.57 / 101.05 ยฑ1.62 / 102.83 ms โ 102.12 / 105.83 ยฑ2.56 / 109.89 ms โ no change โ
โ QQuery 25 โ 137.01 / 139.89 ยฑ2.62 / 144.33 ms โ 138.30 / 141.38 ยฑ1.98 / 143.89 ms โ no change โ
โ QQuery 26 โ 97.41 / 101.24 ยฑ2.23 / 104.09 ms โ 101.09 / 102.11 ยฑ0.97 / 103.87 ms โ no change โ
โ QQuery 27 โ 847.61 / 857.29 ยฑ9.02 / 874.36 ms โ 851.11 / 855.35 ยฑ3.95 / 860.19 ms โ no change โ
โ QQuery 28 โ 3261.48 / 3280.49 ยฑ11.39 / 3291.21 ms โ 3225.97 / 3270.69 ยฑ27.00 / 3303.62 ms โ no change โ
โ QQuery 29 โ 50.64 / 54.24 ยฑ3.56 / 59.52 ms โ 52.98 / 56.02 ยฑ1.98 / 59.08 ms โ no change โ
โ QQuery 30 โ 356.72 / 364.30 ยฑ6.17 / 374.35 ms โ 366.93 / 373.40 ยฑ5.02 / 381.86 ms โ no change โ
โ QQuery 31 โ 364.42 / 381.30 ยฑ12.74 / 397.21 ms โ 378.61 / 382.50 ยฑ6.20 / 394.80 ms โ no change โ
โ QQuery 32 โ 1037.67 / 1052.61 ยฑ16.68 / 1085.11 ms โ 1029.75 / 1047.60 ยฑ16.75 / 1071.73 ms โ no change โ
โ QQuery 33 โ 1431.66 / 1455.50 ยฑ15.60 / 1475.14 ms โ 1434.56 / 1451.41 ยฑ13.90 / 1473.99 ms โ no change โ
โ QQuery 34 โ 1443.40 / 1461.03 ยฑ21.09 / 1495.10 ms โ 1457.67 / 1487.64 ยฑ18.05 / 1504.90 ms โ no change โ
โ QQuery 35 โ 390.99 / 393.43 ยฑ2.21 / 397.23 ms โ 384.48 / 390.40 ยฑ5.51 / 400.04 ms โ no change โ
โ QQuery 36 โ 121.15 / 124.09 ยฑ1.66 / 125.70 ms โ 115.76 / 120.43 ยฑ3.25 / 125.96 ms โ no change โ
โ QQuery 37 โ 47.69 / 49.70 ยฑ1.94 / 52.56 ms โ 49.61 / 50.47 ยฑ0.61 / 51.18 ms โ no change โ
โ QQuery 38 โ 77.52 / 79.13 ยฑ1.13 / 80.45 ms โ 74.90 / 77.50 ยฑ1.53 / 79.30 ms โ no change โ
โ QQuery 39 โ 203.22 / 217.31 ยฑ7.37 / 223.35 ms โ 205.85 / 214.47 ยฑ6.15 / 221.24 ms โ no change โ
โ QQuery 40 โ 24.53 / 26.47 ยฑ1.48 / 29.05 ms โ 24.19 / 25.03 ยฑ0.45 / 25.54 ms โ +1.06x faster โ
โ QQuery 41 โ 19.70 / 21.77 ยฑ1.82 / 25.17 ms โ 19.62 / 21.19 ยฑ1.32 / 23.38 ms โ no change โ
โ QQuery 42 โ 19.43 / 20.15 ยฑ0.50 / 20.72 ms โ 19.41 / 19.83 ยฑ0.39 / 20.35 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 22420.16ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 22247.52ms โ
โ Average Time (HEAD) โ 521.40ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 517.38ms โ
โ Queries Faster โ 4 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 38 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 113.2s |
| Peak memory | 41.3 GiB |
| Avg memory | 29.9 GiB |
| CPU user | 1069.8s |
| CPU sys | 82.5s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 112.3s |
| Peak memory | 38.5 GiB |
| Avg memory | 32.2 GiB |
| CPU user | 1066.4s |
| CPU sys | 79.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266344071
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266245112)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 6.63 / 7.06 ยฑ0.83 / 8.73 ms โ 6.67 / 7.09 ยฑ0.75 / 8.58 ms โ no change โ
โ QQuery 2 โ 144.19 / 145.13 ยฑ0.65 / 146.20 ms โ 144.25 / 145.36 ยฑ0.79 / 146.66 ms โ no change โ
โ QQuery 3 โ 113.71 / 114.38 ยฑ0.49 / 115.05 ms โ 114.46 / 115.19 ยฑ0.59 / 115.81 ms โ no change โ
โ QQuery 4 โ 1282.94 / 1318.08 ยฑ19.10 / 1338.56 ms โ 1284.50 / 1308.21 ยฑ17.51 / 1327.05 ms โ no change โ
โ QQuery 5 โ 174.91 / 176.11 ยฑ0.82 / 177.01 ms โ 173.47 / 174.86 ยฑ1.21 / 176.77 ms โ no change โ
โ QQuery 6 โ 814.02 / 841.47 ยฑ20.34 / 869.69 ms โ 823.90 / 843.78 ยฑ14.43 / 863.33 ms โ no change โ
โ QQuery 7 โ 342.02 / 345.27 ยฑ2.46 / 348.52 ms โ 342.50 / 344.84 ยฑ2.86 / 350.36 ms โ no change โ
โ QQuery 8 โ 116.02 / 116.73 ยฑ0.75 / 117.75 ms โ 115.88 / 116.69 ยฑ0.48 / 117.19 ms โ no change โ
โ QQuery 9 โ 96.89 / 103.10 ยฑ3.96 / 107.73 ms โ 100.49 / 102.59 ยฑ2.43 / 106.95 ms โ no change โ
โ QQuery 10 โ 106.78 / 108.41 ยฑ1.31 / 110.12 ms โ 107.79 / 109.08 ยฑ1.06 / 110.80 ms โ no change โ
โ QQuery 11 โ 881.96 / 909.87 ยฑ15.42 / 925.09 ms โ 876.74 / 889.85 ยฑ9.80 / 903.48 ms โ no change โ
โ QQuery 12 โ 44.26 / 46.44 ยฑ1.99 / 50.14 ms โ 44.25 / 45.43 ยฑ0.88 / 46.23 ms โ no change โ
โ QQuery 13 โ 396.92 / 399.34 ยฑ2.51 / 402.71 ms โ 398.62 / 400.73 ยฑ2.18 / 404.85 ms โ no change โ
โ QQuery 14 โ 989.26 / 1007.45 ยฑ9.83 / 1017.74 ms โ 997.67 / 1001.20 ยฑ3.37 / 1006.85 ms โ no change โ
โ QQuery 15 โ 16.11 / 17.36 ยฑ0.77 / 18.18 ms โ 15.37 / 16.74 ยฑ1.40 / 19.39 ms โ no change โ
โ QQuery 16 โ 7.14 / 7.28 ยฑ0.18 / 7.63 ms โ 7.05 / 7.87 ยฑ0.82 / 9.38 ms โ 1.08x slower โ
โ QQuery 17 โ 227.40 / 229.30 ยฑ1.68 / 231.86 ms โ 226.27 / 228.72 ยฑ1.50 / 230.40 ms โ no change โ
โ QQuery 18 โ 126.92 / 127.24 ยฑ0.29 / 127.70 ms โ 125.79 / 127.19 ยฑ1.04 / 128.45 ms โ no change โ
โ QQuery 19 โ 154.50 / 155.90 ยฑ1.36 / 158.29 ms โ 156.68 / 157.65 ยฑ0.98 / 159.01 ms โ no change โ
โ QQuery 20 โ 14.00 / 14.19 ยฑ0.17 / 14.46 ms โ 13.40 / 14.01 ยฑ0.38 / 14.45 ms โ no change โ
โ QQuery 21 โ 18.63 / 19.30 ยฑ0.38 / 19.79 ms โ 19.40 / 19.97 ยฑ0.54 / 20.90 ms โ no change โ
โ QQuery 22 โ 480.99 / 484.81 ยฑ3.87 / 491.43 ms โ 480.96 / 485.98 ยฑ3.53 / 490.66 ms โ no change โ
โ QQuery 23 โ 854.94 / 866.54 ยฑ7.06 / 876.22 ms โ 867.05 / 876.14 ยฑ6.21 / 883.08 ms โ no change โ
โ QQuery 24 โ 381.08 / 384.66 ยฑ1.97 / 386.22 ms โ 379.30 / 382.21 ยฑ2.01 / 384.23 ms โ no change โ
โ QQuery 25 โ 338.53 / 340.94 ยฑ1.78 / 343.48 ms โ 338.73 / 339.97 ยฑ1.05 / 341.89 ms โ no change โ
โ QQuery 26 โ 81.92 / 83.78 ยฑ1.36 / 85.38 ms โ 80.83 / 83.23 ยฑ1.72 / 85.65 ms โ no change โ
โ QQuery 27 โ 6.79 / 7.27 ยฑ0.41 / 7.72 ms โ 6.86 / 7.76 ยฑ1.00 / 9.52 ms โ 1.07x slower โ
โ QQuery 28 โ 147.89 / 149.23 ยฑ1.65 / 152.33 ms โ 148.23 / 148.80 ยฑ0.53 / 149.69 ms โ no change โ
โ QQuery 29 โ 279.44 / 282.14 ยฑ1.54 / 284.24 ms โ 279.32 / 282.48 ยฑ1.77 / 284.10 ms โ no change โ
โ QQuery 30 โ 41.91 / 44.34 ยฑ2.23 / 47.35 ms โ 41.70 / 43.97 ยฑ2.55 / 48.82 ms โ no change โ
โ QQuery 31 โ 168.67 / 171.64 ยฑ2.52 / 176.29 ms โ 171.16 / 173.75 ยฑ2.07 / 176.07 ms โ no change โ
โ QQuery 32 โ 56.89 / 57.63 ยฑ0.49 / 58.32 ms โ 56.67 / 58.49 ยฑ1.52 / 61.15 ms โ no change โ
โ QQuery 33 โ 140.37 / 141.78 ยฑ0.92 / 143.10 ms โ 141.86 / 143.59 ยฑ1.65 / 146.75 ms โ no change โ
โ QQuery 34 โ 6.92 / 7.16 ยฑ0.30 / 7.74 ms โ 7.02 / 7.25 ยฑ0.17 / 7.49 ms โ no change โ
โ QQuery 35 โ 106.07 / 108.18 ยฑ1.29 / 110.09 ms โ 107.33 / 110.22 ยฑ1.50 / 111.56 ms โ no change โ
โ QQuery 36 โ 6.65 / 7.26 ยฑ0.83 / 8.91 ms โ 6.74 / 6.98 ยฑ0.24 / 7.42 ms โ no change โ
โ QQuery 37 โ 8.26 / 8.93 ยฑ0.85 / 10.57 ms โ 8.09 / 8.34 ยฑ0.28 / 8.86 ms โ +1.07x faster โ
โ QQuery 38 โ 83.03 / 87.84 ยฑ4.01 / 93.88 ms โ 84.23 / 87.88 ยฑ3.77 / 94.59 ms โ no change โ
โ QQuery 39 โ 124.70 / 127.23 ยฑ1.33 / 128.40 ms โ 124.20 / 125.57 ยฑ0.92 / 127.01 ms โ no change โ
โ QQuery 40 โ 108.99 / 114.59 ยฑ5.52 / 124.80 ms โ 107.86 / 113.67 ยฑ6.27 / 125.79 ms โ no change โ
โ QQuery 41 โ 13.82 / 14.41 ยฑ0.37 / 14.88 ms โ 14.19 / 15.43 ยฑ1.16 / 16.88 ms โ 1.07x slower โ
โ QQuery 42 โ 106.67 / 108.27 ยฑ1.43 / 110.11 ms โ 109.04 / 110.72 ยฑ1.05 / 111.88 ms โ no change โ
โ QQuery 43 โ 5.95 / 6.56 ยฑ1.00 / 8.53 ms โ 5.95 / 6.15 ยฑ0.20 / 6.52 ms โ +1.07x faster โ
โ QQuery 44 โ 11.38 / 11.73 ยฑ0.20 / 11.99 ms โ 11.84 / 12.55 ยฑ0.74 / 13.75 ms โ 1.07x slower โ
โ QQuery 45 โ 49.79 / 51.73 ยฑ1.78 / 54.37 ms โ 51.19 / 51.71 ยฑ0.35 / 52.24 ms โ no change โ
โ QQuery 46 โ 8.46 / 8.76 ยฑ0.30 / 9.13 ms โ 8.38 / 8.75 ยฑ0.48 / 9.70 ms โ no change โ
โ QQuery 47 โ 687.41 / 696.91 ยฑ5.25 / 702.59 ms โ 687.35 / 698.02 ยฑ7.13 / 705.11 ms โ no change โ
โ QQuery 48 โ 285.68 / 294.83 ยฑ5.36 / 300.92 ms โ 287.69 / 292.01 ยฑ2.32 / 294.47 ms โ no change โ
โ QQuery 49 โ 252.72 / 256.20 ยฑ3.96 / 263.65 ms โ 251.16 / 253.86 ยฑ3.92 / 261.58 ms โ no change โ
โ QQuery 50 โ 224.73 / 226.71 ยฑ1.69 / 229.72 ms โ 216.89 / 221.64 ยฑ3.03 / 225.40 ms โ no change โ
โ QQuery 51 โ 182.53 / 185.27 ยฑ2.26 / 188.52 ms โ 180.98 / 182.73 ยฑ1.45 / 185.02 ms โ no change โ
โ QQuery 52 โ 107.72 / 108.22 ยฑ0.50 / 109.17 ms โ 107.78 / 110.26 ยฑ1.80 / 112.55 ms โ no change โ
โ QQuery 53 โ 103.00 / 104.48 ยฑ1.43 / 106.80 ms โ 103.30 / 104.34 ยฑ0.87 / 105.66 ms โ no change โ
โ QQuery 54 โ 145.21 / 148.67 ยฑ1.82 / 150.45 ms โ 145.91 / 148.05 ยฑ1.63 / 150.94 ms โ no change โ
โ QQuery 55 โ 106.81 / 107.68 ยฑ0.77 / 109.11 ms โ 107.02 / 108.27 ยฑ0.77 / 109.38 ms โ no change โ
โ QQuery 56 โ 141.09 / 142.78 ยฑ1.43 / 144.51 ms โ 142.08 / 142.92 ยฑ0.64 / 143.95 ms โ no change โ
โ QQuery 57 โ 173.31 / 174.96 ยฑ1.38 / 176.71 ms โ 172.55 / 174.97 ยฑ1.86 / 176.99 ms โ no change โ
โ QQuery 58 โ 291.89 / 300.97 ยฑ9.58 / 318.18 ms โ 292.21 / 300.76 ยฑ7.36 / 310.61 ms โ no change โ
โ QQuery 59 โ 196.30 / 198.33 ยฑ1.43 / 200.25 ms โ 197.30 / 198.33 ยฑ1.45 / 201.20 ms โ no change โ
โ QQuery 60 โ 142.23 / 144.77 ยฑ1.46 / 146.20 ms โ 143.66 / 145.93 ยฑ1.83 / 148.44 ms โ no change โ
โ QQuery 61 โ 12.97 / 13.32 ยฑ0.31 / 13.90 ms โ 12.79 / 13.14 ยฑ0.31 / 13.52 ms โ no change โ
โ QQuery 62 โ 878.34 / 918.84 ยฑ40.50 / 990.26 ms โ 900.52 / 922.61 ยฑ23.27 / 962.50 ms โ no change โ
โ QQuery 63 โ 102.43 / 106.16 ยฑ3.12 / 111.71 ms โ 104.06 / 106.82 ยฑ2.44 / 110.92 ms โ no change โ
โ QQuery 64 โ 680.41 / 686.87 ยฑ6.05 / 696.76 ms โ 673.92 / 683.35 ยฑ5.78 / 690.72 ms โ no change โ
โ QQuery 65 โ 247.09 / 251.85 ยฑ4.39 / 260.16 ms โ 251.94 / 253.60 ยฑ1.58 / 256.39 ms โ no change โ
โ QQuery 66 โ 247.89 / 257.10 ยฑ5.33 / 263.61 ms โ 246.15 / 258.82 ยฑ7.13 / 267.58 ms โ no change โ
โ QQuery 67 โ 304.94 / 311.14 ยฑ4.50 / 317.63 ms โ 312.82 / 318.36 ยฑ3.39 / 321.94 ms โ no change โ
โ QQuery 68 โ 8.86 / 10.51 ยฑ1.28 / 12.02 ms โ 8.50 / 10.65 ยฑ1.79 / 13.07 ms โ no change โ
โ QQuery 69 โ 99.96 / 106.63 ยฑ4.38 / 113.11 ms โ 102.55 / 106.18 ยฑ3.38 / 111.99 ms โ no change โ
โ QQuery 70 โ 334.03 / 347.27 ยฑ14.95 / 376.18 ms โ 321.19 / 355.41 ยฑ19.77 / 377.33 ms โ no change โ
โ QQuery 71 โ 134.66 / 138.15 ยฑ5.88 / 149.89 ms โ 133.81 / 136.32 ยฑ1.82 / 138.41 ms โ no change โ
โ QQuery 72 โ 621.88 / 626.18 ยฑ3.49 / 629.78 ms โ 621.32 / 625.37 ยฑ2.47 / 628.19 ms โ no change โ
โ QQuery 73 โ 7.15 / 7.92 ยฑ0.47 / 8.49 ms โ 6.64 / 8.15 ยฑ1.32 / 10.42 ms โ no change โ
โ QQuery 74 โ 545.74 / 561.73 ยฑ9.64 / 573.88 ms โ 545.73 / 555.92 ยฑ5.89 / 562.76 ms โ no change โ
โ QQuery 75 โ 273.40 / 277.33 ยฑ2.84 / 281.17 ms โ 274.74 / 277.05 ยฑ1.85 / 279.47 ms โ no change โ
โ QQuery 76 โ 132.00 / 133.09 ยฑ0.77 / 134.07 ms โ 130.36 / 132.92 ยฑ2.14 / 136.59 ms โ no change โ
โ QQuery 77 โ 187.61 / 190.26 ยฑ2.05 / 193.47 ms โ 188.41 / 189.83 ยฑ1.52 / 192.74 ms โ no change โ
โ QQuery 78 โ 340.10 / 342.75 ยฑ2.70 / 347.27 ms โ 339.97 / 341.95 ยฑ1.30 / 343.56 ms โ no change โ
โ QQuery 79 โ 230.47 / 234.45 ยฑ3.19 / 238.15 ms โ 228.06 / 231.26 ยฑ1.98 / 233.86 ms โ no change โ
โ QQuery 80 โ 319.87 / 323.28 ยฑ3.12 / 328.07 ms โ 320.08 / 322.13 ยฑ2.62 / 327.18 ms โ no change โ
โ QQuery 81 โ 26.13 / 27.18 ยฑ0.99 / 29.00 ms โ 26.41 / 27.40 ยฑ1.38 / 30.10 ms โ no change โ
โ QQuery 82 โ 200.07 / 202.41 ยฑ1.48 / 204.53 ms โ 201.99 / 204.86 ยฑ2.26 / 207.82 ms โ no change โ
โ QQuery 83 โ 38.26 / 39.97 ยฑ1.35 / 42.08 ms โ 38.80 / 39.53 ยฑ0.86 / 41.05 ms โ no change โ
โ QQuery 84 โ 47.50 / 48.93 ยฑ0.96 / 50.51 ms โ 47.95 / 48.96 ยฑ0.58 / 49.72 ms โ no change โ
โ QQuery 85 โ 147.69 / 149.26 ยฑ1.35 / 151.78 ms โ 148.84 / 151.45 ยฑ1.76 / 153.85 ms โ no change โ
โ QQuery 86 โ 39.17 / 39.96 ยฑ0.74 / 41.25 ms โ 39.18 / 39.74 ยฑ0.36 / 40.25 ms โ no change โ
โ QQuery 87 โ 84.81 / 88.47 ยฑ3.83 / 95.70 ms โ 86.16 / 88.82 ยฑ2.57 / 93.68 ms โ no change โ
โ QQuery 88 โ 99.28 / 100.69 ยฑ1.03 / 102.37 ms โ 100.81 / 101.45 ยฑ0.59 / 102.26 ms โ no change โ
โ QQuery 89 โ 118.84 / 119.81 ยฑ0.97 / 121.45 ms โ 119.87 / 120.68 ยฑ0.73 / 121.75 ms โ no change โ
โ QQuery 90 โ 23.43 / 24.17 ยฑ0.42 / 24.72 ms โ 24.09 / 24.52 ยฑ0.29 / 24.88 ms โ no change โ
โ QQuery 91 โ 62.95 / 64.72 ยฑ1.37 / 66.19 ms โ 63.00 / 63.67 ยฑ0.64 / 64.75 ms โ no change โ
โ QQuery 92 โ 56.47 / 57.78 ยฑ1.34 / 60.31 ms โ 57.48 / 58.72 ยฑ0.93 / 59.72 ms โ no change โ
โ QQuery 93 โ 185.94 / 187.64 ยฑ1.25 / 189.67 ms โ 186.54 / 188.66 ยฑ1.29 / 190.51 ms โ no change โ
โ QQuery 94 โ 59.93 / 61.41 ยฑ1.02 / 62.86 ms โ 61.77 / 62.62 ยฑ0.80 / 64.04 ms โ no change โ
โ QQuery 95 โ 127.28 / 129.13 ยฑ1.68 / 132.29 ms โ 128.91 / 129.36 ยฑ0.31 / 129.79 ms โ no change โ
โ QQuery 96 โ 74.32 / 74.94 ยฑ0.54 / 75.92 ms โ 71.70 / 74.94 ยฑ1.72 / 76.63 ms โ no change โ
โ QQuery 97 โ 125.37 / 126.54 ยฑ0.77 / 127.31 ms โ 126.12 / 129.31 ยฑ2.08 / 132.21 ms โ no change โ
โ QQuery 98 โ 151.55 / 153.87 ยฑ2.64 / 158.57 ms โ 154.25 / 155.29 ยฑ0.60 / 156.12 ms โ no change โ
โ QQuery 99 โ 10735.72 / 10777.16 ยฑ25.59 / 10809.03 ms โ 10745.11 / 10814.03 ยฑ51.36 / 10899.76 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 31296.46ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 31324.53ms โ
โ Average Time (HEAD) โ 316.13ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 316.41ms โ
โ Queries Faster โ 2 โ
โ Queries Slower โ 4 โ
โ Queries with No Change โ 93 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 156.8s |
| Peak memory | 5.5 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 258.1s |
| CPU sys | 17.1s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 156.9s |
| Peak memory | 5.7 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 258.0s |
| CPU sys | 16.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4262989201
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4262974208)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4262974208-1399-lr47q 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (12d314f24e36f620a282e430aa389a38f4cd6660) to 4b8c1d9 (merge-base) [diff](https://github.com/apache/datafusion/compare/4b8c1d9d6b4eb7ebb2876abbed911055b4badfad..12d314f24e36f620a282e430aa389a38f4cd6660) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266139061
@Dandandan , pushed both optimizations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4262118162
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4261984688)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 6.88 / 7.28 ยฑ0.73 / 8.74 ms โ 7.02 / 7.45 ยฑ0.75 / 8.95 ms โ no change โ
โ QQuery 2 โ 144.09 / 145.32 ยฑ0.70 / 146.02 ms โ 145.81 / 146.21 ยฑ0.38 / 146.81 ms โ no change โ
โ QQuery 3 โ 113.46 / 114.19 ยฑ0.58 / 115.09 ms โ 114.13 / 114.52 ยฑ0.34 / 114.97 ms โ no change โ
โ QQuery 4 โ 1416.02 / 1435.82 ยฑ18.39 / 1469.20 ms โ 1415.50 / 1441.76 ยฑ18.09 / 1462.78 ms โ no change โ
โ QQuery 5 โ 173.17 / 175.22 ยฑ1.41 / 177.56 ms โ 174.31 / 175.59 ยฑ1.05 / 177.52 ms โ no change โ
โ QQuery 6 โ 852.37 / 875.94 ยฑ27.10 / 909.33 ms โ 851.97 / 878.29 ยฑ28.64 / 919.26 ms โ no change โ
โ QQuery 7 โ 344.62 / 347.31 ยฑ2.44 / 351.00 ms โ 346.94 / 349.51 ยฑ1.95 / 352.13 ms โ no change โ
โ QQuery 8 โ 116.85 / 118.12 ยฑ0.94 / 119.12 ms โ 116.98 / 118.85 ยฑ1.32 / 120.54 ms โ no change โ
โ QQuery 9 โ 101.31 / 103.11 ยฑ2.18 / 107.17 ms โ 102.85 / 110.57 ยฑ10.13 / 130.42 ms โ 1.07x slower โ
โ QQuery 10 โ 107.52 / 108.38 ยฑ0.63 / 109.37 ms โ 107.83 / 109.04 ยฑ1.22 / 111.39 ms โ no change โ
โ QQuery 11 โ 1003.61 / 1023.63 ยฑ11.94 / 1035.88 ms โ 1022.03 / 1032.20 ยฑ9.15 / 1046.44 ms โ no change โ
โ QQuery 12 โ 46.95 / 49.23 ยฑ1.19 / 50.19 ms โ 45.62 / 47.31 ยฑ1.15 / 49.18 ms โ no change โ
โ QQuery 13 โ 408.88 / 411.19 ยฑ2.32 / 415.00 ms โ 408.46 / 412.76 ยฑ2.99 / 416.13 ms โ no change โ
โ QQuery 14 โ 1020.84 / 1034.42 ยฑ7.80 / 1044.24 ms โ 1011.32 / 1027.23 ยฑ11.18 / 1045.61 ms โ no change โ
โ QQuery 15 โ 17.45 / 18.20 ยฑ0.73 / 19.27 ms โ 16.21 / 18.44 ยฑ2.23 / 22.42 ms โ no change โ
โ QQuery 16 โ 7.36 / 8.06 ยฑ0.70 / 9.32 ms โ 7.58 / 7.80 ยฑ0.20 / 8.07 ms โ no change โ
โ QQuery 17 โ 233.66 / 235.48 ยฑ1.22 / 237.08 ms โ 232.98 / 235.82 ยฑ2.94 / 239.99 ms โ no change โ
โ QQuery 18 โ 128.54 / 131.36 ยฑ1.90 / 134.39 ms โ 130.67 / 131.46 ยฑ0.79 / 132.80 ms โ no change โ
โ QQuery 19 โ 157.98 / 159.62 ยฑ1.19 / 161.69 ms โ 159.87 / 162.09 ยฑ2.06 / 165.36 ms โ no change โ
โ QQuery 20 โ 14.65 / 15.14 ยฑ0.46 / 15.94 ms โ 14.80 / 14.96 ยฑ0.10 / 15.12 ms โ no change โ
โ QQuery 21 โ 20.30 / 20.82 ยฑ0.50 / 21.72 ms โ 20.13 / 20.48 ยฑ0.19 / 20.68 ms โ no change โ
โ QQuery 22 โ 498.38 / 503.58 ยฑ4.34 / 510.94 ms โ 490.81 / 496.58 ยฑ5.16 / 505.21 ms โ no change โ
โ QQuery 23 โ 918.62 / 923.35 ยฑ3.11 / 927.85 ms โ 905.54 / 922.02 ยฑ8.60 / 929.75 ms โ no change โ
โ QQuery 24 โ 391.52 / 395.01 ยฑ1.82 / 396.78 ms โ 393.02 / 396.25 ยฑ2.88 / 400.28 ms โ no change โ
โ QQuery 25 โ 347.54 / 349.91 ยฑ2.35 / 354.17 ms โ 350.53 / 351.07 ยฑ0.49 / 351.86 ms โ no change โ
โ QQuery 26 โ 82.06 / 84.58 ยฑ1.64 / 86.51 ms โ 81.78 / 83.79 ยฑ1.55 / 85.41 ms โ no change โ
โ QQuery 27 โ 7.13 / 7.36 ยฑ0.21 / 7.75 ms โ 7.27 / 8.02 ยฑ0.78 / 9.03 ms โ 1.09x slower โ
โ QQuery 28 โ 149.68 / 151.39 ยฑ1.27 / 153.50 ms โ 152.09 / 153.24 ยฑ1.08 / 155.07 ms โ no change โ
โ QQuery 29 โ 288.29 / 290.05 ยฑ1.21 / 291.37 ms โ 289.19 / 290.40 ยฑ0.75 / 291.52 ms โ no change โ
โ QQuery 30 โ 43.92 / 45.92 ยฑ1.42 / 48.04 ms โ 45.86 / 46.98 ยฑ0.84 / 47.99 ms โ no change โ
โ QQuery 31 โ 171.61 / 173.57 ยฑ1.19 / 175.32 ms โ 173.31 / 175.16 ยฑ1.46 / 177.41 ms โ no change โ
โ QQuery 32 โ 58.31 / 59.40 ยฑ0.91 / 60.58 ms โ 58.29 / 60.12 ยฑ1.19 / 61.84 ms โ no change โ
โ QQuery 33 โ 142.13 / 144.60 ยฑ1.57 / 146.32 ms โ 143.25 / 144.38 ยฑ0.70 / 145.29 ms โ no change โ
โ QQuery 34 โ 7.24 / 7.89 ยฑ0.84 / 9.55 ms โ 7.63 / 7.81 ยฑ0.19 / 8.14 ms โ no change โ
โ QQuery 35 โ 109.93 / 110.25 ยฑ0.45 / 111.11 ms โ 111.02 / 112.54 ยฑ1.12 / 114.35 ms โ no change โ
โ QQuery 36 โ 6.90 / 7.13 ยฑ0.19 / 7.38 ms โ 7.06 / 7.19 ยฑ0.08 / 7.29 ms โ no change โ
โ QQuery 37 โ 8.71 / 9.01 ยฑ0.19 / 9.22 ms โ 8.71 / 9.06 ยฑ0.49 / 10.02 ms โ no change โ
โ QQuery 38 โ 86.59 / 90.83 ยฑ2.97 / 95.71 ms โ 86.53 / 90.30 ยฑ3.55 / 97.02 ms โ no change โ
โ QQuery 39 โ 129.69 / 132.58 ยฑ2.38 / 136.72 ms โ 130.19 / 133.09 ยฑ1.79 / 135.01 ms โ no change โ
โ QQuery 40 โ 114.03 / 119.62 ยฑ6.44 / 130.40 ms โ 116.56 / 119.29 ยฑ3.46 / 125.86 ms โ no change โ
โ QQuery 41 โ 14.70 / 15.54 ยฑ0.56 / 16.46 ms โ 15.18 / 16.16 ยฑ0.61 / 17.09 ms โ no change โ
โ QQuery 42 โ 107.17 / 109.52 ยฑ1.62 / 111.80 ms โ 108.91 / 110.71 ยฑ1.11 / 112.06 ms โ no change โ
โ QQuery 43 โ 6.19 / 6.32 ยฑ0.08 / 6.45 ms โ 6.18 / 6.32 ยฑ0.17 / 6.65 ms โ no change โ
โ QQuery 44 โ 11.86 / 12.44 ยฑ0.33 / 12.82 ms โ 12.31 / 12.56 ยฑ0.15 / 12.77 ms โ no change โ
โ QQuery 45 โ 51.67 / 52.92 ยฑ1.20 / 55.12 ms โ 51.63 / 52.35 ยฑ0.74 / 53.47 ms โ no change โ
โ QQuery 46 โ 8.77 / 9.21 ยฑ0.33 / 9.63 ms โ 9.09 / 9.30 ยฑ0.14 / 9.48 ms โ no change โ
โ QQuery 47 โ 774.88 / 783.52 ยฑ6.69 / 792.36 ms โ 779.00 / 783.43 ยฑ3.37 / 787.84 ms โ no change โ
โ QQuery 48 โ 291.32 / 297.13 ยฑ5.26 / 305.14 ms โ 291.18 / 295.80 ยฑ3.01 / 299.61 ms โ no change โ
โ QQuery 49 โ 254.08 / 255.73 ยฑ1.12 / 257.53 ms โ 253.41 / 257.06 ยฑ3.13 / 261.86 ms โ no change โ
โ QQuery 50 โ 232.69 / 236.73 ยฑ2.58 / 239.49 ms โ 230.00 / 233.48 ยฑ2.61 / 237.73 ms โ no change โ
โ QQuery 51 โ 181.29 / 184.30 ยฑ2.82 / 187.79 ms โ 183.72 / 186.46 ยฑ2.70 / 191.33 ms โ no change โ
โ QQuery 52 โ 106.99 / 108.49 ยฑ1.72 / 111.46 ms โ 109.05 / 110.33 ยฑ1.14 / 112.02 ms โ no change โ
โ QQuery 53 โ 102.85 / 103.73 ยฑ0.90 / 105.12 ms โ 103.77 / 104.64 ยฑ1.20 / 107.03 ms โ no change โ
โ QQuery 54 โ 147.19 / 148.85 ยฑ1.41 / 151.08 ms โ 150.00 / 150.82 ยฑ0.56 / 151.61 ms โ no change โ
โ QQuery 55 โ 107.11 / 109.39 ยฑ1.68 / 111.88 ms โ 107.80 / 110.27 ยฑ1.81 / 112.82 ms โ no change โ
โ QQuery 56 โ 140.24 / 142.21 ยฑ1.03 / 143.05 ms โ 142.66 / 144.00 ยฑ0.75 / 144.89 ms โ no change โ
โ QQuery 57 โ 172.65 / 178.30 ยฑ3.26 / 182.63 ms โ 177.79 / 179.56 ยฑ2.06 / 183.33 ms โ no change โ
โ QQuery 58 โ 300.73 / 305.18 ยฑ3.23 / 309.67 ms โ 298.38 / 310.18 ยฑ7.77 / 319.12 ms โ no change โ
โ QQuery 59 โ 198.01 / 201.48 ยฑ2.61 / 205.18 ms โ 202.91 / 204.99 ยฑ2.00 / 208.66 ms โ no change โ
โ QQuery 60 โ 143.88 / 146.39 ยฑ1.41 / 147.76 ms โ 144.46 / 146.83 ยฑ1.56 / 148.75 ms โ no change โ
โ QQuery 61 โ 13.34 / 13.67 ยฑ0.29 / 14.05 ms โ 13.43 / 13.68 ยฑ0.19 / 13.91 ms โ no change โ
โ QQuery 62 โ 926.01 / 945.20 ยฑ10.13 / 953.22 ms โ 932.23 / 971.29 ยฑ24.61 / 1007.50 ms โ no change โ
โ QQuery 63 โ 102.53 / 104.13 ยฑ1.33 / 106.29 ms โ 105.76 / 107.79 ยฑ1.62 / 110.66 ms โ no change โ
โ QQuery 64 โ 701.72 / 705.64 ยฑ2.81 / 710.47 ms โ 705.17 / 711.56 ยฑ6.11 / 721.54 ms โ no change โ
โ QQuery 65 โ 267.45 / 267.90 ยฑ0.35 / 268.42 ms โ 264.63 / 268.91 ยฑ3.58 / 275.49 ms โ no change โ
โ QQuery 66 โ 258.99 / 261.44 ยฑ2.37 / 264.71 ms โ 250.73 / 258.18 ยฑ6.83 / 268.92 ms โ no change โ
โ QQuery 67 โ 317.94 / 328.77 ยฑ8.89 / 342.85 ms โ 324.01 / 330.40 ยฑ7.17 / 342.62 ms โ no change โ
โ QQuery 68 โ 9.41 / 10.92 ยฑ1.37 / 13.34 ms โ 9.15 / 10.30 ยฑ0.99 / 11.52 ms โ +1.06x faster โ
โ QQuery 69 โ 104.16 / 106.11 ยฑ1.96 / 109.64 ms โ 102.41 / 104.93 ยฑ2.36 / 109.08 ms โ no change โ
โ QQuery 70 โ 334.78 / 347.28 ยฑ12.83 / 371.62 ms โ 342.24 / 357.49 ยฑ10.67 / 373.96 ms โ no change โ
โ QQuery 71 โ 135.10 / 136.16 ยฑ0.84 / 137.56 ms โ 136.55 / 138.16 ยฑ0.93 / 139.27 ms โ no change โ
โ QQuery 72 โ 645.83 / 654.53 ยฑ5.95 / 661.77 ms โ 626.87 / 643.35 ยฑ12.94 / 658.83 ms โ no change โ
โ QQuery 73 โ 7.15 / 7.58 ยฑ0.44 / 8.18 ms โ 7.17 / 8.24 ยฑ0.91 / 9.53 ms โ 1.09x slower โ
โ QQuery 74 โ 650.20 / 655.50 ยฑ3.57 / 660.35 ms โ 642.04 / 648.75 ยฑ4.46 / 654.79 ms โ no change โ
โ QQuery 75 โ 277.88 / 280.66 ยฑ1.69 / 283.02 ms โ 280.66 / 283.18 ยฑ1.80 / 285.65 ms โ no change โ
โ QQuery 76 โ 131.65 / 133.80 ยฑ1.50 / 135.95 ms โ 135.30 / 136.53 ยฑ1.17 / 138.39 ms โ no change โ
โ QQuery 77 โ 186.81 / 188.58 ยฑ1.38 / 190.26 ms โ 188.71 / 191.17 ยฑ2.21 / 195.06 ms โ no change โ
โ QQuery 78 โ 345.23 / 352.12 ยฑ5.63 / 359.21 ms โ 345.17 / 350.30 ยฑ4.09 / 357.35 ms โ no change โ
โ QQuery 79 โ 245.72 / 247.62 ยฑ1.45 / 249.45 ms โ 243.85 / 250.14 ยฑ5.00 / 256.25 ms โ no change โ
โ QQuery 80 โ 322.24 / 325.59 ยฑ2.40 / 329.13 ms โ 321.26 / 324.51 ยฑ2.26 / 328.04 ms โ no change โ
โ QQuery 81 โ 26.40 / 27.62 ยฑ1.25 / 29.91 ms โ 27.23 / 28.41 ยฑ1.02 / 29.94 ms โ no change โ
โ QQuery 82 โ 197.19 / 199.03 ยฑ1.00 / 199.76 ms โ 200.83 / 204.35 ยฑ2.47 / 208.32 ms โ no change โ
โ QQuery 83 โ 40.39 / 41.19 ยฑ0.78 / 42.42 ms โ 39.65 / 40.21 ยฑ0.54 / 40.89 ms โ no change โ
โ QQuery 84 โ 49.15 / 50.43 ยฑ1.13 / 52.27 ms โ 48.64 / 49.57 ยฑ0.58 / 50.42 ms โ no change โ
โ QQuery 85 โ 149.43 / 150.62 ยฑ1.13 / 152.60 ms โ 149.73 / 151.96 ยฑ1.44 / 154.22 ms โ no change โ
โ QQuery 86 โ 38.90 / 40.41 ยฑ1.27 / 41.88 ms โ 39.07 / 40.97 ยฑ1.23 / 42.89 ms โ no change โ
โ QQuery 87 โ 87.18 / 91.86 ยฑ3.97 / 97.85 ms โ 87.89 / 91.28 ยฑ3.56 / 97.92 ms โ no change โ
โ QQuery 88 โ 101.44 / 101.85 ยฑ0.65 / 103.14 ms โ 101.48 / 102.39 ยฑ1.04 / 104.25 ms โ no change โ
โ QQuery 89 โ 118.46 / 120.27 ยฑ1.47 / 122.84 ms โ 121.32 / 123.21 ยฑ1.26 / 124.71 ms โ no change โ
โ QQuery 90 โ 23.70 / 24.34 ยฑ0.38 / 24.70 ms โ 23.87 / 24.37 ยฑ0.30 / 24.63 ms โ no change โ
โ QQuery 91 โ 64.21 / 65.84 ยฑ1.53 / 68.17 ms โ 64.35 / 65.87 ยฑ1.49 / 68.54 ms โ no change โ
โ QQuery 92 โ 58.23 / 59.29 ยฑ0.58 / 59.77 ms โ 59.35 / 60.13 ยฑ0.76 / 61.53 ms โ no change โ
โ QQuery 93 โ 191.94 / 193.27 ยฑ0.94 / 194.49 ms โ 191.08 / 193.66 ยฑ1.32 / 194.69 ms โ no change โ
โ QQuery 94 โ 61.99 / 63.24 ยฑ0.83 / 64.61 ms โ 63.80 / 64.86 ยฑ0.96 / 66.63 ms โ no change โ
โ QQuery 95 โ 130.94 / 132.50 ยฑ0.81 / 133.16 ms โ 129.84 / 131.37 ยฑ1.50 / 134.11 ms โ no change โ
โ QQuery 96 โ 70.97 / 74.91 ยฑ1.98 / 76.08 ms โ 71.94 / 75.96 ยฑ2.17 / 78.57 ms โ no change โ
โ QQuery 97 โ 127.10 / 129.15 ยฑ1.96 / 131.90 ms โ 128.11 / 130.33 ยฑ1.18 / 131.56 ms โ no change โ
โ QQuery 98 โ 154.44 / 157.23 ยฑ2.11 / 159.99 ms โ 158.40 / 159.70 ยฑ1.26 / 161.42 ms โ no change โ
โ QQuery 99 โ 10841.57 / 10884.09 ยฑ42.91 / 10962.83 ms โ 10848.38 / 10884.93 ยฑ28.97 / 10926.85 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 32231.59ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 32327.28ms โ
โ Average Time (HEAD) โ 325.57ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 326.54ms โ
โ Queries Faster โ 1 โ
โ Queries Slower โ 3 โ
โ Queries with No Change โ 95 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 161.5s |
| Peak memory | 5.2 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 268.2s |
| CPU sys | 17.2s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 161.9s |
| Peak memory | 5.2 GiB |
| Avg memory | 4.5 GiB |
| CPU user | 268.3s |
| CPU sys | 17.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4276332686
Rebased with main after count distinct benchmark PR got merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266259118
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266245112)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4266245112-1412-nh7vf 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (617e79032b8930cd89a52ec0f27e761d5b71c5c5) to 5a427cb (merge-base) [diff](https://github.com/apache/datafusion/compare/5a427cb55d8630b51b9590a25b369b434611975e..617e79032b8930cd89a52ec0f27e761d5b71c5c5) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230789981
๐ค Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4230786454)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4230786454-1100-wf4xk 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (f0b2a4aab055f557acc50f4e919d92ab57f18c35) to 16e578d (merge-base) [diff](https://github.com/apache/datafusion/compare/16e578dfdd28c8bffc6a21f54164b889e79bf75c..f0b2a4aab055f557acc50f4e919d92ab57f18c35)
BENCH_NAME=clickbench_partitiomed
BENCH_COMMAND=cargo bench --features=parquet --bench clickbench_partitiomed
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230789220
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4230786454)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4230786454-1098-ljnsh 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (f0b2a4aab055f557acc50f4e919d92ab57f18c35) to 16e578d (merge-base) [diff](https://github.com/apache/datafusion/compare/16e578dfdd28c8bffc6a21f54164b889e79bf75c..f0b2a4aab055f557acc50f4e919d92ab57f18c35) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4292788951
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4292778636)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4292778636-1725-6ppgg 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (54ba4a1026fb743fb5fd9198889aec3bf92ea80c) to 90a8117 (merge-base) [diff](https://github.com/apache/datafusion/compare/90a8117be6c54b2b188149e14ab339f2a9608dff..54ba4a1026fb743fb5fd9198889aec3bf92ea80c) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3077716564
##########
datafusion/functions-aggregate/src/count.rs:
##########
@@ -336,18 +337,70 @@ impl AggregateUDFImpl for Count {
}
fn groups_accumulator_supported(&self, args: AccumulatorArgs) -> bool {
- // groups accumulator only supports `COUNT(c1)`, not
- // `COUNT(c1, c2)`, etc
- if args.is_distinct {
+ if args.exprs.len() != 1 {
return false;
}
- args.exprs.len() == 1
+ if args.is_distinct {
+ // Only support primitive integer types for now
+ matches!(
+ args.expr_fields[0].data_type(),
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::UInt8
+ | DataType::UInt16
+ | DataType::UInt32
+ | DataType::UInt64
+ )
+ } else {
+ true
+ }
}
fn create_groups_accumulator(
&self,
- _args: AccumulatorArgs,
+ args: AccumulatorArgs,
) -> Result<Box<dyn GroupsAccumulator>> {
+ if args.is_distinct {
+ let data_type = args.expr_fields[0].data_type();
+ return match data_type {
+ DataType::Int8 => Ok(Box::new(
Review Comment:
Sorry that link routes me to all open PRs using hashset . Not sure which one would you like me to refer to @Dandandan ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4242011427
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4241897983)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 6.60 / 7.04 ยฑ0.82 / 8.68 ms โ 6.70 / 7.16 ยฑ0.77 / 8.69 ms โ no change โ
โ QQuery 2 โ 145.52 / 147.18 ยฑ1.22 / 149.30 ms โ 145.95 / 147.02 ยฑ0.80 / 148.38 ms โ no change โ
โ QQuery 3 โ 113.30 / 114.09 ยฑ0.53 / 114.86 ms โ 113.76 / 114.32 ยฑ0.45 / 114.82 ms โ no change โ
โ QQuery 4 โ 1290.99 / 1311.47 ยฑ16.55 / 1332.30 ms โ 1314.60 / 1350.74 ยฑ24.14 / 1379.42 ms โ no change โ
โ QQuery 5 โ 171.01 / 172.44 ยฑ0.74 / 173.03 ms โ 172.65 / 173.79 ยฑ1.18 / 176.00 ms โ no change โ
โ QQuery 6 โ 828.87 / 839.28 ยฑ9.41 / 857.01 ms โ 817.44 / 832.66 ยฑ10.55 / 850.02 ms โ no change โ
โ QQuery 7 โ 339.58 / 343.43 ยฑ1.95 / 344.74 ms โ 343.25 / 346.04 ยฑ2.91 / 350.78 ms โ no change โ
โ QQuery 8 โ 116.11 / 117.09 ยฑ0.69 / 117.95 ms โ 114.67 / 116.12 ยฑ0.81 / 117.11 ms โ no change โ
โ QQuery 9 โ 100.93 / 108.92 ยฑ10.56 / 129.63 ms โ 100.81 / 103.38 ยฑ2.42 / 106.68 ms โ +1.05x faster โ
โ QQuery 10 โ 105.07 / 106.85 ยฑ1.63 / 109.64 ms โ 106.72 / 107.94 ยฑ1.22 / 110.07 ms โ no change โ
โ QQuery 11 โ 877.89 / 911.74 ยฑ23.88 / 947.85 ms โ 880.98 / 913.38 ยฑ24.56 / 942.10 ms โ no change โ
โ QQuery 12 โ 43.97 / 46.34 ยฑ1.57 / 48.90 ms โ 44.35 / 46.07 ยฑ0.94 / 47.11 ms โ no change โ
โ QQuery 13 โ 399.22 / 401.42 ยฑ2.14 / 405.48 ms โ 395.70 / 401.18 ยฑ4.02 / 406.78 ms โ no change โ
โ QQuery 14 โ 996.62 / 1007.72 ยฑ7.01 / 1015.67 ms โ 1003.37 / 1008.43 ยฑ4.44 / 1016.01 ms โ no change โ
โ QQuery 15 โ 15.11 / 15.92 ยฑ0.66 / 16.86 ms โ 15.88 / 16.35 ยฑ0.57 / 17.45 ms โ no change โ
โ QQuery 16 โ 7.21 / 8.24 ยฑ0.91 / 9.89 ms โ 7.20 / 7.69 ยฑ0.64 / 8.95 ms โ +1.07x faster โ
โ QQuery 17 โ 225.14 / 228.06 ยฑ2.40 / 231.95 ms โ 225.79 / 228.78 ยฑ1.95 / 230.65 ms โ no change โ
โ QQuery 18 โ 124.80 / 126.77 ยฑ1.32 / 128.72 ms โ 126.13 / 128.50 ยฑ1.39 / 130.17 ms โ no change โ
โ QQuery 19 โ 153.80 / 154.94 ยฑ0.81 / 156.05 ms โ 153.60 / 154.40 ยฑ0.41 / 154.73 ms โ no change โ
โ QQuery 20 โ 13.76 / 14.00 ยฑ0.18 / 14.31 ms โ 13.77 / 14.23 ยฑ0.33 / 14.72 ms โ no change โ
โ QQuery 21 โ 19.42 / 19.81 ยฑ0.35 / 20.31 ms โ 18.81 / 19.37 ยฑ0.51 / 20.29 ms โ no change โ
โ QQuery 22 โ 487.23 / 489.45 ยฑ1.73 / 492.46 ms โ 484.82 / 489.29 ยฑ4.11 / 495.92 ms โ no change โ
โ QQuery 23 โ 861.51 / 864.54 ยฑ4.72 / 873.94 ms โ 854.42 / 866.29 ยฑ10.98 / 881.01 ms โ no change โ
โ QQuery 24 โ 376.56 / 382.61 ยฑ3.98 / 387.63 ms โ 378.96 / 381.70 ยฑ2.16 / 385.46 ms โ no change โ
โ QQuery 25 โ 336.43 / 339.15 ยฑ1.51 / 340.40 ms โ 338.19 / 339.58 ยฑ1.06 / 341.14 ms โ no change โ
โ QQuery 26 โ 79.91 / 81.94 ยฑ1.61 / 84.60 ms โ 81.94 / 83.27 ยฑ0.88 / 84.47 ms โ no change โ
โ QQuery 27 โ 6.74 / 7.08 ยฑ0.31 / 7.63 ms โ 6.83 / 7.03 ยฑ0.16 / 7.22 ms โ no change โ
โ QQuery 28 โ 148.43 / 149.52 ยฑ0.99 / 151.10 ms โ 148.50 / 149.75 ยฑ1.05 / 151.63 ms โ no change โ
โ QQuery 29 โ 277.35 / 281.16 ยฑ1.96 / 282.73 ms โ 279.25 / 280.02 ยฑ0.92 / 281.77 ms โ no change โ
โ QQuery 30 โ 42.29 / 43.96 ยฑ1.21 / 46.05 ms โ 43.51 / 45.09 ยฑ1.18 / 47.07 ms โ no change โ
โ QQuery 31 โ 168.43 / 170.11 ยฑ1.55 / 172.72 ms โ 168.72 / 169.74 ยฑ0.94 / 171.47 ms โ no change โ
โ QQuery 32 โ 55.96 / 56.76 ยฑ0.88 / 58.44 ms โ 57.50 / 66.52 ยฑ15.87 / 98.21 ms โ 1.17x slower โ
โ QQuery 33 โ 139.62 / 141.23 ยฑ1.77 / 143.70 ms โ 140.15 / 142.47 ยฑ1.33 / 144.09 ms โ no change โ
โ QQuery 34 โ 6.78 / 7.03 ยฑ0.24 / 7.47 ms โ 6.80 / 7.09 ยฑ0.25 / 7.52 ms โ no change โ
โ QQuery 35 โ 105.50 / 107.13 ยฑ1.08 / 108.45 ms โ 105.15 / 106.87 ยฑ1.37 / 108.42 ms โ no change โ
โ QQuery 36 โ 6.37 / 6.62 ยฑ0.19 / 6.92 ms โ 6.68 / 6.93 ยฑ0.19 / 7.18 ms โ no change โ
โ QQuery 37 โ 8.60 / 8.87 ยฑ0.26 / 9.32 ms โ 8.72 / 8.97 ยฑ0.22 / 9.36 ms โ no change โ
โ QQuery 38 โ 80.41 / 87.14 ยฑ4.71 / 94.49 ms โ 83.00 / 85.56 ยฑ2.58 / 90.22 ms โ no change โ
โ QQuery 39 โ 122.37 / 125.56 ยฑ1.81 / 127.97 ms โ 121.67 / 123.79 ยฑ2.27 / 127.86 ms โ no change โ
โ QQuery 40 โ 107.20 / 114.48 ยฑ8.16 / 129.99 ms โ 109.29 / 114.07 ยฑ5.58 / 124.73 ms โ no change โ
โ QQuery 41 โ 13.91 / 15.47 ยฑ1.76 / 18.74 ms โ 14.08 / 15.09 ยฑ0.82 / 16.38 ms โ no change โ
โ QQuery 42 โ 106.53 / 107.99 ยฑ0.91 / 109.02 ms โ 108.13 / 109.27 ยฑ1.13 / 110.93 ms โ no change โ
โ QQuery 43 โ 5.81 / 6.04 ยฑ0.16 / 6.30 ms โ 5.96 / 6.17 ยฑ0.18 / 6.46 ms โ no change โ
โ QQuery 44 โ 11.30 / 11.58 ยฑ0.20 / 11.87 ms โ 11.78 / 12.10 ยฑ0.50 / 13.09 ms โ no change โ
โ QQuery 45 โ 49.30 / 50.86 ยฑ1.21 / 52.75 ms โ 50.23 / 51.23 ยฑ0.64 / 51.98 ms โ no change โ
โ QQuery 46 โ 8.15 / 8.66 ยฑ0.45 / 9.46 ms โ 8.24 / 8.48 ยฑ0.21 / 8.84 ms โ no change โ
โ QQuery 47 โ 686.16 / 692.32 ยฑ3.31 / 695.20 ms โ 678.30 / 690.21 ยฑ9.07 / 706.18 ms โ no change โ
โ QQuery 48 โ 286.89 / 291.00 ยฑ4.22 / 296.73 ms โ 280.25 / 289.60 ยฑ5.50 / 296.22 ms โ no change โ
โ QQuery 49 โ 249.45 / 251.53 ยฑ2.28 / 255.97 ms โ 251.48 / 253.19 ยฑ1.36 / 255.02 ms โ no change โ
โ QQuery 50 โ 213.56 / 220.64 ยฑ3.80 / 223.94 ms โ 216.68 / 220.44 ยฑ3.78 / 226.49 ms โ no change โ
โ QQuery 51 โ 180.66 / 182.93 ยฑ1.17 / 183.90 ms โ 178.98 / 181.58 ยฑ1.99 / 184.06 ms โ no change โ
โ QQuery 52 โ 105.37 / 107.48 ยฑ1.76 / 110.30 ms โ 106.70 / 107.91 ยฑ1.31 / 110.38 ms โ no change โ
โ QQuery 53 โ 101.15 / 102.13 ยฑ0.80 / 103.37 ms โ 101.95 / 102.43 ยฑ0.39 / 103.13 ms โ no change โ
โ QQuery 54 โ 145.58 / 146.50 ยฑ0.74 / 147.21 ms โ 146.95 / 147.23 ยฑ0.16 / 147.40 ms โ no change โ
โ QQuery 55 โ 106.33 / 108.56 ยฑ1.49 / 110.17 ms โ 105.94 / 107.89 ยฑ1.26 / 109.77 ms โ no change โ
โ QQuery 56 โ 139.67 / 140.70 ยฑ1.31 / 143.23 ms โ 139.53 / 141.04 ยฑ1.12 / 142.76 ms โ no change โ
โ QQuery 57 โ 171.70 / 173.75 ยฑ1.25 / 174.85 ms โ 171.93 / 175.30 ยฑ2.25 / 178.49 ms โ no change โ
โ QQuery 58 โ 285.19 / 293.59 ยฑ5.66 / 300.89 ms โ 286.47 / 298.90 ยฑ9.37 / 313.91 ms โ no change โ
โ QQuery 59 โ 196.33 / 198.66 ยฑ2.31 / 202.17 ms โ 198.27 / 199.35 ยฑ0.73 / 200.31 ms โ no change โ
โ QQuery 60 โ 141.41 / 143.97 ยฑ2.14 / 147.37 ms โ 142.17 / 144.29 ยฑ1.45 / 146.31 ms โ no change โ
โ QQuery 61 โ 13.02 / 13.15 ยฑ0.17 / 13.47 ms โ 12.87 / 13.28 ยฑ0.58 / 14.37 ms โ no change โ
โ QQuery 62 โ 872.33 / 911.86 ยฑ26.02 / 950.08 ms โ 881.69 / 902.84 ยฑ15.31 / 925.62 ms โ no change โ
โ QQuery 63 โ 102.63 / 105.33 ยฑ2.68 / 109.99 ms โ 104.62 / 105.41 ยฑ0.66 / 106.20 ms โ no change โ
โ QQuery 64 โ 675.55 / 681.81 ยฑ4.48 / 688.05 ms โ 677.91 / 681.33 ยฑ2.53 / 685.52 ms โ no change โ
โ QQuery 65 โ 249.31 / 253.83 ยฑ3.86 / 260.44 ms โ 245.61 / 251.40 ยฑ3.23 / 255.25 ms โ no change โ
โ QQuery 66 โ 254.89 / 260.11 ยฑ3.75 / 264.79 ms โ 243.14 / 251.65 ยฑ4.93 / 258.25 ms โ no change โ
โ QQuery 67 โ 307.84 / 318.04 ยฑ5.82 / 325.90 ms โ 302.17 / 312.38 ยฑ9.35 / 325.98 ms โ no change โ
โ QQuery 68 โ 9.16 / 9.65 ยฑ0.50 / 10.30 ms โ 8.61 / 10.81 ยฑ1.42 / 12.90 ms โ 1.12x slower โ
โ QQuery 69 โ 102.91 / 107.18 ยฑ3.67 / 113.47 ms โ 101.44 / 104.54 ยฑ2.13 / 108.10 ms โ no change โ
โ QQuery 70 โ 335.99 / 349.01 ยฑ16.20 / 380.32 ms โ 325.33 / 343.82 ยฑ13.66 / 367.77 ms โ no change โ
โ QQuery 71 โ 133.18 / 135.94 ยฑ2.95 / 141.48 ms โ 134.28 / 136.10 ยฑ1.24 / 138.12 ms โ no change โ
โ QQuery 72 โ 630.36 / 636.56 ยฑ4.98 / 644.30 ms โ 612.94 / 615.56 ยฑ2.60 / 620.38 ms โ no change โ
โ QQuery 73 โ 7.37 / 8.52 ยฑ1.10 / 10.29 ms โ 6.45 / 7.82 ยฑ0.77 / 8.80 ms โ +1.09x faster โ
โ QQuery 74 โ 631.76 / 644.16 ยฑ9.41 / 660.87 ms โ 550.65 / 556.39 ยฑ4.89 / 563.11 ms โ +1.16x faster โ
โ QQuery 75 โ 277.06 / 279.64 ยฑ1.63 / 281.45 ms โ 272.13 / 276.41 ยฑ2.91 / 280.72 ms โ no change โ
โ QQuery 76 โ 133.02 / 134.92 ยฑ1.02 / 135.80 ms โ 130.84 / 132.46 ยฑ1.35 / 134.91 ms โ no change โ
โ QQuery 77 โ 187.27 / 190.11 ยฑ1.65 / 192.42 ms โ 187.47 / 188.76 ยฑ1.05 / 190.55 ms โ no change โ
โ QQuery 78 โ 343.57 / 349.24 ยฑ4.02 / 353.23 ms โ 339.50 / 341.25 ยฑ2.14 / 345.34 ms โ no change โ
โ QQuery 79 โ 235.23 / 240.01 ยฑ3.62 / 244.62 ms โ 233.66 / 235.74 ยฑ2.08 / 239.34 ms โ no change โ
โ QQuery 80 โ 319.76 / 324.22 ยฑ2.64 / 327.62 ms โ 323.63 / 325.68 ยฑ2.30 / 329.79 ms โ no change โ
โ QQuery 81 โ 26.68 / 27.22 ยฑ0.67 / 28.53 ms โ 26.39 / 26.91 ยฑ0.72 / 28.33 ms โ no change โ
โ QQuery 82 โ 198.28 / 200.16 ยฑ1.40 / 202.01 ms โ 200.83 / 202.30 ยฑ1.50 / 204.92 ms โ no change โ
โ QQuery 83 โ 38.52 / 39.47 ยฑ1.00 / 41.19 ms โ 38.55 / 39.08 ยฑ0.34 / 39.59 ms โ no change โ
โ QQuery 84 โ 48.56 / 49.02 ยฑ0.42 / 49.63 ms โ 48.26 / 49.43 ยฑ0.67 / 50.19 ms โ no change โ
โ QQuery 85 โ 147.49 / 148.78 ยฑ0.98 / 150.46 ms โ 149.86 / 151.30 ยฑ0.93 / 152.25 ms โ no change โ
โ QQuery 86 โ 39.29 / 39.67 ยฑ0.25 / 40.02 ms โ 39.21 / 40.26 ยฑ0.68 / 41.28 ms โ no change โ
โ QQuery 87 โ 86.59 / 88.94 ยฑ2.30 / 93.20 ms โ 84.79 / 89.09 ยฑ3.69 / 95.72 ms โ no change โ
โ QQuery 88 โ 100.18 / 101.95 ยฑ2.09 / 105.87 ms โ 100.51 / 101.38 ยฑ0.93 / 103.15 ms โ no change โ
โ QQuery 89 โ 118.04 / 119.67 ยฑ0.98 / 120.68 ms โ 119.64 / 121.60 ยฑ1.03 / 122.57 ms โ no change โ
โ QQuery 90 โ 22.70 / 23.55 ยฑ0.58 / 24.42 ms โ 23.25 / 23.98 ยฑ0.56 / 24.64 ms โ no change โ
โ QQuery 91 โ 63.34 / 64.31 ยฑ0.96 / 65.91 ms โ 64.60 / 65.84 ยฑ1.32 / 68.24 ms โ no change โ
โ QQuery 92 โ 56.96 / 57.73 ยฑ0.78 / 59.09 ms โ 57.18 / 57.97 ยฑ0.47 / 58.56 ms โ no change โ
โ QQuery 93 โ 187.21 / 188.28 ยฑ0.90 / 189.21 ms โ 186.48 / 188.34 ยฑ1.13 / 189.99 ms โ no change โ
โ QQuery 94 โ 61.28 / 62.08 ยฑ0.59 / 62.90 ms โ 61.73 / 63.15 ยฑ1.34 / 65.45 ms โ no change โ
โ QQuery 95 โ 127.19 / 128.09 ยฑ0.64 / 128.72 ms โ 128.12 / 129.66 ยฑ1.26 / 131.87 ms โ no change โ
โ QQuery 96 โ 71.95 / 73.28 ยฑ0.90 / 74.78 ms โ 72.07 / 73.96 ยฑ1.60 / 76.23 ms โ no change โ
โ QQuery 97 โ 123.10 / 126.66 ยฑ2.16 / 128.93 ms โ 123.46 / 125.84 ยฑ1.97 / 128.64 ms โ no change โ
โ QQuery 98 โ 153.37 / 157.15 ยฑ2.09 / 159.46 ms โ 151.13 / 154.19 ยฑ1.76 / 156.16 ms โ no change โ
โ QQuery 99 โ 10721.88 / 10753.62 ยฑ32.23 / 10798.99 ms โ 10704.70 / 10717.02 ยฑ13.96 / 10737.34 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 31324.36ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 31188.19ms โ
โ Average Time (HEAD) โ 316.41ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 315.03ms โ
โ Queries Faster โ 4 โ
โ Queries Slower โ 2 โ
โ Queries with No Change โ 93 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 157.0s |
| Peak memory | 5.6 GiB |
| Avg memory | 4.5 GiB |
| CPU user | 259.4s |
| CPU sys | 16.6s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 156.2s |
| Peak memory | 5.7 GiB |
| Avg memory | 4.7 GiB |
| CPU user | 257.9s |
| CPU sys | 16.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3080858573
##########
datafusion/functions-aggregate/src/count.rs:
##########
@@ -336,18 +337,70 @@ impl AggregateUDFImpl for Count {
}
fn groups_accumulator_supported(&self, args: AccumulatorArgs) -> bool {
- // groups accumulator only supports `COUNT(c1)`, not
- // `COUNT(c1, c2)`, etc
- if args.is_distinct {
+ if args.exprs.len() != 1 {
return false;
}
- args.exprs.len() == 1
+ if args.is_distinct {
+ // Only support primitive integer types for now
+ matches!(
+ args.expr_fields[0].data_type(),
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::UInt8
+ | DataType::UInt16
+ | DataType::UInt32
+ | DataType::UInt64
+ )
+ } else {
+ true
+ }
}
fn create_groups_accumulator(
&self,
- _args: AccumulatorArgs,
+ args: AccumulatorArgs,
) -> Result<Box<dyn GroupsAccumulator>> {
+ if args.is_distinct {
+ let data_type = args.expr_fields[0].data_type();
+ return match data_type {
+ DataType::Int8 => Ok(Box::new(
Review Comment:
Yup ! thats the plan once we fix on the right group accumulator strategy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230789975
๐ค Criterion benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4230786454)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4230786454-1101-4lf7q 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (f0b2a4aab055f557acc50f4e919d92ab57f18c35) to 16e578d (merge-base) [diff](https://github.com/apache/datafusion/compare/16e578dfdd28c8bffc6a21f54164b889e79bf75c..f0b2a4aab055f557acc50f4e919d92ab57f18c35)
BENCH_NAME=clickbemch_extended
BENCH_COMMAND=cargo bench --features=parquet --bench clickbemch_extended
BENCH_FILTER=
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230790092
Benchmark for [this request](https://github.com/apache/datafusion/pull/21561#issuecomment-4230786454) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
struct_query_sql
substr
substr_index
substring
sum
to_char
to_hex
to_local_time
to_time
to_timestamp
topk_aggregate
topk_repartition
translate
trim
trunc
unhex
upper
uuid
window_query_sql
with_hashes
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3068996348
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, BooleanArray, Int64Array, ListBuilder, PrimitiveBuilder,
+};
+use arrow::datatypes::ArrowPrimitiveType;
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ /// Count distinct per group.
+ values: Vec<HashSet<T::Native, RandomState>>,
Review Comment:
Local testing did yield mixed results :| so far
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4231155752
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4231151449)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4231151449-1103-pfrtl 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (ee0c8650aacfb92d852062a1c0bcfb5465c93482) to 16e578d (merge-base) [diff](https://github.com/apache/datafusion/compare/16e578dfdd28c8bffc6a21f54164b889e79bf75c..ee0c8650aacfb92d852062a1c0bcfb5465c93482) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4231151449
run benchmark tpch tpcds clickbench_partitioned clickbench_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4232538300
Comparison with main
```
group impl_groups main
----- ----------- ----
count_distinct_groups i64 10 groups 1.00 85.6ยฑ1.80ยตs ? ?/sec 4.97 425.0ยฑ5.72ยตs ? ?/sec
count_distinct_groups i64 10 groups (GroupsAccumulator) 1.00 82.6ยฑ0.83ยตs ? ?/sec
count_distinct_groups i64 100 groups 1.00 86.1ยฑ1.41ยตs ? ?/sec 5.18 445.9ยฑ5.79ยตs ? ?/sec
count_distinct_groups i64 100 groups (GroupsAccumulator) 1.00 83.1ยฑ1.25ยตs ? ?/sec
count_distinct_groups i64 1000 groups 1.00 86.7ยฑ2.55ยตs ? ?/sec 7.06 612.7ยฑ4.72ยตs ? ?/sec
count_distinct_groups i64 1000 groups (GroupsAccumulator) 1.00 84.1ยฑ2.01ยตs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3080858573
##########
datafusion/functions-aggregate/src/count.rs:
##########
@@ -336,18 +337,70 @@ impl AggregateUDFImpl for Count {
}
fn groups_accumulator_supported(&self, args: AccumulatorArgs) -> bool {
- // groups accumulator only supports `COUNT(c1)`, not
- // `COUNT(c1, c2)`, etc
- if args.is_distinct {
+ if args.exprs.len() != 1 {
return false;
}
- args.exprs.len() == 1
+ if args.is_distinct {
+ // Only support primitive integer types for now
+ matches!(
+ args.expr_fields[0].data_type(),
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::UInt8
+ | DataType::UInt16
+ | DataType::UInt32
+ | DataType::UInt64
+ )
+ } else {
+ true
+ }
}
fn create_groups_accumulator(
&self,
- _args: AccumulatorArgs,
+ args: AccumulatorArgs,
) -> Result<Box<dyn GroupsAccumulator>> {
+ if args.is_distinct {
+ let data_type = args.expr_fields[0].data_type();
+ return match data_type {
+ DataType::Int8 => Ok(Box::new(
Review Comment:
If you meant using bitmaps for smaller data types , yes I absolutely plan to do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4242530837
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4242372936)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.18 / 4.38 ยฑ6.31 / 16.99 ms โ 1.19 / 4.46 ยฑ6.41 / 17.28 ms โ no change โ
โ QQuery 1 โ 14.34 / 14.48 ยฑ0.10 / 14.64 ms โ 14.33 / 14.71 ยฑ0.20 / 14.87 ms โ no change โ
โ QQuery 2 โ 43.74 / 44.20 ยฑ0.39 / 44.71 ms โ 44.26 / 44.52 ยฑ0.32 / 45.12 ms โ no change โ
โ QQuery 3 โ 42.46 / 45.44 ยฑ2.06 / 47.60 ms โ 42.82 / 45.34 ยฑ2.69 / 49.06 ms โ no change โ
โ QQuery 4 โ 279.55 / 291.46 ยฑ6.44 / 297.08 ms โ 294.87 / 320.18 ยฑ22.85 / 362.99 ms โ 1.10x slower โ
โ QQuery 5 โ 337.75 / 341.38 ยฑ2.03 / 343.21 ms โ 344.49 / 349.08 ยฑ4.27 / 354.63 ms โ no change โ
โ QQuery 6 โ 5.76 / 6.39 ยฑ0.49 / 7.17 ms โ 5.57 / 7.21 ยฑ0.94 / 8.10 ms โ 1.13x slower โ
โ QQuery 7 โ 16.73 / 17.39 ยฑ0.76 / 18.62 ms โ 16.54 / 17.77 ยฑ1.14 / 19.60 ms โ no change โ
โ QQuery 8 โ 402.49 / 418.57 ยฑ11.18 / 435.38 ms โ 420.19 / 432.19 ยฑ10.52 / 451.82 ms โ no change โ
โ QQuery 9 โ 632.90 / 640.35 ยฑ6.06 / 647.26 ms โ 584.09 / 592.25 ยฑ6.87 / 604.68 ms โ +1.08x faster โ
โ QQuery 10 โ 90.29 / 92.87 ยฑ2.78 / 98.18 ms โ 92.52 / 93.37 ยฑ0.93 / 95.08 ms โ no change โ
โ QQuery 11 โ 103.35 / 104.78 ยฑ1.39 / 106.58 ms โ 103.98 / 105.48 ยฑ1.45 / 108.24 ms โ no change โ
โ QQuery 12 โ 336.24 / 343.62 ยฑ8.76 / 359.66 ms โ 339.74 / 344.08 ยฑ2.90 / 347.79 ms โ no change โ
โ QQuery 13 โ 445.61 / 465.51 ยฑ15.36 / 488.12 ms โ 450.96 / 463.18 ยฑ8.53 / 473.76 ms โ no change โ
โ QQuery 14 โ 342.31 / 344.56 ยฑ1.17 / 345.68 ms โ 342.12 / 345.40 ยฑ2.63 / 347.99 ms โ no change โ
โ QQuery 15 โ 345.56 / 357.32 ยฑ11.42 / 376.69 ms โ 349.83 / 370.52 ยฑ24.97 / 416.63 ms โ no change โ
โ QQuery 16 โ 699.87 / 714.57 ยฑ17.36 / 745.45 ms โ 714.58 / 723.97 ยฑ11.49 / 744.59 ms โ no change โ
โ QQuery 17 โ 699.87 / 706.89 ยฑ6.59 / 718.18 ms โ 698.97 / 710.24 ยฑ7.20 / 720.96 ms โ no change โ
โ QQuery 18 โ 1381.51 / 1460.28 ยฑ41.90 / 1493.90 ms โ 1386.36 / 1471.80 ยฑ48.78 / 1537.72 ms โ no change โ
โ QQuery 19 โ 36.55 / 44.80 ยฑ9.70 / 62.91 ms โ 34.28 / 40.48 ยฑ8.96 / 58.22 ms โ +1.11x faster โ
โ QQuery 20 โ 721.76 / 739.27 ยฑ17.82 / 764.05 ms โ 711.27 / 728.36 ยฑ15.54 / 756.12 ms โ no change โ
โ QQuery 21 โ 768.33 / 769.62 ยฑ0.78 / 770.76 ms โ 760.34 / 766.84 ยฑ5.28 / 774.60 ms โ no change โ
โ QQuery 22 โ 1144.77 / 1152.55 ยฑ7.38 / 1166.57 ms โ 1122.25 / 1125.48 ยฑ1.66 / 1126.63 ms โ no change โ
โ QQuery 23 โ 3074.81 / 3097.62 ยฑ15.86 / 3120.12 ms โ 3065.64 / 3100.90 ยฑ20.65 / 3123.14 ms โ no change โ
โ QQuery 24 โ 99.61 / 101.89 ยฑ1.92 / 105.22 ms โ 100.98 / 102.94 ยฑ1.99 / 106.59 ms โ no change โ
โ QQuery 25 โ 138.52 / 141.26 ยฑ1.86 / 144.00 ms โ 138.34 / 141.34 ยฑ2.85 / 146.01 ms โ no change โ
โ QQuery 26 โ 99.38 / 101.04 ยฑ1.60 / 103.29 ms โ 99.00 / 101.80 ยฑ1.71 / 103.85 ms โ no change โ
โ QQuery 27 โ 851.19 / 854.05 ยฑ3.88 / 861.73 ms โ 846.19 / 850.30 ยฑ4.39 / 857.64 ms โ no change โ
โ QQuery 28 โ 3238.21 / 3251.78 ยฑ8.89 / 3263.54 ms โ 3256.24 / 3284.06 ยฑ17.74 / 3311.06 ms โ no change โ
โ QQuery 29 โ 49.49 / 53.72 ยฑ2.81 / 56.73 ms โ 50.56 / 53.87 ยฑ2.11 / 56.27 ms โ no change โ
โ QQuery 30 โ 358.11 / 360.54 ยฑ4.22 / 368.94 ms โ 358.58 / 365.18 ยฑ6.67 / 373.51 ms โ no change โ
โ QQuery 31 โ 358.61 / 377.63 ยฑ12.16 / 397.02 ms โ 368.64 / 378.41 ยฑ6.20 / 387.77 ms โ no change โ
โ QQuery 32 โ 1178.74 / 1268.49 ยฑ46.60 / 1307.83 ms โ 1209.99 / 1244.88 ยฑ38.12 / 1318.88 ms โ no change โ
โ QQuery 33 โ 1512.90 / 1557.46 ยฑ38.54 / 1618.73 ms โ 1440.32 / 1478.64 ยฑ49.54 / 1571.59 ms โ +1.05x faster โ
โ QQuery 34 โ 1447.28 / 1513.00 ยฑ71.63 / 1647.04 ms โ 1426.61 / 1446.56 ยฑ13.49 / 1458.41 ms โ no change โ
โ QQuery 35 โ 390.67 / 450.12 ยฑ82.49 / 612.64 ms โ 372.25 / 381.55 ยฑ6.61 / 391.99 ms โ +1.18x faster โ
โ QQuery 36 โ 116.90 / 120.27 ยฑ3.34 / 126.04 ms โ 109.97 / 117.62 ยฑ4.13 / 121.93 ms โ no change โ
โ QQuery 37 โ 48.48 / 50.23 ยฑ1.69 / 53.42 ms โ 47.57 / 49.11 ยฑ1.89 / 52.72 ms โ no change โ
โ QQuery 38 โ 75.85 / 87.22 ยฑ9.84 / 105.42 ms โ 73.20 / 75.88 ยฑ1.40 / 77.26 ms โ +1.15x faster โ
โ QQuery 39 โ 212.77 / 222.31 ยฑ9.20 / 238.67 ms โ 213.07 / 218.67 ยฑ4.74 / 225.27 ms โ no change โ
โ QQuery 40 โ 21.57 / 23.90 ยฑ1.26 / 25.02 ms โ 22.24 / 25.46 ยฑ1.75 / 27.09 ms โ 1.07x slower โ
โ QQuery 41 โ 20.77 / 21.87 ยฑ1.22 / 24.17 ms โ 18.48 / 20.40 ยฑ1.52 / 22.95 ms โ +1.07x faster โ
โ QQuery 42 โ 19.65 / 20.32 ยฑ0.39 / 20.79 ms โ 18.92 / 19.91 ยฑ0.57 / 20.65 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 22795.38ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 22574.38ms โ
โ Average Time (HEAD) โ 530.13ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 524.99ms โ
โ Queries Faster โ 6 โ
โ Queries Slower โ 3 โ
โ Queries with No Change โ 34 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 114.6s |
| Peak memory | 40.4 GiB |
| Avg memory | 29.1 GiB |
| CPU user | 1063.5s |
| CPU sys | 103.7s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 114.0s |
| Peak memory | 41.0 GiB |
| Avg memory | 28.8 GiB |
| CPU user | 1058.8s |
| CPU sys | 97.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4256816829
I am trying an approach with storing counts per group along with hashtable to see if we could speedup `evaluate()` method's complexity to O(1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4251569584
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4251531266)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4251531266-1284-sgljj 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (28c607aad4745f0b021b60bf42e1f3b7d49c0971) to 2818abb (merge-base) [diff](https://github.com/apache/datafusion/compare/2818abb8a10bf8b001669c15cf597f9ddfb86ed1..28c607aad4745f0b021b60bf42e1f3b7d49c0971) using: clickbench_extended
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266107489
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266210954
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266107489)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 6.57 / 7.10 ยฑ0.83 / 8.75 ms โ 6.73 / 7.23 ยฑ0.70 / 8.59 ms โ no change โ
โ QQuery 2 โ 144.18 / 145.48 ยฑ1.23 / 147.08 ms โ 145.02 / 146.93 ยฑ1.71 / 149.37 ms โ no change โ
โ QQuery 3 โ 114.04 / 114.46 ยฑ0.34 / 115.07 ms โ 113.93 / 114.99 ยฑ0.86 / 116.41 ms โ no change โ
โ QQuery 4 โ 1382.62 / 1406.29 ยฑ18.24 / 1425.85 ms โ 1394.88 / 1405.16 ยฑ9.84 / 1420.28 ms โ no change โ
โ QQuery 5 โ 175.03 / 176.62 ยฑ1.79 / 180.07 ms โ 172.78 / 174.97 ยฑ1.69 / 177.18 ms โ no change โ
โ QQuery 6 โ 836.49 / 866.96 ยฑ15.95 / 881.18 ms โ 821.57 / 871.57 ยฑ34.10 / 921.41 ms โ no change โ
โ QQuery 7 โ 342.91 / 345.41 ยฑ2.22 / 349.41 ms โ 343.74 / 347.24 ยฑ1.77 / 348.45 ms โ no change โ
โ QQuery 8 โ 115.83 / 118.93 ยฑ2.01 / 121.05 ms โ 114.60 / 118.32 ยฑ2.15 / 120.74 ms โ no change โ
โ QQuery 9 โ 102.84 / 105.81 ยฑ2.08 / 108.06 ms โ 101.91 / 106.11 ยฑ2.90 / 109.07 ms โ no change โ
โ QQuery 10 โ 107.25 / 108.82 ยฑ0.81 / 109.53 ms โ 106.96 / 107.78 ยฑ0.57 / 108.47 ms โ no change โ
โ QQuery 11 โ 974.65 / 1003.99 ยฑ21.10 / 1025.26 ms โ 982.28 / 1000.57 ยฑ21.19 / 1039.68 ms โ no change โ
โ QQuery 12 โ 46.06 / 47.46 ยฑ0.91 / 48.62 ms โ 45.54 / 47.60 ยฑ1.23 / 49.32 ms โ no change โ
โ QQuery 13 โ 406.79 / 410.92 ยฑ2.59 / 414.27 ms โ 404.65 / 407.31 ยฑ1.89 / 409.49 ms โ no change โ
โ QQuery 14 โ 1005.06 / 1016.91 ยฑ10.87 / 1034.30 ms โ 997.24 / 1012.90 ยฑ8.93 / 1022.88 ms โ no change โ
โ QQuery 15 โ 15.90 / 16.79 ยฑ0.57 / 17.41 ms โ 15.46 / 16.92 ยฑ1.25 / 19.19 ms โ no change โ
โ QQuery 16 โ 7.37 / 7.79 ยฑ0.51 / 8.77 ms โ 7.14 / 7.97 ยฑ0.72 / 9.22 ms โ no change โ
โ QQuery 17 โ 230.56 / 232.80 ยฑ2.02 / 236.34 ms โ 230.35 / 232.16 ยฑ1.21 / 234.16 ms โ no change โ
โ QQuery 18 โ 129.95 / 130.37 ยฑ0.34 / 130.93 ms โ 127.42 / 129.12 ยฑ1.28 / 131.13 ms โ no change โ
โ QQuery 19 โ 157.64 / 159.45 ยฑ1.09 / 160.54 ms โ 157.89 / 160.14 ยฑ1.41 / 162.00 ms โ no change โ
โ QQuery 20 โ 13.77 / 15.00 ยฑ0.66 / 15.72 ms โ 13.98 / 14.62 ยฑ0.58 / 15.61 ms โ no change โ
โ QQuery 21 โ 19.59 / 19.94 ยฑ0.19 / 20.15 ms โ 19.23 / 19.88 ยฑ0.75 / 21.28 ms โ no change โ
โ QQuery 22 โ 489.49 / 497.47 ยฑ4.65 / 502.21 ms โ 493.10 / 498.16 ยฑ3.11 / 502.00 ms โ no change โ
โ QQuery 23 โ 898.65 / 912.61 ยฑ9.99 / 926.20 ms โ 888.18 / 898.78 ยฑ7.03 / 906.88 ms โ no change โ
โ QQuery 24 โ 388.18 / 391.86 ยฑ3.03 / 397.32 ms โ 387.06 / 390.00 ยฑ3.96 / 397.69 ms โ no change โ
โ QQuery 25 โ 342.86 / 346.90 ยฑ2.46 / 350.25 ms โ 346.95 / 348.93 ยฑ1.89 / 352.41 ms โ no change โ
โ QQuery 26 โ 80.69 / 82.35 ยฑ0.98 / 83.45 ms โ 82.55 / 84.62 ยฑ1.27 / 86.23 ms โ no change โ
โ QQuery 27 โ 7.14 / 8.15 ยฑ0.97 / 9.71 ms โ 7.22 / 7.63 ยฑ0.70 / 9.03 ms โ +1.07x faster โ
โ QQuery 28 โ 150.19 / 151.54 ยฑ1.36 / 153.96 ms โ 150.41 / 151.68 ยฑ0.92 / 153.21 ms โ no change โ
โ QQuery 29 โ 286.05 / 288.87 ยฑ3.59 / 295.90 ms โ 281.60 / 285.51 ยฑ2.63 / 288.76 ms โ no change โ
โ QQuery 30 โ 43.67 / 45.67 ยฑ1.21 / 47.36 ms โ 43.33 / 45.79 ยฑ1.80 / 47.83 ms โ no change โ
โ QQuery 31 โ 171.46 / 173.21 ยฑ2.51 / 178.14 ms โ 173.83 / 175.75 ยฑ1.80 / 178.35 ms โ no change โ
โ QQuery 32 โ 57.36 / 57.63 ยฑ0.28 / 58.09 ms โ 56.87 / 59.20 ยฑ1.44 / 61.06 ms โ no change โ
โ QQuery 33 โ 140.15 / 142.08 ยฑ1.15 / 143.34 ms โ 143.40 / 144.46 ยฑ0.59 / 145.22 ms โ no change โ
โ QQuery 34 โ 7.06 / 7.27 ยฑ0.29 / 7.83 ms โ 6.88 / 7.13 ยฑ0.27 / 7.65 ms โ no change โ
โ QQuery 35 โ 108.20 / 109.88 ยฑ0.87 / 110.68 ms โ 108.51 / 109.54 ยฑ0.88 / 111.01 ms โ no change โ
โ QQuery 36 โ 6.81 / 7.21 ยฑ0.34 / 7.78 ms โ 6.48 / 6.94 ยฑ0.27 / 7.25 ms โ no change โ
โ QQuery 37 โ 8.36 / 8.84 ยฑ0.26 / 9.14 ms โ 8.77 / 9.18 ยฑ0.40 / 9.68 ms โ no change โ
โ QQuery 38 โ 84.57 / 89.86 ยฑ3.93 / 95.95 ms โ 85.82 / 88.29 ยฑ2.99 / 94.03 ms โ no change โ
โ QQuery 39 โ 129.36 / 130.81 ยฑ0.95 / 132.22 ms โ 126.50 / 129.88 ยฑ2.44 / 133.92 ms โ no change โ
โ QQuery 40 โ 111.21 / 117.51 ยฑ6.25 / 129.14 ms โ 112.69 / 116.79 ยฑ5.30 / 127.17 ms โ no change โ
โ QQuery 41 โ 15.01 / 15.95 ยฑ0.84 / 17.34 ms โ 14.61 / 15.28 ยฑ0.35 / 15.63 ms โ no change โ
โ QQuery 42 โ 109.19 / 111.06 ยฑ0.98 / 112.06 ms โ 109.33 / 110.22 ยฑ0.64 / 111.31 ms โ no change โ
โ QQuery 43 โ 5.93 / 6.37 ยฑ0.68 / 7.73 ms โ 5.89 / 6.15 ยฑ0.18 / 6.46 ms โ no change โ
โ QQuery 44 โ 11.94 / 12.10 ยฑ0.10 / 12.24 ms โ 12.09 / 12.58 ยฑ0.51 / 13.30 ms โ no change โ
โ QQuery 45 โ 51.70 / 52.44 ยฑ0.60 / 53.07 ms โ 51.51 / 52.30 ยฑ0.77 / 53.62 ms โ no change โ
โ QQuery 46 โ 8.73 / 8.95 ยฑ0.19 / 9.25 ms โ 8.34 / 8.75 ยฑ0.24 / 9.06 ms โ no change โ
โ QQuery 47 โ 738.62 / 746.81 ยฑ6.48 / 754.45 ms โ 745.50 / 759.14 ยฑ7.50 / 765.66 ms โ no change โ
โ QQuery 48 โ 284.86 / 292.65 ยฑ4.14 / 296.18 ms โ 296.00 / 298.98 ยฑ2.24 / 301.68 ms โ no change โ
โ QQuery 49 โ 253.19 / 255.42 ยฑ1.45 / 257.52 ms โ 252.98 / 255.88 ยฑ2.02 / 258.32 ms โ no change โ
โ QQuery 50 โ 222.86 / 228.25 ยฑ3.66 / 232.42 ms โ 224.12 / 231.38 ยฑ5.58 / 238.74 ms โ no change โ
โ QQuery 51 โ 180.09 / 182.02 ยฑ2.18 / 186.04 ms โ 182.60 / 183.18 ยฑ0.65 / 184.39 ms โ no change โ
โ QQuery 52 โ 108.08 / 110.25 ยฑ1.62 / 112.75 ms โ 108.98 / 110.47 ยฑ1.26 / 112.57 ms โ no change โ
โ QQuery 53 โ 103.68 / 105.19 ยฑ1.11 / 106.43 ms โ 103.92 / 104.41 ยฑ0.33 / 104.87 ms โ no change โ
โ QQuery 54 โ 146.36 / 149.83 ยฑ2.21 / 152.84 ms โ 147.71 / 150.71 ยฑ1.88 / 153.45 ms โ no change โ
โ QQuery 55 โ 107.18 / 109.45 ยฑ1.45 / 111.64 ms โ 107.12 / 108.95 ยฑ0.94 / 109.84 ms โ no change โ
โ QQuery 56 โ 140.76 / 142.74 ยฑ1.29 / 144.44 ms โ 144.66 / 146.32 ยฑ1.25 / 148.04 ms โ no change โ
โ QQuery 57 โ 173.87 / 175.92 ยฑ1.27 / 177.74 ms โ 173.98 / 177.64 ยฑ2.66 / 181.90 ms โ no change โ
โ QQuery 58 โ 290.19 / 302.03 ยฑ9.56 / 317.91 ms โ 292.60 / 300.11 ยฑ4.83 / 305.59 ms โ no change โ
โ QQuery 59 โ 197.78 / 199.96 ยฑ1.69 / 202.37 ms โ 200.14 / 203.80 ยฑ3.26 / 209.09 ms โ no change โ
โ QQuery 60 โ 145.77 / 146.69 ยฑ0.87 / 148.21 ms โ 144.13 / 146.77 ยฑ1.77 / 149.43 ms โ no change โ
โ QQuery 61 โ 13.38 / 13.61 ยฑ0.15 / 13.77 ms โ 13.27 / 13.58 ยฑ0.41 / 14.36 ms โ no change โ
โ QQuery 62 โ 903.65 / 944.32 ยฑ30.90 / 989.16 ms โ 893.54 / 941.78 ยฑ38.32 / 985.59 ms โ no change โ
โ QQuery 63 โ 103.97 / 106.53 ยฑ1.91 / 109.54 ms โ 107.33 / 109.26 ยฑ2.82 / 114.82 ms โ no change โ
โ QQuery 64 โ 689.58 / 696.34 ยฑ6.26 / 707.21 ms โ 697.14 / 701.29 ยฑ2.86 / 704.38 ms โ no change โ
โ QQuery 65 โ 255.86 / 259.63 ยฑ3.94 / 267.21 ms โ 261.35 / 263.61 ยฑ1.83 / 266.08 ms โ no change โ
โ QQuery 66 โ 243.31 / 257.02 ยฑ10.89 / 272.71 ms โ 253.16 / 267.83 ยฑ10.65 / 285.55 ms โ no change โ
โ QQuery 67 โ 318.29 / 327.14 ยฑ6.08 / 335.86 ms โ 318.86 / 328.30 ยฑ6.76 / 339.35 ms โ no change โ
โ QQuery 68 โ 9.31 / 11.02 ยฑ1.40 / 13.20 ms โ 9.18 / 9.83 ยฑ0.51 / 10.36 ms โ +1.12x faster โ
โ QQuery 69 โ 103.98 / 105.37 ยฑ0.86 / 106.44 ms โ 102.57 / 105.34 ยฑ1.45 / 106.52 ms โ no change โ
โ QQuery 70 โ 349.23 / 361.34 ยฑ12.50 / 383.43 ms โ 344.18 / 365.65 ยฑ12.60 / 379.40 ms โ no change โ
โ QQuery 71 โ 133.74 / 136.44 ยฑ1.86 / 139.04 ms โ 136.31 / 139.98 ยฑ3.15 / 144.02 ms โ no change โ
โ QQuery 72 โ 635.03 / 641.28 ยฑ6.93 / 650.70 ms โ 621.69 / 631.48 ยฑ5.26 / 636.90 ms โ no change โ
โ QQuery 73 โ 6.93 / 7.54 ยฑ0.52 / 8.38 ms โ 7.46 / 8.84 ยฑ0.94 / 9.84 ms โ 1.17x slower โ
โ QQuery 74 โ 623.69 / 632.58 ยฑ5.67 / 641.00 ms โ 627.02 / 647.58 ยฑ13.00 / 665.08 ms โ no change โ
โ QQuery 75 โ 277.95 / 279.34 ยฑ1.43 / 281.23 ms โ 277.86 / 281.60 ยฑ2.08 / 284.27 ms โ no change โ
โ QQuery 76 โ 133.21 / 135.35 ยฑ2.00 / 138.18 ms โ 134.81 / 137.12 ยฑ1.78 / 138.97 ms โ no change โ
โ QQuery 77 โ 188.61 / 190.53 ยฑ1.79 / 193.70 ms โ 190.08 / 192.28 ยฑ1.37 / 194.33 ms โ no change โ
โ QQuery 78 โ 340.35 / 345.88 ยฑ4.67 / 354.19 ms โ 350.27 / 352.25 ยฑ1.61 / 354.93 ms โ no change โ
โ QQuery 79 โ 235.05 / 242.07 ยฑ5.23 / 249.92 ms โ 246.28 / 249.34 ยฑ4.01 / 257.26 ms โ no change โ
โ QQuery 80 โ 323.66 / 326.69 ยฑ2.76 / 330.17 ms โ 323.72 / 326.03 ยฑ2.16 / 329.95 ms โ no change โ
โ QQuery 81 โ 27.47 / 27.89 ยฑ0.36 / 28.52 ms โ 26.64 / 27.56 ยฑ0.73 / 28.60 ms โ no change โ
โ QQuery 82 โ 201.96 / 204.37 ยฑ1.76 / 206.69 ms โ 201.01 / 204.32 ยฑ2.05 / 207.00 ms โ no change โ
โ QQuery 83 โ 38.83 / 41.53 ยฑ2.88 / 46.45 ms โ 39.27 / 41.43 ยฑ1.50 / 43.86 ms โ no change โ
โ QQuery 84 โ 48.88 / 50.04 ยฑ0.83 / 51.41 ms โ 49.95 / 51.46 ยฑ1.33 / 53.18 ms โ no change โ
โ QQuery 85 โ 148.32 / 149.46 ยฑ1.14 / 151.49 ms โ 149.23 / 150.20 ยฑ0.66 / 151.07 ms โ no change โ
โ QQuery 86 โ 39.34 / 40.66 ยฑ0.81 / 41.88 ms โ 39.38 / 40.36 ยฑ0.74 / 41.60 ms โ no change โ
โ QQuery 87 โ 87.13 / 90.63 ยฑ4.10 / 98.42 ms โ 85.74 / 90.97 ยฑ3.04 / 95.16 ms โ no change โ
โ QQuery 88 โ 100.27 / 101.81 ยฑ1.00 / 102.97 ms โ 100.77 / 101.71 ยฑ0.54 / 102.23 ms โ no change โ
โ QQuery 89 โ 119.58 / 122.00 ยฑ1.82 / 124.96 ms โ 120.39 / 121.35 ยฑ0.63 / 122.24 ms โ no change โ
โ QQuery 90 โ 24.62 / 24.90 ยฑ0.21 / 25.22 ms โ 24.02 / 24.54 ยฑ0.40 / 25.11 ms โ no change โ
โ QQuery 91 โ 64.30 / 65.98 ยฑ1.27 / 67.74 ms โ 60.90 / 64.72 ยฑ2.71 / 69.09 ms โ no change โ
โ QQuery 92 โ 57.54 / 58.31 ยฑ0.94 / 59.99 ms โ 59.28 / 59.87 ยฑ0.57 / 60.82 ms โ no change โ
โ QQuery 93 โ 188.92 / 191.55 ยฑ2.55 / 196.10 ms โ 191.27 / 193.77 ยฑ1.90 / 196.08 ms โ no change โ
โ QQuery 94 โ 61.25 / 62.23 ยฑ1.11 / 64.36 ms โ 62.69 / 63.34 ยฑ0.72 / 64.65 ms โ no change โ
โ QQuery 95 โ 128.39 / 131.56 ยฑ1.83 / 134.14 ms โ 129.31 / 130.20 ยฑ0.81 / 131.70 ms โ no change โ
โ QQuery 96 โ 71.76 / 75.11 ยฑ1.98 / 77.95 ms โ 76.01 / 77.00 ยฑ0.63 / 77.65 ms โ no change โ
โ QQuery 97 โ 127.19 / 129.33 ยฑ1.18 / 130.78 ms โ 131.28 / 132.93 ยฑ1.15 / 134.79 ms โ no change โ
โ QQuery 98 โ 155.25 / 157.28 ยฑ1.70 / 159.93 ms โ 155.22 / 157.49 ยฑ3.36 / 164.02 ms โ no change โ
โ QQuery 99 โ 10864.64 / 10914.19 ยฑ43.83 / 10975.24 ms โ 10942.12 / 11009.01 ยฑ66.80 / 11117.08 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 32039.95ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 32206.01ms โ
โ Average Time (HEAD) โ 323.64ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 325.31ms โ
โ Queries Faster โ 2 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 96 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.5s |
| Peak memory | 5.1 GiB |
| Avg memory | 4.3 GiB |
| CPU user | 264.8s |
| CPU sys | 17.5s |
| Peak spill | 0 B |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 161.3s |
| Peak memory | 5.0 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 265.1s |
| CPU sys | 17.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266210938
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266107489)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.19 / 4.47 ยฑ6.39 / 17.26 ms โ 1.17 / 4.47 ยฑ6.43 / 17.32 ms โ no change โ
โ QQuery 1 โ 14.88 / 15.00 ยฑ0.12 / 15.16 ms โ 14.36 / 14.77 ยฑ0.21 / 14.91 ms โ no change โ
โ QQuery 2 โ 44.18 / 44.29 ยฑ0.12 / 44.50 ms โ 44.25 / 45.26 ยฑ1.44 / 48.12 ms โ no change โ
โ QQuery 3 โ 40.17 / 40.83 ยฑ0.83 / 42.43 ms โ 39.70 / 40.18 ยฑ0.56 / 41.24 ms โ no change โ
โ QQuery 4 โ 286.12 / 299.55 ยฑ9.39 / 311.61 ms โ 281.80 / 293.05 ยฑ5.87 / 297.78 ms โ no change โ
โ QQuery 5 โ 344.33 / 349.58 ยฑ2.96 / 352.76 ms โ 343.83 / 350.49 ยฑ5.04 / 357.72 ms โ no change โ
โ QQuery 6 โ 5.24 / 6.59 ยฑ0.83 / 7.68 ms โ 5.34 / 6.16 ยฑ0.65 / 7.04 ms โ +1.07x faster โ
โ QQuery 7 โ 17.24 / 17.70 ยฑ0.60 / 18.89 ms โ 17.06 / 17.73 ยฑ0.58 / 18.54 ms โ no change โ
โ QQuery 8 โ 416.23 / 434.84 ยฑ13.14 / 449.85 ms โ 425.88 / 434.10 ยฑ5.73 / 439.43 ms โ no change โ
โ QQuery 9 โ 658.97 / 663.76 ยฑ3.43 / 668.16 ms โ 601.16 / 605.75 ยฑ4.25 / 612.17 ms โ +1.10x faster โ
โ QQuery 10 โ 89.74 / 92.78 ยฑ2.12 / 95.32 ms โ 90.32 / 92.09 ยฑ1.27 / 93.98 ms โ no change โ
โ QQuery 11 โ 104.30 / 104.57 ยฑ0.22 / 104.96 ms โ 103.50 / 104.40 ยฑ0.78 / 105.30 ms โ no change โ
โ QQuery 12 โ 344.93 / 349.70 ยฑ3.96 / 354.55 ms โ 343.40 / 348.99 ยฑ3.94 / 355.53 ms โ no change โ
โ QQuery 13 โ 450.52 / 464.22 ยฑ7.96 / 471.28 ms โ 477.68 / 487.69 ยฑ8.29 / 500.93 ms โ 1.05x slower โ
โ QQuery 14 โ 345.06 / 352.42 ยฑ5.46 / 361.01 ms โ 345.14 / 357.83 ยฑ8.47 / 369.73 ms โ no change โ
โ QQuery 15 โ 349.77 / 365.17 ยฑ14.01 / 391.32 ms โ 352.09 / 364.27 ยฑ11.54 / 378.35 ms โ no change โ
โ QQuery 16 โ 704.52 / 723.06 ยฑ17.63 / 748.05 ms โ 714.26 / 723.08 ยฑ9.96 / 739.97 ms โ no change โ
โ QQuery 17 โ 705.68 / 724.78 ยฑ10.28 / 735.21 ms โ 703.74 / 715.07 ยฑ6.41 / 723.56 ms โ no change โ
โ QQuery 18 โ 1413.21 / 1490.73 ยฑ43.97 / 1544.87 ms โ 1422.85 / 1470.53 ยฑ37.32 / 1520.03 ms โ no change โ
โ QQuery 19 โ 35.02 / 37.52 ยฑ2.66 / 42.55 ms โ 36.30 / 36.72 ยฑ0.46 / 37.54 ms โ no change โ
โ QQuery 20 โ 719.35 / 736.94 ยฑ20.24 / 770.86 ms โ 711.87 / 726.42 ยฑ14.22 / 753.01 ms โ no change โ
โ QQuery 21 โ 762.00 / 764.37 ยฑ2.52 / 768.87 ms โ 763.06 / 773.89 ยฑ12.08 / 794.69 ms โ no change โ
โ QQuery 22 โ 1131.47 / 1135.14 ยฑ4.00 / 1142.91 ms โ 1124.94 / 1131.80 ยฑ4.66 / 1137.02 ms โ no change โ
โ QQuery 23 โ 3080.83 / 3107.62 ยฑ20.78 / 3140.10 ms โ 3065.23 / 3090.77 ยฑ13.43 / 3104.69 ms โ no change โ
โ QQuery 24 โ 97.97 / 102.68 ยฑ3.02 / 106.62 ms โ 100.07 / 103.16 ยฑ2.13 / 106.52 ms โ no change โ
โ QQuery 25 โ 139.31 / 141.70 ยฑ1.58 / 143.33 ms โ 139.50 / 140.69 ยฑ0.81 / 142.00 ms โ no change โ
โ QQuery 26 โ 96.12 / 99.82 ยฑ2.95 / 105.02 ms โ 101.14 / 101.68 ยฑ0.39 / 102.15 ms โ no change โ
โ QQuery 27 โ 851.89 / 855.20 ยฑ3.04 / 860.10 ms โ 851.22 / 853.23 ยฑ1.07 / 854.15 ms โ no change โ
โ QQuery 28 โ 3282.10 / 3301.26 ยฑ12.46 / 3319.42 ms โ 3248.50 / 3265.86 ยฑ20.78 / 3306.17 ms โ no change โ
โ QQuery 29 โ 50.16 / 55.34 ยฑ3.16 / 58.93 ms โ 50.76 / 56.74 ยฑ4.69 / 64.75 ms โ no change โ
โ QQuery 30 โ 361.00 / 369.14 ยฑ8.55 / 385.36 ms โ 367.80 / 373.80 ยฑ7.84 / 389.12 ms โ no change โ
โ QQuery 31 โ 361.68 / 379.48 ยฑ10.13 / 390.07 ms โ 367.82 / 380.18 ยฑ8.13 / 390.59 ms โ no change โ
โ QQuery 32 โ 1025.97 / 1046.58 ยฑ18.12 / 1080.17 ms โ 1204.81 / 1247.16 ยฑ32.43 / 1301.01 ms โ 1.19x slower โ
โ QQuery 33 โ 1455.81 / 1466.22 ยฑ8.56 / 1479.10 ms โ 1456.70 / 1487.14 ยฑ41.39 / 1568.30 ms โ no change โ
โ QQuery 34 โ 1469.63 / 1501.95 ยฑ26.23 / 1541.18 ms โ 1481.93 / 1489.75 ยฑ11.04 / 1510.86 ms โ no change โ
โ QQuery 35 โ 391.42 / 398.26 ยฑ5.41 / 406.98 ms โ 382.18 / 394.70 ยฑ7.70 / 404.01 ms โ no change โ
โ QQuery 36 โ 118.91 / 121.91 ยฑ1.59 / 123.26 ms โ 118.76 / 123.89 ยฑ3.40 / 127.47 ms โ no change โ
โ QQuery 37 โ 46.96 / 49.17 ยฑ1.60 / 51.29 ms โ 46.74 / 48.65 ยฑ1.45 / 50.27 ms โ no change โ
โ QQuery 38 โ 74.34 / 76.61 ยฑ1.59 / 78.57 ms โ 74.91 / 76.83 ยฑ1.57 / 79.37 ms โ no change โ
โ QQuery 39 โ 205.85 / 217.00 ยฑ9.26 / 230.25 ms โ 205.15 / 218.09 ยฑ9.01 / 228.42 ms โ no change โ
โ QQuery 40 โ 24.13 / 26.67 ยฑ1.77 / 29.04 ms โ 23.85 / 26.27 ยฑ1.67 / 28.37 ms โ no change โ
โ QQuery 41 โ 21.14 / 22.03 ยฑ1.21 / 24.34 ms โ 20.81 / 21.13 ยฑ0.35 / 21.79 ms โ no change โ
โ QQuery 42 โ 20.29 / 20.40 ยฑ0.07 / 20.47 ms โ 19.55 / 20.02 ยฑ0.42 / 20.78 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 22577.06ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 22664.46ms โ
โ Average Time (HEAD) โ 525.05ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 527.08ms โ
โ Queries Faster โ 2 โ
โ Queries Slower โ 2 โ
โ Queries with No Change โ 39 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 113.9s |
| Peak memory | 37.5 GiB |
| Avg memory | 26.8 GiB |
| CPU user | 1076.5s |
| CPU sys | 86.2s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 114.2s |
| Peak memory | 37.0 GiB |
| Avg memory | 26.3 GiB |
| CPU user | 1069.1s |
| CPU sys | 96.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4241906267
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4241897983)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4241897983-1203-9qx2s 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (8d60834b3b09bad5093b74d0a32799e3dc98dd0a) to a13c23d (merge-base) [diff](https://github.com/apache/datafusion/compare/a13c23df67c218e45342b80238cae490609d8f3d..8d60834b3b09bad5093b74d0a32799e3dc98dd0a) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4241897983
run benchmark tpch tpcds clickbench_partitioned clickbench_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3078111076
##########
datafusion/functions-aggregate/src/count.rs:
##########
@@ -336,18 +337,70 @@ impl AggregateUDFImpl for Count {
}
fn groups_accumulator_supported(&self, args: AccumulatorArgs) -> bool {
- // groups accumulator only supports `COUNT(c1)`, not
- // `COUNT(c1, c2)`, etc
- if args.is_distinct {
+ if args.exprs.len() != 1 {
return false;
}
- args.exprs.len() == 1
+ if args.is_distinct {
+ // Only support primitive integer types for now
+ matches!(
+ args.expr_fields[0].data_type(),
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::UInt8
+ | DataType::UInt16
+ | DataType::UInt32
+ | DataType::UInt64
+ )
+ } else {
+ true
+ }
}
fn create_groups_accumulator(
&self,
- _args: AccumulatorArgs,
+ args: AccumulatorArgs,
) -> Result<Box<dyn GroupsAccumulator>> {
+ if args.is_distinct {
+ let data_type = args.expr_fields[0].data_type();
+ return match data_type {
+ DataType::Int8 => Ok(Box::new(
Review Comment:
Ah sorry, I meant https://github.com/apache/datafusion/pull/21456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4242372936
run benchmark clickbench_partitioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on code in PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#discussion_r3068985374
##########
datafusion/functions-aggregate-common/src/aggregate/count_distinct/groups.rs:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, BooleanArray, Int64Array, ListBuilder, PrimitiveBuilder,
+};
+use arrow::datatypes::ArrowPrimitiveType;
+use datafusion_common::HashSet;
+use datafusion_common::hash_utils::RandomState;
+use datafusion_expr_common::groups_accumulator::{EmitTo, GroupsAccumulator};
+use std::hash::Hash;
+use std::mem::size_of;
+use std::sync::Arc;
+
+pub struct PrimitiveDistinctCountGroupsAccumulator<T: ArrowPrimitiveType>
+where
+ T::Native: Eq + Hash,
+{
+ /// Count distinct per group.
+ values: Vec<HashSet<T::Native, RandomState>>,
Review Comment:
Yup! I am evaluating that approach but it doesn't seem to be working out so far
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4230790093
Benchmark for [this request](https://github.com/apache/datafusion/pull/21561#issuecomment-4230786454) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
struct_query_sql
substr
substr_index
substring
sum
to_char
to_hex
to_local_time
to_time
to_timestamp
topk_aggregate
topk_repartition
translate
trim
trunc
unhex
upper
uuid
window_query_sql
with_hashes
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4231155633
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4231151449)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4231151449-1105-xpttn 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (ee0c8650aacfb92d852062a1c0bcfb5465c93482) to 16e578d (merge-base) [diff](https://github.com/apache/datafusion/compare/16e578dfdd28c8bffc6a21f54164b889e79bf75c..ee0c8650aacfb92d852062a1c0bcfb5465c93482) using: clickbench_extended
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4231155521
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4231151449)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4231151449-1102-g9w92 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (ee0c8650aacfb92d852062a1c0bcfb5465c93482) to 16e578d (merge-base) [diff](https://github.com/apache/datafusion/compare/16e578dfdd28c8bffc6a21f54164b889e79bf75c..ee0c8650aacfb92d852062a1c0bcfb5465c93482) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4251569595
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4251531266)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4251531266-1283-96mtn 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (28c607aad4745f0b021b60bf42e1f3b7d49c0971) to 2818abb (merge-base) [diff](https://github.com/apache/datafusion/compare/2818abb8a10bf8b001669c15cf597f9ddfb86ed1..28c607aad4745f0b021b60bf42e1f3b7d49c0971) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4251692895
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4251531266)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and implement_groups_accumulator_count_distinct_primitive_types
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ implement_groups_accumulator_count_distinct_primitive_types โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.27 / 4.62 ยฑ6.53 / 17.68 ms โ 1.26 / 4.65 ยฑ6.56 / 17.78 ms โ no change โ
โ QQuery 1 โ 15.42 / 15.93 ยฑ0.39 / 16.59 ms โ 14.46 / 15.32 ยฑ0.69 / 16.56 ms โ no change โ
โ QQuery 2 โ 44.72 / 45.12 ยฑ0.30 / 45.63 ms โ 44.82 / 45.12 ยฑ0.32 / 45.70 ms โ no change โ
โ QQuery 3 โ 41.98 / 44.66 ยฑ2.70 / 48.19 ms โ 45.26 / 46.67 ยฑ1.72 / 49.77 ms โ no change โ
โ QQuery 4 โ 311.45 / 315.31 ยฑ3.72 / 321.49 ms โ 312.68 / 318.16 ยฑ4.41 / 324.74 ms โ no change โ
โ QQuery 5 โ 344.82 / 364.62 ยฑ11.36 / 375.79 ms โ 350.08 / 367.43 ยฑ8.76 / 373.30 ms โ no change โ
โ QQuery 6 โ 5.82 / 6.74 ยฑ0.78 / 7.88 ms โ 6.09 / 7.60 ยฑ1.18 / 9.29 ms โ 1.13x slower โ
โ QQuery 7 โ 16.79 / 17.70 ยฑ0.53 / 18.39 ms โ 16.37 / 18.68 ยฑ2.34 / 23.10 ms โ 1.06x slower โ
โ QQuery 8 โ 423.62 / 445.75 ยฑ22.14 / 479.49 ms โ 426.95 / 436.55 ยฑ9.53 / 453.96 ms โ no change โ
โ QQuery 9 โ 637.21 / 665.64 ยฑ21.50 / 691.88 ms โ 608.71 / 631.10 ยฑ24.11 / 671.05 ms โ +1.05x faster โ
โ QQuery 10 โ 92.23 / 94.95 ยฑ3.12 / 99.45 ms โ 96.48 / 98.47 ยฑ1.47 / 100.82 ms โ no change โ
โ QQuery 11 โ 104.73 / 110.57 ยฑ3.58 / 115.12 ms โ 103.03 / 105.24 ยฑ2.26 / 109.20 ms โ no change โ
โ QQuery 12 โ 341.03 / 354.87 ยฑ14.12 / 375.64 ms โ 332.85 / 355.42 ยฑ16.35 / 376.11 ms โ no change โ
โ QQuery 13 โ 457.75 / 483.94 ยฑ15.90 / 502.56 ms โ 451.32 / 480.23 ยฑ22.44 / 507.61 ms โ no change โ
โ QQuery 14 โ 344.56 / 360.26 ยฑ13.92 / 384.94 ms โ 338.74 / 347.21 ยฑ9.84 / 365.28 ms โ no change โ
โ QQuery 15 โ 354.38 / 369.76 ยฑ10.79 / 387.38 ms โ 346.08 / 361.86 ยฑ17.72 / 385.99 ms โ no change โ
โ QQuery 16 โ 736.66 / 756.38 ยฑ19.61 / 792.46 ms โ 724.07 / 735.91 ยฑ11.49 / 757.14 ms โ no change โ
โ QQuery 17 โ 741.30 / 750.40 ยฑ6.51 / 757.20 ms โ 718.75 / 736.25 ยฑ16.69 / 760.37 ms โ no change โ
โ QQuery 18 โ 1467.08 / 1503.86 ยฑ27.99 / 1533.36 ms โ 1389.85 / 1415.00 ยฑ18.91 / 1445.40 ms โ +1.06x faster โ
โ QQuery 19 โ 37.47 / 41.28 ยฑ4.88 / 50.76 ms โ 35.98 / 36.62 ยฑ0.61 / 37.66 ms โ +1.13x faster โ
โ QQuery 20 โ 715.65 / 738.13 ยฑ18.30 / 764.60 ms โ 717.89 / 724.87 ยฑ6.37 / 736.52 ms โ no change โ
โ QQuery 21 โ 765.11 / 769.84 ยฑ2.49 / 772.12 ms โ 756.80 / 766.10 ยฑ4.97 / 770.62 ms โ no change โ
โ QQuery 22 โ 1142.87 / 1153.12 ยฑ9.71 / 1171.15 ms โ 1139.94 / 1146.19 ยฑ4.57 / 1152.76 ms โ no change โ
โ QQuery 23 โ 3095.62 / 3158.97 ยฑ41.12 / 3210.34 ms โ 3117.51 / 3180.78 ยฑ42.53 / 3248.05 ms โ no change โ
โ QQuery 24 โ 103.32 / 106.87 ยฑ3.43 / 112.87 ms โ 103.88 / 107.48 ยฑ3.03 / 113.04 ms โ no change โ
โ QQuery 25 โ 142.12 / 143.98 ยฑ1.83 / 147.13 ms โ 140.96 / 144.16 ยฑ1.91 / 146.40 ms โ no change โ
โ QQuery 26 โ 103.41 / 108.17 ยฑ2.85 / 112.12 ms โ 102.99 / 105.08 ยฑ1.85 / 107.17 ms โ no change โ
โ QQuery 27 โ 856.95 / 868.87 ยฑ10.24 / 882.07 ms โ 851.71 / 858.69 ยฑ8.80 / 873.82 ms โ no change โ
โ QQuery 28 โ 3301.20 / 3316.62 ยฑ15.44 / 3343.52 ms โ 3251.45 / 3271.33 ยฑ16.92 / 3296.87 ms โ no change โ
โ QQuery 29 โ 51.21 / 55.33 ยฑ4.40 / 61.80 ms โ 51.45 / 55.58 ยฑ3.88 / 62.93 ms โ no change โ
โ QQuery 30 โ 355.38 / 372.01 ยฑ17.73 / 403.64 ms โ 369.28 / 382.46 ยฑ8.79 / 394.23 ms โ no change โ
โ QQuery 31 โ 376.80 / 393.28 ยฑ14.87 / 415.93 ms โ 365.15 / 386.36 ยฑ14.48 / 405.75 ms โ no change โ
โ QQuery 32 โ 1245.62 / 1288.78 ยฑ35.46 / 1351.63 ms โ 1081.60 / 1092.09 ยฑ11.51 / 1113.19 ms โ +1.18x faster โ
โ QQuery 33 โ 1512.72 / 1586.95 ยฑ43.72 / 1634.88 ms โ 1473.39 / 1515.48 ยฑ27.91 / 1553.76 ms โ no change โ
โ QQuery 34 โ 1496.85 / 1568.58 ยฑ75.06 / 1712.16 ms โ 1458.53 / 1500.80 ยฑ23.91 / 1532.14 ms โ no change โ
โ QQuery 35 โ 387.58 / 419.20 ยฑ24.77 / 445.72 ms โ 387.02 / 411.72 ยฑ17.69 / 434.56 ms โ no change โ
โ QQuery 36 โ 122.41 / 125.05 ยฑ2.70 / 130.11 ms โ 117.85 / 126.38 ยฑ5.23 / 133.81 ms โ no change โ
โ QQuery 37 โ 47.78 / 49.30 ยฑ0.92 / 50.10 ms โ 51.05 / 51.93 ยฑ0.91 / 53.58 ms โ 1.05x slower โ
โ QQuery 38 โ 78.34 / 79.98 ยฑ0.85 / 80.74 ms โ 75.17 / 78.33 ยฑ2.98 / 82.44 ms โ no change โ
โ QQuery 39 โ 234.64 / 243.46 ยฑ6.47 / 254.40 ms โ 215.10 / 221.43 ยฑ4.58 / 226.30 ms โ +1.10x faster โ
โ QQuery 40 โ 26.76 / 28.90 ยฑ1.35 / 30.54 ms โ 23.38 / 26.41 ยฑ1.90 / 28.38 ms โ +1.09x faster โ
โ QQuery 41 โ 21.28 / 22.25 ยฑ0.55 / 22.99 ms โ 20.00 / 21.00 ยฑ0.62 / 21.81 ms โ +1.06x faster โ
โ QQuery 42 โ 20.85 / 21.27 ยฑ0.28 / 21.73 ms โ 18.73 / 20.09 ยฑ1.61 / 23.12 ms โ +1.06x faster โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 23371.28ms โ
โ Total Time (implement_groups_accumulator_count_distinct_primitive_types) โ 22758.22ms โ
โ Average Time (HEAD) โ 543.52ms โ
โ Average Time (implement_groups_accumulator_count_distinct_primitive_types) โ 529.26ms โ
โ Queries Faster โ 8 โ
โ Queries Slower โ 3 โ
โ Queries with No Change โ 32 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 118.0s |
| Peak memory | 37.4 GiB |
| Avg memory | 27.1 GiB |
| CPU user | 1099.7s |
| CPU sys | 100.1s |
| Peak spill | 0 B |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 114.9s |
| Peak memory | 43.0 GiB |
| Avg memory | 33.7 GiB |
| CPU user | 1097.1s |
| CPU sys | 79.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4262001493
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4261984688)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4261984688-1388-7h2pw 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (ba140a00af402b7f8e6ff3470d1f50b369766d69) to cc4717a (merge-base) [diff](https://github.com/apache/datafusion/compare/cc4717a14452e65a4d99c4b180d62c2793b713bd..ba140a00af402b7f8e6ff3470d1f50b369766d69) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4265739919
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4265729704)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4265729704-1402-jjz9w 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (47110a861000c38c0c3acc6a3fbdd352f480fee3) to 882be98 (merge-base) [diff](https://github.com/apache/datafusion/compare/882be98a82bdf37390b9dbfa77ffefd131c65ce5..47110a861000c38c0c3acc6a3fbdd352f480fee3) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4266117461
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4266107489)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4266107489-1408-w7df7 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (066504d269b8e402d6b732729ae5265a27ed8458) to 5a427cb (merge-base) [diff](https://github.com/apache/datafusion/compare/5a427cb55d8630b51b9590a25b369b434611975e..066504d269b8e402d6b732729ae5265a27ed8458) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4256994993
Benchmark for [this request](https://github.com/apache/datafusion/pull/21561#issuecomment-4256919523) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
BENCHMARK: tpch
QUERY: All
DATAFUSION_DIR: /workspace/datafusion-base
BRANCH_NAME: HEAD
DATA_DIR: /workspace/datafusion-bench/benchmarks/data
RESULTS_DIR: /workspace/datafusion-bench/benchmarks/results/HEAD
CARGO_COMMAND: cargo run --release
PREFER_HASH_JOIN: true
SIMULATE_LATENCY: false
***************************
RESULTS_FILE: /workspace/datafusion-bench/benchmarks/results/HEAD/tpch_sf1.json
Running tpch benchmark...
+ cargo run --release --bin dfbench -- tpch --iterations 5 --path /workspace/datafusion-bench/benchmarks/data/tpch_sf1 --scale-factor 1 --prefer_hash_join true --format parquet -o /workspace/datafusion-bench/benchmarks/results/HEAD/tpch_sf1.json
Finished `release` profile [optimized] target(s) in 0.22s
Running `/workspace/datafusion-base/target/release/dfbench tpch --iterations 5 --path /workspace/datafusion-bench/benchmarks/data/tpch_sf1 --scale-factor 1 --prefer_hash_join true --format parquet -o /workspace/datafusion-bench/benchmarks/results/HEAD/tpch_sf1.json`
error: unexpected argument '--scale-factor' found
Usage: dfbench tpch --path <PATH> --iterations <ITERATIONS>
For more information, try '--help'.
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4256932516
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21561#issuecomment-4256919523)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4256919523-1336-6rf2n 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing implement_groups_accumulator_count_distinct_primitive_types (b9dcf85869706b90288153c487aaebd186cf30c8) to 2818abb (merge-base) [diff](https://github.com/apache/datafusion/compare/2818abb8a10bf8b001669c15cf597f9ddfb86ed1..b9dcf85869706b90288153c487aaebd186cf30c8) using: clickbench_extended
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: Implement groups accumulator count distinct primitive types [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21561:
URL: https://github.com/apache/datafusion/pull/21561#issuecomment-4292778636
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org