You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Rachelint (via GitHub)" <gi...@apache.org> on 2025/04/05 07:48:02 UTC
[PR] Impl Intermeidate result blocked approach framework [datafusion]
Rachelint opened a new pull request, #15591:
URL: https://github.com/apache/datafusion/pull/15591
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3731202925
> Maybe someday we'll get back to it π’
Plan to reopen to push it forward again, due to it has been proved to be helpful, and have free time again recently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3084020685
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ collections::VecDeque,
+ fmt::Debug,
+ iter,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct Blocks<B: Block> {
+ inner: VecDeque<B>,
Review Comment:
Yes, it is better to use `Vec<T>` and I tried it when I still see it a performance improvement feature.
However, after many tries, I found it actually can't help dafafusion run faster (it is only something can help to better memory management)... And I finally switch to use `VecDeque` for simplicity...
The experiments can be saw in this archived branch:
https://github.com/Rachelint/arrow-datafusion/compare/intermeidate-result-blocked-approach-bak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276059331
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4276055952)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4276055952-1550-b9fqh 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (a68716b1b99123fef41259899c0c67adda870dbb) to dc973cc (merge-base) [diff](https://github.com/apache/datafusion/compare/dc973cc9e67e3519ec4bc5bd15962e2692debb3e..a68716b1b99123fef41259899c0c67adda870dbb) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276055952
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072425201
##########
datafusion/common/src/config.rs:
##########
@@ -405,6 +405,18 @@ config_namespace! {
/// in joins can reduce memory usage when joining large
/// tables with a highly-selective join filter, but is also slightly slower.
pub enforce_batch_size_in_joins: bool, default = false
+
Review Comment:
Yes, it is just a way for `going back + testing`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2850176179
Hi @alamb , I have addressed most points mentioned in code review.
And the new added sql in `extended.sql` I think can show the improvement has been merged. I checked it in my local, and it got improvment as expected (see Q7).
```
--------------------
Benchmark clickbench_extended.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β main β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1963.28ms β 1947.08ms β no change β
β QQuery 1 β 1180.94ms β 1135.65ms β no change β
β QQuery 2 β 2420.21ms β 2372.42ms β no change β
β QQuery 3 β 968.18ms β 1091.04ms β 1.13x slower β
β QQuery 4 β 2955.90ms β 2975.36ms β no change β
β QQuery 5 β 32623.33ms β 32108.79ms β no change β
β QQuery 6 β 5201.07ms β 5205.19ms β no change β
β QQuery 7 β 7865.80ms β 7237.53ms β +1.09x faster β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (main) β 55178.69ms β
β Total Time (intermeidate-result-blocked-approach) β 54073.06ms β
β Average Time (main) β 6897.34ms β
β Average Time (intermeidate-result-blocked-approach) β 6759.13ms β
β Queries Faster β 1 β
β Queries Slower β 1 β
β Queries with No Change β 6 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
> My big concern is that we have a plan to eventually avoid both single and multi blocked management
And here is my thoughts about how we remove `single block management` code path eventually:
- The hard point is that `streaming aggreagtion` need `Emit::First(n)`, and as I see we should not support `Emit::First(n)` (mainly performance reason, it is expansive and complex now, and even more expansive and complex in `blocked management`).
- However, I think it may be possible to remove the usage of `Emit::First(n)`, I am considering how to make it. And if we made it, the `single block management` path can be removed after we implement `blocked management` for all exists `GroupsAccumulator` and `GroupValues`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075197407
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,20 +94,27 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
Review Comment:
Ah I see that batch size is used by default :).
Yeah I think it makes sense to test it a bit further, maybe for this a slightly larger value (e.g. 2x, 4x batch size) will be beneficial when the cardinality is above the batch size.
Also at some point might make sense to think of it in size in memory instead of number of elements (e.g. block of `u8` values might hold 16x more values than u128).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890539606
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_extended.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1916.94ms β 1866.52ms β no change β
β QQuery 1 β 709.88ms β 673.73ms β +1.05x faster β
β QQuery 2 β 1482.93ms β 1391.57ms β +1.07x faster β
β QQuery 3 β 701.88ms β 721.01ms β no change β
β QQuery 4 β 1494.91ms β 1494.55ms β no change β
β QQuery 5 β 15627.54ms β 15418.84ms β no change β
β QQuery 6 β 2070.08ms β 2042.39ms β no change β
β QQuery 7 β 2108.54ms β 1952.66ms β +1.08x faster β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 26112.69ms β
β Total Time (intermeidate-result-blocked-approach) β 25561.26ms β
β Average Time (HEAD) β 3264.09ms β
β Average Time (intermeidate-result-blocked-approach) β 3195.16ms β
β Queries Faster β 3 β
β Queries Slower β 0 β
β Queries with No Change β 5 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890663260
> Me too -- I looked at the flamegraph you provided and I agree it seems like almost half the allocation time is spent with pagefaults / zeroing memory. However, I can't tell if that is because there is slowness with the underlying Vec that wasn't initialized or if there is something else going on.
I think I nearly understand why about this, it is possibly be led by `lto`, `lto` seems found the initialization is unnecessary actually, so it remove it(just like calling `set_len` manually).
> I suspect you already know this, but I think you can get back the original Vec from an array via
Got it!
> From those numbers, is it a fair assessment that the blocked approach improves performance when there is a large number of intermediate groups, but does not when there is a small number of groups?
I think it maybe be possible in this situation?
- When `input` for `accumualtor` and `group values` is consumed, we collect them, and transform them back to `Vec`.
- Then we push them in `accumualtor` and `group values`.
- Finally we reuse them in next round computation of `accumualtor` and `group values`?
> From those numbers, is it a fair assessment that the blocked approach improves performance when there is a large number of intermediate groups, but does not when there is a small number of groups?
Yes, in current implementation, it only help to performance with `large amount` of intermediate groups, because the `slice` will be called many many time and the cost become unacceptable.
And for query with only small groups, it nearly make no difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2889120962
@alamb Hello, is it ok to trigger the benchmark again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2909172790
> My biggest concern is if we can get more obvious improvement to make the change worthy...
As I understood the way to get a bigger improvement would be to implement the chunked approach for more group storage / aggregates so that more queries in our benchmarks (like ClickBench) could use the new code path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2911070606
I agree with @adriangb , even when it doesn't provide performance improvements still super valuable to push it forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4321818488
I think the regression possibly comes from the introducing of `VecDeque` in `GroupValuesPrimitive`, I am removing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3080688241
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,21 +94,31 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
+ const DEFAULT_BLOCK_CAP: usize = 128;
+
assert_eq!(values.len(), 1, "single argument to update_batch");
let values = values[0].as_primitive::<T>();
- // update values
- self.values.resize(total_num_groups, self.starting_value);
+ // Expand to ensure values are large enough
+ let new_block = |block_size: Option<usize>| {
+ let cap = block_size.unwrap_or(DEFAULT_BLOCK_CAP);
+ Vec::with_capacity(cap)
+ };
+ self.values
+ .resize(total_num_groups, new_block, self.starting_value);
// NullState dispatches / handles tracking nulls and groups that saw no values
self.null_state.accumulate(
group_indices,
values,
opt_filter,
total_num_groups,
- |group_index, new_value| {
- // SAFETY: group_index is guaranteed to be in bounds
- let value = unsafe { self.values.get_unchecked_mut(group_index) };
+ |block_id, block_offset, new_value| {
+ // SAFETY: `block_id` and `block_offset` are guaranteed to be in bounds
+ let value = unsafe {
+ self.values[block_id as usize]
Review Comment:
this can use unsafe index as well (with a plain `Vec` it would certainly be faster)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4343733161
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4343604373)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 40.80 / 42.01 Β±1.13 / 43.54 ms β 40.34 / 41.03 Β±1.08 / 43.17 ms β no change β
β QQuery 2 β 21.45 / 21.62 Β±0.09 / 21.73 ms β 21.46 / 21.80 Β±0.51 / 22.81 ms β no change β
β QQuery 3 β 38.57 / 40.03 Β±1.59 / 42.27 ms β 38.29 / 40.24 Β±1.18 / 41.68 ms β no change β
β QQuery 4 β 18.47 / 18.90 Β±0.75 / 20.40 ms β 18.43 / 18.76 Β±0.40 / 19.53 ms β no change β
β QQuery 5 β 49.41 / 52.85 Β±2.22 / 55.90 ms β 49.47 / 52.69 Β±2.74 / 56.93 ms β no change β
β QQuery 6 β 18.25 / 18.62 Β±0.57 / 19.75 ms β 17.95 / 18.23 Β±0.20 / 18.55 ms β no change β
β QQuery 7 β 54.57 / 55.57 Β±0.56 / 56.16 ms β 53.37 / 55.60 Β±2.05 / 58.28 ms β no change β
β QQuery 8 β 48.55 / 48.94 Β±0.23 / 49.17 ms β 47.93 / 48.38 Β±0.28 / 48.78 ms β no change β
β QQuery 9 β 54.81 / 57.25 Β±2.08 / 60.30 ms β 54.04 / 55.16 Β±1.41 / 57.89 ms β no change β
β QQuery 10 β 66.09 / 68.44 Β±1.77 / 70.86 ms β 66.52 / 68.64 Β±1.75 / 71.53 ms β no change β
β QQuery 11 β 14.20 / 14.35 Β±0.12 / 14.53 ms β 14.32 / 14.52 Β±0.25 / 14.96 ms β no change β
β QQuery 12 β 27.75 / 28.23 Β±0.38 / 28.68 ms β 27.29 / 27.56 Β±0.23 / 27.83 ms β no change β
β QQuery 13 β 37.73 / 40.20 Β±3.16 / 46.41 ms β 38.48 / 39.24 Β±0.54 / 40.10 ms β no change β
β QQuery 14 β 28.02 / 28.48 Β±0.46 / 29.34 ms β 28.09 / 28.28 Β±0.13 / 28.47 ms β no change β
β QQuery 15 β 35.84 / 36.57 Β±0.53 / 37.18 ms β 35.61 / 36.11 Β±0.32 / 36.46 ms β no change β
β QQuery 16 β 15.53 / 16.27 Β±0.41 / 16.69 ms β 16.05 / 16.40 Β±0.22 / 16.71 ms β no change β
β QQuery 17 β 78.74 / 80.64 Β±1.77 / 83.42 ms β 84.26 / 85.25 Β±0.80 / 86.15 ms β 1.06x slower β
β QQuery 18 β 76.83 / 78.67 Β±2.05 / 82.50 ms β 76.34 / 78.26 Β±1.63 / 81.20 ms β no change β
β QQuery 19 β 38.44 / 40.05 Β±0.87 / 41.03 ms β 38.80 / 40.39 Β±0.89 / 41.50 ms β no change β
β QQuery 20 β 39.88 / 40.68 Β±1.50 / 43.69 ms β 40.01 / 40.22 Β±0.18 / 40.55 ms β no change β
β QQuery 21 β 63.73 / 65.15 Β±1.50 / 68.04 ms β 62.87 / 64.26 Β±1.10 / 65.87 ms β no change β
β QQuery 22 β 24.02 / 24.80 Β±0.94 / 26.54 ms β 23.97 / 24.28 Β±0.19 / 24.56 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 918.32ms β
β Total Time (intermeidate-result-blocked-approach) β 915.29ms β
β Average Time (HEAD) β 41.74ms β
β Average Time (intermeidate-result-blocked-approach) β 41.60ms β
β Queries Faster β 0 β
β Queries Slower β 1 β
β Queries with No Change β 21 β
β Queries with Failure β 0 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 5.7 GiB |
| Avg memory | 5.0 GiB |
| CPU user | 34.2s |
| CPU sys | 2.7s |
| Peak spill | 0 B |
**tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 5.7 GiB |
| Avg memory | 5.0 GiB |
| CPU user | 34.5s |
| CPU sys | 2.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4785295926
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4785188060)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.23 / 3.91 Β±5.30 / 14.51 ms β 1.22 / 3.90 Β±5.29 / 14.48 ms β no change β
β QQuery 1 β 12.52 / 12.80 Β±0.16 / 13.00 ms β 12.77 / 12.87 Β±0.05 / 12.93 ms β no change β
β QQuery 2 β 35.72 / 35.98 Β±0.25 / 36.35 ms β 35.93 / 36.41 Β±0.60 / 37.43 ms β no change β
β QQuery 3 β 30.19 / 30.83 Β±0.71 / 32.21 ms β 30.03 / 30.36 Β±0.27 / 30.80 ms β no change β
β QQuery 4 β 220.44 / 222.86 Β±1.27 / 223.76 ms β 232.67 / 236.08 Β±2.39 / 239.41 ms β 1.06x slower β
β QQuery 5 β 275.25 / 278.35 Β±2.41 / 281.95 ms β 265.43 / 268.94 Β±3.19 / 273.97 ms β no change β
β QQuery 6 β 1.26 / 1.41 Β±0.23 / 1.86 ms β 1.25 / 1.40 Β±0.22 / 1.84 ms β no change β
β QQuery 7 β 14.28 / 14.44 Β±0.09 / 14.54 ms β 13.64 / 13.81 Β±0.15 / 14.01 ms β no change β
β QQuery 8 β 330.48 / 335.49 Β±3.02 / 339.34 ms β 315.81 / 318.80 Β±2.80 / 323.81 ms β no change β
β QQuery 9 β 473.44 / 483.37 Β±5.48 / 490.16 ms β 454.22 / 464.83 Β±7.98 / 477.60 ms β no change β
β QQuery 10 β 70.99 / 73.29 Β±3.74 / 80.69 ms β 68.19 / 72.03 Β±6.11 / 84.20 ms β no change β
β QQuery 11 β 81.42 / 83.29 Β±1.46 / 84.98 ms β 79.85 / 81.96 Β±1.08 / 82.78 ms β no change β
β QQuery 12 β 277.56 / 283.69 Β±5.17 / 289.56 ms β 262.77 / 267.14 Β±3.56 / 271.51 ms β +1.06x faster β
β QQuery 13 β 380.32 / 394.52 Β±11.41 / 413.87 ms β 363.62 / 380.65 Β±9.00 / 389.67 ms β no change β
β QQuery 14 β 288.98 / 295.10 Β±4.33 / 301.87 ms β 277.79 / 281.39 Β±2.72 / 285.07 ms β no change β
β QQuery 15 β 283.82 / 292.30 Β±5.70 / 300.28 ms β 281.75 / 293.14 Β±7.76 / 302.75 ms β no change β
β QQuery 16 β 624.39 / 638.83 Β±10.83 / 657.01 ms β 605.26 / 613.60 Β±4.85 / 619.95 ms β no change β
β QQuery 17 β 630.65 / 646.02 Β±8.95 / 656.39 ms β 617.25 / 623.47 Β±6.64 / 632.77 ms β no change β
β QQuery 18 β 1291.95 / 1322.58 Β±24.46 / 1363.70 ms β 1240.33 / 1261.01 Β±16.16 / 1285.74 ms β no change β
β QQuery 19 β 27.94 / 28.41 Β±0.26 / 28.70 ms β 27.41 / 29.59 Β±4.28 / 38.15 ms β no change β
β QQuery 20 β 515.48 / 529.33 Β±11.54 / 544.72 ms β 517.30 / 521.34 Β±3.04 / 526.61 ms β no change β
β QQuery 21 β 518.96 / 526.74 Β±5.53 / 535.97 ms β 509.46 / 519.34 Β±5.92 / 526.44 ms β no change β
β QQuery 22 β 1003.60 / 1030.83 Β±23.45 / 1060.15 ms β 982.29 / 992.46 Β±5.89 / 999.88 ms β no change β
β QQuery 23 β 3163.56 / 3208.28 Β±48.54 / 3295.42 ms β 3151.81 / 3211.18 Β±40.42 / 3266.77 ms β no change β
β QQuery 24 β 41.49 / 51.52 Β±5.77 / 58.79 ms β 41.80 / 42.67 Β±0.98 / 44.35 ms β +1.21x faster β
β QQuery 25 β 111.27 / 111.96 Β±0.89 / 113.61 ms β 111.04 / 113.57 Β±4.15 / 121.80 ms β no change β
β QQuery 26 β 41.50 / 42.35 Β±0.50 / 42.87 ms β 41.51 / 42.27 Β±0.70 / 43.54 ms β no change β
β QQuery 27 β 664.58 / 675.43 Β±5.79 / 680.45 ms β 674.35 / 685.78 Β±14.19 / 710.56 ms β no change β
β QQuery 28 β 3030.00 / 3034.51 Β±4.91 / 3042.90 ms β 3020.15 / 3034.08 Β±7.84 / 3043.74 ms β no change β
β QQuery 29 β 40.73 / 45.01 Β±6.31 / 57.27 ms β 40.36 / 40.66 Β±0.50 / 41.65 ms β +1.11x faster β
β QQuery 30 β 296.81 / 309.65 Β±12.07 / 331.50 ms β 297.35 / 308.84 Β±12.39 / 328.12 ms β no change β
β QQuery 31 β 278.98 / 288.15 Β±7.22 / 299.13 ms β 276.96 / 284.72 Β±5.08 / 289.46 ms β no change β
β QQuery 32 β 916.77 / 959.08 Β±35.57 / 1013.44 ms β 925.97 / 977.05 Β±44.21 / 1052.50 ms β no change β
β QQuery 33 β 1426.83 / 1454.86 Β±18.33 / 1478.22 ms β 1437.69 / 1473.51 Β±29.58 / 1512.16 ms β no change β
β QQuery 34 β 1486.59 / 1541.22 Β±54.65 / 1639.59 ms β 1487.08 / 1563.69 Β±109.94 / 1781.94 ms β no change β
β QQuery 35 β 274.38 / 311.25 Β±31.15 / 354.24 ms β 288.85 / 331.06 Β±72.24 / 475.19 ms β 1.06x slower β
β QQuery 36 β 63.27 / 68.48 Β±3.65 / 73.27 ms β 65.90 / 70.54 Β±4.44 / 78.58 ms β no change β
β QQuery 37 β 36.26 / 43.59 Β±12.86 / 69.24 ms β 35.18 / 40.41 Β±4.38 / 44.95 ms β +1.08x faster β
β QQuery 38 β 42.87 / 46.52 Β±3.37 / 50.52 ms β 40.11 / 42.59 Β±1.46 / 44.01 ms β +1.09x faster β
β QQuery 39 β 143.45 / 148.86 Β±6.48 / 160.77 ms β 146.04 / 151.46 Β±5.63 / 160.01 ms β no change β
β QQuery 40 β 13.90 / 21.07 Β±9.56 / 39.77 ms β 13.71 / 14.23 Β±0.68 / 15.52 ms β +1.48x faster β
β QQuery 41 β 13.54 / 15.77 Β±2.88 / 21.42 ms β 13.26 / 13.56 Β±0.19 / 13.81 ms β +1.16x faster β
β QQuery 42 β 12.83 / 13.14 Β±0.17 / 13.37 ms β 12.93 / 13.12 Β±0.10 / 13.20 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 19955.07ms β
β Total Time (intermeidate-result-blocked-approach) β 19779.53ms β
β Average Time (HEAD) β 464.07ms β
β Average Time (intermeidate-result-blocked-approach) β 459.99ms β
β Queries Faster β 7 β
β Queries Slower β 2 β
β Queries with No Change β 34 β
β Queries with Failure β 0 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 11.0 GiB |
| Avg memory | 4.3 GiB |
| CPU user | 1013.6s |
| CPU sys | 70.7s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 11.4 GiB |
| Avg memory | 4.3 GiB |
| CPU user | 1003.5s |
| CPU sys | 71.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5477388220
> So perhaps a simpler alternative is to add EmitTo::Last.
This is an interesting idea -- is your idea to keep batches managed by blocks internally, but not change the group_index API? And then EmitTo::Last ? Or were you imagining coping the EmitTo::Last rows to the output from the inner Vec allocation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2917518553
Thanks @adriangb @Dandandan .
I just start my new job this week and a bit busy, and I will continue to push it forward this weekend.
The new targets for this one may be
- Mainly as a base of better aggregation spilling
- Slightly improve performance
So the rest works I think:
- File an issue about the new aggregation spilling proposal based on this one
- Support `block mode` for `NullState` again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890398419
@alamb @Dandandan It is sorry that, I still working on other some new ideas about this pr, so switch to draft again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alchemist51 (via GitHub)" <gi...@apache.org>.
alchemist51 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3830440962
Hi @Rachelint any update on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4195598208
> > This branch still have conflicts.
>
> Yes, mainly solved the correctness0 problems in `row_hash.rs` and `accumulate.rs`. Will clean up codes and solve other small problems today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276111051
> These look like some regression.
OKοΌwill inspect them after some final small refactors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4738872559
> Marking as a draft as I don't think this one is ready to merge quite yet and I am trying to clean up the review / merge queue
Yes, and I think the whole feature will be suitable to push forward after the aggregation refactoring.
Howerver, actually to parts are included in this:
- One part is about refactoring `GroupValues` and `GroupAccumulator`
- The other part is about applying the blocked logic in aggreagating
How about we split this pr into twos or mores?
And push forward the part one (`GroupValues` and `GroupAccumulator`) in parallel with the aggregation refactoring?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072414793
##########
benchmarks/queries/clickbench/extended.sql:
##########
@@ -5,3 +5,4 @@ SELECT "SocialSourceNetworkID", "RegionID", COUNT(*), AVG("Age"), AVG("ParamPric
SELECT "ClientIP", "WatchID", COUNT(*) c, MIN("ResponseStartTiming") tmin, MEDIAN("ResponseStartTiming") tmed, MAX("ResponseStartTiming") tmax FROM hits WHERE "JavaEnable" = 0 GROUP BY "ClientIP", "WatchID" HAVING c > 1 ORDER BY tmed DESC LIMIT 10;
SELECT "ClientIP", "WatchID", COUNT(*) c, MIN("ResponseStartTiming") tmin, APPROX_PERCENTILE_CONT("ResponseStartTiming", 0.95) tp95, MAX("ResponseStartTiming") tmax FROM 'hits' WHERE "JavaEnable" = 0 GROUP BY "ClientIP", "WatchID" HAVING c > 1 ORDER BY tp95 DESC LIMIT 10;
SELECT COUNT(*) AS ShareCount FROM hits WHERE "IsMobile" = 1 AND "MobilePhoneModel" LIKE 'iPhone%' AND "SocialAction" = 'share' AND "SocialSourceNetworkID" IN (5, 12) AND "ClientTimeZone" BETWEEN -5 AND 5 AND regexp_match("Referer", '\/campaign\/(spring|summer)_promo') IS NOT NULL AND CASE WHEN split_part(split_part("URL", 'resolution=', 2), '&', 1) ~ '^\d+$' THEN split_part(split_part("URL", 'resolution=', 2), '&', 1)::INT ELSE 0 END > 1920 AND levenshtein(CAST("UTMSource" AS STRING), CAST("UTMCampaign" AS STRING)) < 3;
+SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Review Comment:
Addressed with #15936
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848551270
> > π€: Benchmark completed
>
> (I am surprised this PR didn't yield better results, I am reruning now to see if the results are reproducable
I am back from holiday, and continue to work on this today.
Because for simplicity for the initial pr, I just:
- Implement the blocked version `GroupValuesPrimitive` for `GroupValues`
- and `PrimitiveGroupsAccumulator` for `GroupAccumulator`
And actually still don't have such a simple query can show the improvement now...
And for showing the improvement, I add an new query in `clickbench/extended.sql`:
```sql
SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2887661504
π€ `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing intermeidate-result-blocked-approach (7869779cc59494baa88fb94b096c995eabd87535) to 07fe23f03dcb53b2715d3f3b3e0c476d54202c00 [diff](https://github.com/apache/datafusion/compare/07fe23f03dcb53b2715d3f3b3e0c476d54202c00..7869779cc59494baa88fb94b096c995eabd87535)
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2854200337
π€ `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing intermeidate-result-blocked-approach (62157a9ab896014c17927c7ae97d2177d20e9ec6) to 6cc49533f9c1c60c1737b7139d867b6544caa993 [diff](https://github.com/apache/datafusion/compare/6cc49533f9c1c60c1737b7139d867b6544caa993..62157a9ab896014c17927c7ae97d2177d20e9ec6)
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2854413639
Marking as draft as @Rachelint works on the next set of things
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ahmed-mez (via GitHub)" <gi...@apache.org>.
ahmed-mez commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4133351121
Hi @Rachelint π just wanted to check in on this! The last commit was about a month ago, any update on where things stand? Also worth noting that this work could help fix or mitigate #19906, so there's renewed interest in getting it over the line.
Thanks for all the effort you've put into this. It's really appreciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4335050316
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4335018412)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4335018412-1865-d62wc 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (a818cd8923a8b19ce6befad53e0b052aa47be9bf) to 22bb4e6 (merge-base) [diff](https://github.com/apache/datafusion/compare/22bb4e6b752c7a62b677d94a63bcf08b68e8d5ec..a818cd8923a8b19ce6befad53e0b052aa47be9bf) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4335192214
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4335018412)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 7.01 / 7.51 Β±0.83 / 9.16 ms β 6.97 / 7.43 Β±0.80 / 9.02 ms β no change β
β QQuery 2 β 146.03 / 146.75 Β±0.56 / 147.75 ms β 149.58 / 149.91 Β±0.24 / 150.32 ms β no change β
β QQuery 3 β 113.71 / 114.62 Β±0.73 / 115.67 ms β 114.49 / 115.04 Β±0.61 / 115.95 ms β no change β
β QQuery 4 β 1320.29 / 1328.53 Β±5.14 / 1335.27 ms β 1293.09 / 1302.21 Β±6.87 / 1312.53 ms β no change β
β QQuery 5 β 172.47 / 174.00 Β±1.21 / 175.96 ms β 175.76 / 176.60 Β±1.20 / 178.98 ms β no change β
β QQuery 6 β 142.02 / 144.92 Β±3.01 / 148.60 ms β 135.06 / 137.22 Β±1.47 / 138.97 ms β +1.06x faster β
β QQuery 7 β 335.95 / 337.13 Β±1.06 / 338.50 ms β 338.12 / 340.12 Β±1.55 / 342.53 ms β no change β
β QQuery 8 β 114.08 / 115.03 Β±0.61 / 115.97 ms β 114.34 / 115.20 Β±0.71 / 116.48 ms β no change β
β QQuery 9 β 96.41 / 101.14 Β±2.42 / 103.21 ms β 97.70 / 103.88 Β±5.33 / 112.35 ms β no change β
β QQuery 10 β 102.58 / 102.88 Β±0.22 / 103.25 ms β 103.94 / 104.66 Β±0.69 / 105.57 ms β no change β
β QQuery 11 β 928.89 / 934.32 Β±5.24 / 944.08 ms β 915.01 / 923.02 Β±6.61 / 933.42 ms β no change β
β QQuery 12 β 43.37 / 43.66 Β±0.25 / 44.07 ms β 43.62 / 43.96 Β±0.31 / 44.43 ms β no change β
β QQuery 13 β 389.52 / 391.27 Β±1.70 / 393.96 ms β 390.28 / 392.62 Β±1.85 / 395.57 ms β no change β
β QQuery 14 β 1036.94 / 1047.09 Β±8.48 / 1056.90 ms β 1040.20 / 1042.69 Β±2.62 / 1046.02 ms β no change β
β QQuery 15 β 15.09 / 15.26 Β±0.15 / 15.54 ms β 14.98 / 15.24 Β±0.24 / 15.57 ms β no change β
β QQuery 16 β 7.47 / 9.32 Β±3.34 / 15.99 ms β 7.56 / 7.72 Β±0.21 / 8.14 ms β +1.21x faster β
β QQuery 17 β 224.18 / 225.51 Β±0.92 / 226.59 ms β 222.56 / 224.13 Β±0.95 / 225.47 ms β no change β
β QQuery 18 β 127.38 / 128.40 Β±0.57 / 128.95 ms β 126.84 / 128.08 Β±1.20 / 130.34 ms β no change β
β QQuery 19 β 156.88 / 157.80 Β±1.26 / 160.27 ms β 155.69 / 156.47 Β±0.79 / 157.46 ms β no change β
β QQuery 20 β 13.26 / 13.60 Β±0.20 / 13.88 ms β 13.26 / 13.50 Β±0.21 / 13.84 ms β no change β
β QQuery 21 β 19.76 / 19.96 Β±0.20 / 20.30 ms β 19.21 / 19.55 Β±0.23 / 19.87 ms β no change β
β QQuery 22 β 485.67 / 488.76 Β±2.48 / 492.60 ms β 479.46 / 486.46 Β±5.04 / 491.67 ms β no change β
β QQuery 23 β 1086.04 / 1101.64 Β±11.47 / 1116.62 ms β 1099.98 / 1108.35 Β±6.37 / 1115.87 ms β no change β
β QQuery 24 β 713.85 / 717.23 Β±1.90 / 719.53 ms β 713.00 / 717.39 Β±3.14 / 720.66 ms β no change β
β QQuery 25 β 334.21 / 335.63 Β±1.42 / 338.11 ms β 334.46 / 336.95 Β±2.09 / 340.74 ms β no change β
β QQuery 26 β 77.54 / 79.70 Β±3.62 / 86.93 ms β 77.61 / 78.96 Β±1.98 / 82.85 ms β no change β
β QQuery 27 β 7.01 / 7.17 Β±0.17 / 7.50 ms β 7.00 / 7.11 Β±0.18 / 7.47 ms β no change β
β QQuery 28 β 148.75 / 151.22 Β±3.90 / 158.96 ms β 147.96 / 149.65 Β±0.95 / 150.77 ms β no change β
β QQuery 29 β 271.71 / 274.31 Β±2.21 / 278.18 ms β 274.00 / 277.74 Β±4.18 / 283.98 ms β no change β
β QQuery 30 β 42.07 / 43.23 Β±0.81 / 44.31 ms β 41.95 / 42.36 Β±0.31 / 42.72 ms β no change β
β QQuery 31 β 166.48 / 169.74 Β±3.76 / 177.11 ms β 165.94 / 168.50 Β±4.05 / 176.58 ms β no change β
β QQuery 32 β 13.51 / 13.77 Β±0.22 / 14.14 ms β 13.38 / 13.80 Β±0.28 / 14.22 ms β no change β
β QQuery 33 β 139.21 / 140.40 Β±1.59 / 143.54 ms β 139.88 / 140.79 Β±0.60 / 141.54 ms β no change β
β QQuery 34 β 6.98 / 7.21 Β±0.29 / 7.78 ms β 7.03 / 7.16 Β±0.20 / 7.57 ms β no change β
β QQuery 35 β 99.93 / 101.74 Β±1.12 / 103.44 ms β 100.33 / 102.39 Β±1.16 / 103.65 ms β no change β
β QQuery 36 β 6.71 / 6.92 Β±0.15 / 7.11 ms β 6.55 / 6.98 Β±0.33 / 7.54 ms β no change β
β QQuery 37 β 8.35 / 8.43 Β±0.08 / 8.53 ms β 8.23 / 8.37 Β±0.08 / 8.45 ms β no change β
β QQuery 38 β 87.36 / 92.94 Β±6.28 / 105.18 ms β 86.85 / 88.54 Β±1.66 / 91.38 ms β no change β
β QQuery 39 β 117.84 / 118.68 Β±0.52 / 119.33 ms β 116.65 / 117.89 Β±1.00 / 119.40 ms β no change β
β QQuery 40 β 103.03 / 107.95 Β±6.04 / 119.83 ms β 102.30 / 105.53 Β±2.43 / 109.63 ms β no change β
β QQuery 41 β 14.82 / 15.06 Β±0.16 / 15.28 ms β 14.70 / 14.81 Β±0.10 / 14.99 ms β no change β
β QQuery 42 β 108.13 / 110.21 Β±3.17 / 116.51 ms β 108.19 / 109.26 Β±1.98 / 113.21 ms β no change β
β QQuery 43 β 5.79 / 5.90 Β±0.16 / 6.21 ms β 5.77 / 5.88 Β±0.17 / 6.21 ms β no change β
β QQuery 44 β 10.94 / 11.71 Β±1.02 / 13.73 ms β 10.84 / 10.99 Β±0.13 / 11.22 ms β +1.07x faster β
β QQuery 45 β 48.35 / 48.97 Β±0.42 / 49.58 ms β 49.29 / 50.29 Β±1.56 / 53.39 ms β no change β
β QQuery 46 β 8.53 / 8.72 Β±0.23 / 9.08 ms β 8.48 / 8.75 Β±0.23 / 9.07 ms β no change β
β QQuery 47 β 708.88 / 718.15 Β±9.26 / 734.86 ms β 705.07 / 711.71 Β±4.44 / 717.12 ms β no change β
β QQuery 48 β 273.42 / 279.44 Β±5.45 / 289.65 ms β 275.43 / 279.45 Β±4.00 / 285.09 ms β no change β
β QQuery 49 β 249.37 / 250.36 Β±0.56 / 250.89 ms β 248.62 / 251.09 Β±1.77 / 253.05 ms β no change β
β QQuery 50 β 205.38 / 211.59 Β±5.78 / 222.10 ms β 205.12 / 207.51 Β±1.65 / 209.88 ms β no change β
β QQuery 51 β 176.05 / 178.44 Β±1.65 / 180.82 ms β 176.03 / 178.83 Β±2.03 / 182.22 ms β no change β
β QQuery 52 β 107.56 / 108.68 Β±2.00 / 112.68 ms β 107.68 / 108.71 Β±0.89 / 109.74 ms β no change β
β QQuery 53 β 102.57 / 103.84 Β±0.94 / 105.12 ms β 103.38 / 103.99 Β±0.88 / 105.70 ms β no change β
β QQuery 54 β 144.03 / 146.82 Β±2.47 / 151.42 ms β 145.95 / 147.38 Β±1.61 / 150.48 ms β no change β
β QQuery 55 β 106.16 / 106.79 Β±0.76 / 108.28 ms β 107.26 / 108.01 Β±0.63 / 108.83 ms β no change β
β QQuery 56 β 139.80 / 141.21 Β±1.66 / 144.46 ms β 139.12 / 141.38 Β±1.39 / 143.47 ms β no change β
β QQuery 57 β 165.18 / 168.79 Β±2.60 / 172.69 ms β 164.41 / 166.79 Β±1.92 / 168.69 ms β no change β
β QQuery 58 β 265.45 / 266.53 Β±1.05 / 268.41 ms β 265.33 / 266.48 Β±0.96 / 268.03 ms β no change β
β QQuery 59 β 196.01 / 197.37 Β±1.24 / 199.32 ms β 195.12 / 196.74 Β±1.69 / 199.92 ms β no change β
β QQuery 60 β 141.53 / 143.33 Β±1.64 / 145.76 ms β 140.67 / 142.24 Β±1.25 / 144.34 ms β no change β
β QQuery 61 β 13.80 / 13.89 Β±0.11 / 14.11 ms β 13.38 / 13.55 Β±0.22 / 13.98 ms β no change β
β QQuery 62 β 871.60 / 875.44 Β±3.20 / 880.66 ms β 870.85 / 876.43 Β±3.69 / 882.42 ms β no change β
β QQuery 63 β 103.07 / 105.73 Β±2.73 / 109.06 ms β 103.95 / 106.04 Β±3.00 / 111.98 ms β no change β
β QQuery 64 β 660.02 / 666.44 Β±7.03 / 678.93 ms β 657.00 / 663.23 Β±4.26 / 668.51 ms β no change β
β QQuery 65 β 248.22 / 251.28 Β±3.78 / 258.65 ms β 245.17 / 248.04 Β±3.08 / 253.55 ms β no change β
β QQuery 66 β 214.87 / 219.12 Β±7.18 / 233.39 ms β 213.46 / 218.04 Β±6.17 / 230.11 ms β no change β
β QQuery 67 β 294.52 / 307.49 Β±15.48 / 335.82 ms β 292.83 / 299.79 Β±5.77 / 307.03 ms β no change β
β QQuery 68 β 8.50 / 8.77 Β±0.31 / 9.35 ms β 9.22 / 10.78 Β±2.80 / 16.38 ms β 1.23x slower β
β QQuery 69 β 96.78 / 98.93 Β±2.46 / 103.69 ms β 97.32 / 98.69 Β±0.99 / 100.28 ms β no change β
β QQuery 70 β 309.63 / 328.13 Β±16.87 / 357.37 ms β 314.80 / 325.46 Β±7.17 / 334.97 ms β no change β
β QQuery 71 β 133.23 / 136.15 Β±2.62 / 139.87 ms β 133.94 / 136.13 Β±2.35 / 140.24 ms β no change β
β QQuery 72 β 579.07 / 590.47 Β±10.12 / 604.94 ms β 589.66 / 604.99 Β±12.16 / 623.37 ms β no change β
β QQuery 73 β 6.83 / 6.92 Β±0.16 / 7.24 ms β 6.83 / 7.05 Β±0.20 / 7.42 ms β no change β
β QQuery 74 β 584.99 / 587.64 Β±2.08 / 590.24 ms β 557.08 / 599.99 Β±32.30 / 642.74 ms β no change β
β QQuery 75 β 268.04 / 270.56 Β±3.32 / 277.10 ms β 266.77 / 267.65 Β±0.49 / 268.22 ms β no change β
β QQuery 76 β 130.62 / 133.50 Β±3.23 / 139.74 ms β 129.21 / 131.81 Β±2.57 / 136.56 ms β no change β
β QQuery 77 β 187.12 / 189.29 Β±2.41 / 193.86 ms β 188.64 / 189.93 Β±1.27 / 192.36 ms β no change β
β QQuery 78 β 330.11 / 331.54 Β±1.48 / 333.75 ms β 327.62 / 330.34 Β±1.57 / 332.28 ms β no change β
β QQuery 79 β 232.21 / 236.10 Β±2.98 / 239.51 ms β 228.19 / 231.54 Β±4.17 / 239.22 ms β no change β
β QQuery 80 β 318.84 / 322.15 Β±2.90 / 327.36 ms β 319.01 / 322.14 Β±2.38 / 325.97 ms β no change β
β QQuery 81 β 26.04 / 26.39 Β±0.27 / 26.76 ms β 25.71 / 26.15 Β±0.38 / 26.72 ms β no change β
β QQuery 82 β 39.37 / 39.90 Β±0.44 / 40.42 ms β 38.94 / 39.10 Β±0.13 / 39.31 ms β no change β
β QQuery 83 β 37.32 / 38.74 Β±1.93 / 42.57 ms β 36.85 / 37.89 Β±1.53 / 40.93 ms β no change β
β QQuery 84 β 46.74 / 47.06 Β±0.21 / 47.31 ms β 46.28 / 46.62 Β±0.32 / 47.16 ms β no change β
β QQuery 85 β 141.56 / 143.20 Β±2.26 / 147.65 ms β 140.80 / 141.18 Β±0.36 / 141.72 ms β no change β
β QQuery 86 β 37.59 / 38.08 Β±0.34 / 38.66 ms β 37.42 / 38.37 Β±0.81 / 39.85 ms β no change β
β QQuery 87 β 3.56 / 3.69 Β±0.21 / 4.10 ms β 3.55 / 3.67 Β±0.15 / 3.96 ms β no change β
β QQuery 88 β 99.58 / 100.24 Β±0.59 / 101.01 ms β 99.83 / 100.54 Β±0.47 / 100.99 ms β no change β
β QQuery 89 β 117.19 / 121.00 Β±4.84 / 130.34 ms β 116.94 / 121.24 Β±6.83 / 134.79 ms β no change β
β QQuery 90 β 22.71 / 23.18 Β±0.39 / 23.69 ms β 22.18 / 22.64 Β±0.52 / 23.43 ms β no change β
β QQuery 91 β 58.68 / 59.53 Β±0.69 / 60.42 ms β 58.13 / 59.39 Β±0.96 / 60.55 ms β no change β
β QQuery 92 β 56.66 / 57.53 Β±0.96 / 59.30 ms β 57.17 / 57.70 Β±0.44 / 58.22 ms β no change β
β QQuery 93 β 180.23 / 181.45 Β±0.73 / 182.41 ms β 179.62 / 180.68 Β±1.47 / 183.58 ms β no change β
β QQuery 94 β 62.26 / 63.99 Β±3.26 / 70.51 ms β 60.73 / 61.86 Β±0.81 / 62.81 ms β no change β
β QQuery 95 β 126.59 / 127.36 Β±0.69 / 128.40 ms β 125.90 / 126.90 Β±0.87 / 128.44 ms β no change β
β QQuery 96 β 68.69 / 71.62 Β±1.64 / 73.72 ms β 68.92 / 70.80 Β±2.18 / 74.93 ms β no change β
β QQuery 97 β 116.96 / 118.53 Β±0.89 / 119.48 ms β 117.75 / 118.44 Β±0.48 / 119.12 ms β no change β
β QQuery 98 β 150.33 / 153.22 Β±2.05 / 156.27 ms β 150.22 / 152.64 Β±2.33 / 155.63 ms β no change β
β QQuery 99 β 10759.39 / 10914.99 Β±86.21 / 11011.33 ms β 10695.39 / 10826.26 Β±115.61 / 11010.63 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 30756.49ms β
β Total Time (intermeidate-result-blocked-approach) β 30608.12ms β
β Average Time (HEAD) β 310.67ms β
β Average Time (intermeidate-result-blocked-approach) β 309.17ms β
β Queries Faster β 3 β
β Queries Slower β 1 β
β Queries with No Change β 95 β
β Queries with Failure β 0 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.4 GiB |
| Avg memory | 5.7 GiB |
| CPU user | 242.9s |
| CPU sys | 8.5s |
| Peak spill | 0 B |
**tpcds β branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.3 GiB |
| Avg memory | 5.7 GiB |
| CPU user | 242.2s |
| CPU sys | 8.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4334374251
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4334353557)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4334353557-1861-gzv2h 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (a84d71cf9cc8161aca8398c329749d00137965d3) to bbf67d9 (merge-base) [diff](https://github.com/apache/datafusion/compare/bbf67d999c0dcbaf47956233a5d8e78458c13ffa..a84d71cf9cc8161aca8398c329749d00137965d3) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4343636635
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4343604373)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4343604373-1910-rgfv2 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (47b69765e44dbac7ec4173315b64ca1d55f259dc) to 8f033e4 (merge-base) [diff](https://github.com/apache/datafusion/compare/8f033e411aa4e097a146718e61d90897fe6e9c3d..47b69765e44dbac7ec4173315b64ca1d55f259dc) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5432645435
I wrote up a summary of what I think is going on here, and the challenges, and what we have done so far
- https://github.com/apache/datafusion/issues/24704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5419284871
> my implementation ditched the emit to first and emit all and instead only have emit next block, since it will simplify implementations to not have enums and a lot of generics like this PR have to support both
>
> The problem with that is that this is breaking change that cannot be done incrementally, like, all the helpers can be created in different pr, but changing to blocked implementation cannot be done incremently.
Based on my previous experiments, I agree with using only `EmitTo::NextBlock`, and I also think it would be challenging to roll this out incrementally.
I can participate asynchronously by reviewing PoC PRs. Given the complexity of the implementation, I think weβll need to look at the actual code to make meaningful progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2888248445
OK... I guess it is due to the computation of `block_id` and `block_offset`.
I found a machine with similar cores and profiling.
Then I found in the old and so good intel core, the `/` and `%` are really expansive.
I am trying to optimize it.
```rust
impl GroupIndexOperations for BlockedGroupIndexOperations {
fn get_block_id(group_index: usize, block_size: usize) -> usize {
group_index / block_size
}
fn get_block_offset(group_index: usize, block_size: usize) -> usize {
group_index % block_size
}
}
```
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3154598833
> It is unfortunate we never figured out how to get this over the line π’
>
> Thank you for the effort anyways @Rachelint
It is sorry... but actually I still want to continue push it forward... However it is too busy recent few months...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2095456466
##########
datafusion/functions-aggregate-common/benches/null_state_accumulate.rs:
##########
@@ -0,0 +1,202 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+extern crate criterion;
+
+use std::sync::Arc;
+
+use arrow::array::ArrowNativeTypeOp;
+use arrow::{
+ array::{ArrayRef, AsArray, BooleanArray, Int64Array},
+ datatypes::Int64Type,
+};
+use criterion::{criterion_group, criterion_main, Criterion};
+use datafusion_functions_aggregate_common::aggregate::groups_accumulator::group_index_operations::FlatGroupIndexOperations;
+use datafusion_functions_aggregate_common::aggregate::groups_accumulator::{
+ accumulate::{self, accumulate_indices, NullStateAdapter},
+ blocks::GeneralBlocks,
+ group_index_operations::{BlockedGroupIndexOperations, GroupIndexOperations},
+};
+
+fn generate_group_indices(len: usize) -> Vec<usize> {
+ (0..len).collect()
Review Comment:
Does using indices `0..len` mean this benchmark is testing the case where each input has a new unique group?
If this is the case, then I think it would make sense that the benchmark shows lots of time allocating / zeroing memory: the bulk of the work will be copying each value into a new accumulator slot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2852294444
π€ `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing intermeidate-result-blocked-approach (7542b49a2add7acae700de6e88a7cf8227cc6735) to 6cc49533f9c1c60c1737b7139d867b6544caa993 [diff](https://github.com/apache/datafusion/compare/6cc49533f9c1c60c1737b7139d867b6544caa993..7542b49a2add7acae700de6e88a7cf8227cc6735)
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2853164420
I suspect if it is due to the row-level random access of `VecDeque`?
I have replace `VecDeque` with `Vec`.
And I am trying to run the benchmark in more enviorments rather than only my dev machine (x86 + centos + 6core).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075218539
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,20 +94,27 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
Review Comment:
> Ah I see that batch size is used by default :). Yeah I think it makes sense to test it a bit further, maybe for this a slightly larger value (e.g. 2x, 4x batch size) will be beneficial when the cardinality is above the batch size.
>
> Also at some point might make sense to think of it in size in memory instead of number of elements (e.g. block of `u8` values might hold 16x more values than u128).
It is really a good idea to split block by size rather than num rows, I will experiment on it after trying some exist ideas later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2854237216
> FWIW the machine I am running this benchmark on has 16 not very good cores
OK, it may be really related to `target_partitions`.
I also found only slight improvement in my local when `target_partitions = 16` (default to cpu num).
The `target_partitions` is large, so `intermediate results Vec` in `partition` will be small, so `blocked approach` make few difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2863584067
The current benchmark results in my 16cores local:
- target_partitions = 8 (amazing 1.71x faster)
```
// main
Query 7 iteration 0 took 4256.6 ms and returned 10 rows
Query 7 iteration 1 took 4265.5 ms and returned 10 rows
Query 7 iteration 2 took 3822.5 ms and returned 10 rows
Query 7 iteration 3 took 4136.2 ms and returned 10 rows
Query 7 iteration 4 took 3851.7 ms and returned 10 rows
Query 7 avg time: 4066.49 ms
// blocked
Query 7 iteration 0 took 2278.9 ms and returned 10 rows
Query 7 iteration 1 took 2123.4 ms and returned 10 rows
Query 7 iteration 2 took 2400.0 ms and returned 10 rows
Query 7 iteration 3 took 2315.6 ms and returned 10 rows
Query 7 iteration 4 took 2716.3 ms and returned 10 rows
Query 7 avg time: 2366.87 ms
```
- target_partitions = 16(also 1.35x faster)
```
// main
Query 7 iteration 0 took 1955.1 ms and returned 10 rows
Query 7 iteration 1 took 2002.4 ms and returned 10 rows
Query 7 iteration 2 took 1762.5 ms and returned 10 rows
Query 7 iteration 3 took 1805.2 ms and returned 10 rows
Query 7 iteration 4 took 1758.7 ms and returned 10 rows
Query 7 avg time: 1856.78 ms
// blocked
Query 7 iteration 0 took 1392.8 ms and returned 10 rows
Query 7 iteration 1 took 1357.3 ms and returned 10 rows
Query 7 iteration 2 took 1378.4 ms and returned 10 rows
Query 7 iteration 3 took 1376.6 ms and returned 10 rows
Query 7 iteration 4 took 1367.1 ms and returned 10 rows
Query 7 avg time: 1374.47 ms
```
- target_partitions = 32(still 1.10x faster)
```
// main
Query 7 iteration 0 took 1471.3 ms and returned 10 rows
Query 7 iteration 1 took 1350.5 ms and returned 10 rows
Query 7 iteration 2 took 1361.4 ms and returned 10 rows
Query 7 iteration 3 took 1393.6 ms and returned 10 rows
Query 7 iteration 4 took 1427.4 ms and returned 10 rows
Query 7 avg time: 1400.82 ms
// blocked
Query 7 iteration 0 took 1289.6 ms and returned 10 rows
Query 7 iteration 1 took 1281.9 ms and returned 10 rows
Query 7 iteration 2 took 1293.4 ms and returned 10 rows
Query 7 iteration 3 took 1258.8 ms and returned 10 rows
Query 7 iteration 4 took 1279.4 ms and returned 10 rows
Query 7 avg time: 1280.62 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2847988587
π€ `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing intermeidate-result-blocked-approach (ff9c3adc4523e7df83ccccee635679926f083dda) to 74dc4196858784d7872b21bbfc97edc564e47c5e [diff](https://github.com/apache/datafusion/compare/74dc4196858784d7872b21bbfc97edc564e47c5e..ff9c3adc4523e7df83ccccee635679926f083dda)
Benchmarks: clickbench_1 tpch_mem
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848559813
> I am back from holiday, and continue to work on this today.
Welcome back!
> Emmm... As expected the new added query is not convered, I think I should submit an new pr for adding the query to extended.sql for checking the improvement... Just wait a minute.
Makes sense
> And actually still don't have such a simple query can show the improvement now...
What would be required to improve the performance for one or more of the real clickbench queries? Implementing group management for other data types?
Given we have past evidence this approach will work I think we could merge it before the queries really sped up. My big concern is that we have a plan to eventually avoid both single and multi blocked management
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4335018412
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4334377486
Benchmark for [this request](https://github.com/apache/datafusion/pull/15591#issuecomment-4334353557) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
106 | let num_emitted = match emit_to {
| ^^^^^^^ pattern `EmitTo::NextBlock` not covered
|
note: `EmitTo` defined here
--> datafusion/expr-common/src/groups_accumulator.rs:25:1
|
25 | pub enum EmitTo {
| ^^^^^^^^^^^^^^^
...
37 | NextBlock,
| --------- not covered
= note: the matched value is of type `EmitTo`
help: ensure that all possible cases are being handled by adding a match arm with a wildcard pattern or an explicit pattern as shown
|
108 ~ EmitTo::First(n) => n,
109 ~ EmitTo::NextBlock => todo!(),
|
For more information about this error, try `rustc --explain E0004`.
error: could not compile `datafusion-functions-aggregate-common` (lib) due to 2 previous errors
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4334682132
I think it may be ready now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4335192244
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4335018412)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.21 / 4.69 Β±6.82 / 18.34 ms β 1.25 / 4.74 Β±6.87 / 18.48 ms β no change β
β QQuery 1 β 12.42 / 12.98 Β±0.31 / 13.28 ms β 13.04 / 13.41 Β±0.24 / 13.74 ms β no change β
β QQuery 2 β 36.77 / 37.00 Β±0.28 / 37.55 ms β 36.72 / 37.01 Β±0.21 / 37.27 ms β no change β
β QQuery 3 β 31.82 / 32.55 Β±0.61 / 33.29 ms β 31.66 / 32.03 Β±0.21 / 32.24 ms β no change β
β QQuery 4 β 246.13 / 252.06 Β±5.38 / 261.04 ms β 288.46 / 293.26 Β±2.85 / 296.74 ms β 1.16x slower β
β QQuery 5 β 287.55 / 289.80 Β±1.47 / 291.18 ms β 290.72 / 294.74 Β±2.80 / 298.27 ms β no change β
β QQuery 6 β 6.44 / 8.32 Β±2.41 / 13.07 ms β 6.69 / 7.41 Β±0.44 / 7.79 ms β +1.12x faster β
β QQuery 7 β 13.76 / 13.95 Β±0.13 / 14.16 ms β 14.49 / 14.70 Β±0.20 / 15.07 ms β 1.05x slower β
β QQuery 8 β 337.06 / 340.72 Β±2.75 / 344.90 ms β 339.33 / 345.93 Β±4.24 / 351.57 ms β no change β
β QQuery 9 β 467.45 / 471.13 Β±4.79 / 480.04 ms β 482.13 / 493.24 Β±9.53 / 509.78 ms β no change β
β QQuery 10 β 73.96 / 75.46 Β±0.77 / 76.18 ms β 75.11 / 75.84 Β±0.72 / 77.15 ms β no change β
β QQuery 11 β 85.74 / 86.49 Β±1.01 / 88.48 ms β 87.51 / 88.88 Β±2.41 / 93.69 ms β no change β
β QQuery 12 β 283.78 / 286.86 Β±3.08 / 291.09 ms β 284.25 / 292.10 Β±6.74 / 302.54 ms β no change β
β QQuery 13 β 406.46 / 420.25 Β±8.86 / 430.15 ms β 413.33 / 423.15 Β±8.11 / 432.17 ms β no change β
β QQuery 14 β 292.45 / 301.14 Β±7.98 / 316.27 ms β 295.63 / 299.22 Β±2.46 / 303.32 ms β no change β
β QQuery 15 β 305.37 / 309.09 Β±3.65 / 315.83 ms β 332.46 / 345.11 Β±9.11 / 360.85 ms β 1.12x slower β
β QQuery 16 β 648.86 / 651.03 Β±1.24 / 652.66 ms β 634.06 / 650.45 Β±12.92 / 673.27 ms β no change β
β QQuery 17 β 651.63 / 652.81 Β±0.80 / 653.90 ms β 645.18 / 655.54 Β±7.56 / 665.60 ms β no change β
β QQuery 18 β 1308.89 / 1318.44 Β±11.32 / 1338.43 ms β 1298.23 / 1322.43 Β±17.61 / 1342.60 ms β no change β
β QQuery 19 β 28.78 / 29.46 Β±0.74 / 30.71 ms β 29.42 / 31.13 Β±1.91 / 33.96 ms β 1.06x slower β
β QQuery 20 β 516.43 / 525.56 Β±6.74 / 534.53 ms β 520.66 / 531.41 Β±7.26 / 539.58 ms β no change β
β QQuery 21 β 597.45 / 603.19 Β±4.52 / 611.30 ms β 598.63 / 602.55 Β±3.35 / 607.56 ms β no change β
β QQuery 22 β 1069.91 / 1078.43 Β±6.11 / 1084.80 ms β 1071.85 / 1077.67 Β±4.60 / 1083.29 ms β no change β
β QQuery 23 β 3384.14 / 3413.94 Β±28.36 / 3459.00 ms β 3349.35 / 3377.43 Β±17.97 / 3404.89 ms β no change β
β QQuery 24 β 42.01 / 48.94 Β±7.81 / 62.88 ms β 41.90 / 42.85 Β±0.52 / 43.38 ms β +1.14x faster β
β QQuery 25 β 115.62 / 117.15 Β±1.29 / 118.94 ms β 115.26 / 116.65 Β±1.25 / 118.64 ms β no change β
β QQuery 26 β 42.93 / 44.05 Β±1.25 / 45.65 ms β 42.90 / 43.47 Β±0.39 / 43.99 ms β no change β
β QQuery 27 β 676.09 / 685.03 Β±7.26 / 696.46 ms β 682.12 / 687.49 Β±4.86 / 696.48 ms β no change β
β QQuery 28 β 3054.14 / 3069.82 Β±19.42 / 3106.66 ms β 3047.09 / 3062.80 Β±12.41 / 3081.02 ms β no change β
β QQuery 29 β 43.50 / 48.21 Β±5.58 / 55.92 ms β 43.09 / 48.72 Β±10.62 / 69.95 ms β no change β
β QQuery 30 β 318.52 / 321.71 Β±3.29 / 326.37 ms β 323.61 / 324.85 Β±1.09 / 326.49 ms β no change β
β QQuery 31 β 308.45 / 316.54 Β±5.71 / 325.25 ms β 313.78 / 319.24 Β±3.69 / 323.63 ms β no change β
β QQuery 32 β 1038.99 / 1049.02 Β±10.88 / 1068.44 ms β 1053.87 / 1068.04 Β±12.20 / 1087.34 ms β no change β
β QQuery 33 β 1486.15 / 1515.97 Β±24.06 / 1552.98 ms β 1481.92 / 1502.10 Β±23.35 / 1546.35 ms β no change β
β QQuery 34 β 1514.35 / 1522.62 Β±7.98 / 1535.04 ms β 1496.49 / 1506.89 Β±10.55 / 1526.49 ms β no change β
β QQuery 35 β 305.85 / 317.75 Β±17.70 / 352.89 ms β 345.22 / 355.32 Β±15.40 / 385.82 ms β 1.12x slower β
β QQuery 36 β 63.97 / 71.87 Β±6.22 / 79.36 ms β 63.14 / 70.24 Β±5.16 / 77.65 ms β no change β
β QQuery 37 β 36.62 / 39.88 Β±2.25 / 43.30 ms β 36.77 / 41.69 Β±3.79 / 47.37 ms β no change β
β QQuery 38 β 41.44 / 45.41 Β±4.60 / 54.23 ms β 40.78 / 42.38 Β±1.48 / 44.71 ms β +1.07x faster β
β QQuery 39 β 128.25 / 135.32 Β±5.73 / 142.96 ms β 127.25 / 138.93 Β±8.91 / 152.38 ms β no change β
β QQuery 40 β 15.33 / 19.61 Β±5.91 / 31.19 ms β 15.62 / 15.81 Β±0.13 / 15.98 ms β +1.24x faster β
β QQuery 41 β 14.70 / 14.77 Β±0.05 / 14.85 ms β 14.50 / 14.96 Β±0.38 / 15.52 ms β no change β
β QQuery 42 β 14.12 / 14.27 Β±0.15 / 14.54 ms β 14.75 / 14.83 Β±0.07 / 14.95 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 20613.32ms β
β Total Time (intermeidate-result-blocked-approach) β 20726.64ms β
β Average Time (HEAD) β 479.38ms β
β Average Time (intermeidate-result-blocked-approach) β 482.01ms β
β Queries Faster β 4 β
β Queries Slower β 5 β
β Queries with No Change β 34 β
β Queries with Failure β 0 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.0 GiB |
| Avg memory | 23.1 GiB |
| CPU user | 1091.5s |
| CPU sys | 64.8s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 30.2 GiB |
| Avg memory | 23.1 GiB |
| CPU user | 1097.9s |
| CPU sys | 65.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4757509816
We can run benchmarks with memory limits to force spilling if that helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5427843909
My few struggles I encounter with implementing blocked agg and accumulators
you need to create a lot of helpers, and maybe recreate some from scratch to store data in blocks as well - for example `ArrowBytesMap`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2093618262
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
@@ -339,6 +344,35 @@ impl SkipAggregationProbe {
/// β 2 β 2 β 3.0 β β 2 β 2 β 3.0 β ββββββββββββββ
/// βββββββββββββββββββ βββββββββββββββββββ
/// ```
+///
+/// # Blocked approach for intermediate results
+///
+/// An important optimization for [`group_values`] and [`accumulators`]
+/// is to manage such intermediate results using the blocked approach.
+///
+/// In the original method, intermediate results are managed within a single large block
Review Comment:
Is it possible (not in this PR) to eventually remove the original, single block approach?
##########
datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs:
##########
@@ -205,20 +284,462 @@ where
Some(_) => self.null_group.take(),
None => None,
};
- let mut split = self.values.split_off(n);
- std::mem::swap(&mut self.values, &mut split);
+
+ let single_block = self.values.last_mut().unwrap();
+ let mut split = single_block.split_off(n);
+ mem::swap(single_block, &mut split);
build_primitive(split, null_group)
}
+
+ // ===============================================
+ // Emitting in blocked mode
+ // ===============================================
+ EmitTo::NextBlock => {
+ let (total_num_groups, block_size) = if !self.is_emitting() {
+ // Similar as `EmitTo:All`, we will clear the old index infos(like `map`)
Review Comment:
is this still a todo item?
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
@@ -639,6 +686,53 @@ pub(crate) fn create_group_accumulator(
}
}
+/// Check if we can enable the blocked optimization for `GroupValues` and `GroupsAccumulator`s.
+/// The blocked optimization will be enabled when:
+/// - When `enable_aggregation_blocked_groups` is true(default to true)
+/// - It is not streaming aggregation(because blocked mode can't support Emit::first(exact n))
+/// - The spilling is disabled(still need to consider more to support it efficiently)
+/// - The accumulator is not empty(I am still not sure about logic in this case)
+/// - [`GroupValues::supports_blocked_groups`] and all [`GroupsAccumulator::supports_blocked_groups`] are true
+///
+/// [`GroupValues::supports_blocked_groups`]: crate::aggregates::group_values::GroupValues::supports_blocked_groups
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr::GroupsAccumulator::supports_blocked_groups
+///
+// TODO: support blocked optimization in streaming, spilling, and maybe empty accumulators case?
Review Comment:
yes, I think the goal should be to support blocked optimizations in all these cases (so we can remove the existing code)
I can file tickets to help organize this work (obviously you don't have to do so)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2888271159
> OK... I guess it is due to the computation of `block_id` and `block_offset`.
>
> I found a machine with similar inteal cores like the testing machine and profiled.
>
> Then I found in the old and not so good intel core, the `/` and `%` are really expansive.
>
> I am trying to optimize it, but still not sure if it the most important bottleneck
>
> ```rust
> impl GroupIndexOperations for BlockedGroupIndexOperations {
> fn get_block_id(group_index: usize, block_size: usize) -> usize {
> group_index / block_size
> }
>
> fn get_block_offset(group_index: usize, block_size: usize) -> usize {
> group_index % block_size
> }
> }
> ```
we could enforce blocksize being a power of two so we can avoid the expensive div and mod operations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2888829262
My plan is to adjust the `block size`, make `block size = N * batch size`.
We allocate larger memory in once, also make cpu easier to prefetch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890527834
π€ `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing intermeidate-result-blocked-approach (600318f348908f3aa3a80ddc3c8cb6d9c76deabc) to 07fe23f03dcb53b2715d3f3b3e0c476d54202c00 [diff](https://github.com/apache/datafusion/compare/07fe23f03dcb53b2715d3f3b3e0c476d54202c00..600318f348908f3aa3a80ddc3c8cb6d9c76deabc)
Benchmarks: clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2945391975
thanks @Rachelint and congratulations!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072895547
##########
datafusion/functions-aggregate/src/correlation.rs:
##########
@@ -448,6 +448,9 @@ impl GroupsAccumulator for CorrelationGroupsAccumulator {
let n = match emit_to {
EmitTo::All => self.count.len(),
EmitTo::First(n) => n,
+ EmitTo::NextBlock => {
+ unreachable!("this accumulator still not support blocked groups")
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848409762
how is the benchmark triggered and can we run clickbench extended too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4146909778
> Hi @Rachelint π just wanted to check in on this! The last commit was about a month ago, any update on where things stand? Also worth noting that this work could help fix or mitigate #19906, so there's renewed interest in getting it over the line.
>
> Thanks for all the effort you've put into this. It's really appreciated!
Sorry for long delay for some private reasons, will try to make it ready this weekend:
- already fixed bugs in accumulate
- I am porting this pr in the new spilling logic in `row_hash.rs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4275998435
Hi @Rachelint, thanks for the request (https://github.com/apache/datafusion/pull/15591#issuecomment-4275998373). Only whitelisted users can trigger benchmarks. Allowed users: [Dandandan](https://github.com/Dandandan), [Fokko](https://github.com/Fokko), [Jefffrey](https://github.com/Jefffrey), [Omega359](https://github.com/Omega359), [adriangb](https://github.com/adriangb), [alamb](https://github.com/alamb), [asubiotto](https://github.com/asubiotto), [brunal](https://github.com/brunal), [buraksenn](https://github.com/buraksenn), [cetra3](https://github.com/cetra3), [codephage2020](https://github.com/codephage2020), [comphead](https://github.com/comphead), [erenavsarogullari](https://github.com/erenavsarogullari), [etseidl](https://github.com/etseidl), [friendlymatthew](https://github.com/friendlymatthew), [gabotechs](https://github.com/gabotechs), [geoffreyclaude](https://github.com/geoffreyclaude), [grtlr](https://github.com/grtlr), [haohuaijin](https://github.com/haohuaijin), [jo
nathanc-n](https://github.com/jonathanc-n), [kevinjqliu](https://github.com/kevinjqliu), [klion26](https://github.com/klion26), [kosiew](https://github.com/kosiew), [kumarUjjawal](https://github.com/kumarUjjawal), [kunalsinghdadhwal](https://github.com/kunalsinghdadhwal), [liamzwbao](https://github.com/liamzwbao), [mbutrovich](https://github.com/mbutrovich), [mkleen](https://github.com/mkleen), [mzabaluev](https://github.com/mzabaluev), [neilconway](https://github.com/neilconway), [rluvaton](https://github.com/rluvaton), [sdf-jkl](https://github.com/sdf-jkl), [timsaucer](https://github.com/timsaucer), [xudong963](https://github.com/xudong963), [zhuqi-lucas](https://github.com/zhuqi-lucas).
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276096427
Very nice, I think we're getting closer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4335040166
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4335018412)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4335018412-1864-w5tw2 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (a818cd8923a8b19ce6befad53e0b052aa47be9bf) to 22bb4e6 (merge-base) [diff](https://github.com/apache/datafusion/compare/22bb4e6b752c7a62b677d94a63bcf08b68e8d5ec..a818cd8923a8b19ce6befad53e0b052aa47be9bf) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4341852652
@Dandandan hi, can help trigger the benchmark again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4343604373
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4334377709
Benchmark for [this request](https://github.com/apache/datafusion/pull/15591#issuecomment-4334353557) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
106 | let num_emitted = match emit_to {
| ^^^^^^^ pattern `EmitTo::NextBlock` not covered
|
note: `EmitTo` defined here
--> datafusion/expr-common/src/groups_accumulator.rs:25:1
|
25 | pub enum EmitTo {
| ^^^^^^^^^^^^^^^
...
37 | NextBlock,
| --------- not covered
= note: the matched value is of type `EmitTo`
help: ensure that all possible cases are being handled by adding a match arm with a wildcard pattern or an explicit pattern as shown
|
108 ~ EmitTo::First(n) => n,
109 ~ EmitTo::NextBlock => todo!(),
|
For more information about this error, try `rustc --explain E0004`.
error: could not compile `datafusion-functions-aggregate-common` (lib) due to 2 previous errors
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3797570070
alreay resolve most conflicts(mainly `row_hash.rs` and `accumulate.rs`) and pass tests, near to be ready again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5369598027
> > In the very hot loop of doing aggregation and jumping in memory you are adding another memory jump
>
> I think this is the key observation -- I think there is no way we can add the extra indirection and not slow down performance for small aggregates.
>
> For higher cardinality, I think it may not be as bad as we are going to be doing lots of random memory acesses anyways as the table doesn't fit in the processor caches.
>
> So I am guessing what we will need is some way to use direct indexing for small hash tables, but when the table grows above a certain size switch to 2 part indexes
>
> But that is just a theory and I have not tested it yet
Yeah I think this should be possible -- have two paths, direct indexing (for a single block under the threshold) / the block-based approach that is chosen per batch.
I think some other things that could be tried:
* Chose block-size large enough (1-4MB or so?) to increase locality / reduce cache misses
* Make sure the heave methods are inlined / can be auto-vectorized
* Make the block size / calculation a `const` so it can be precomputed/optimized (instead of computing it based on config).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5463126272
Actually, the main goal of the blocked approach is to release aggregation state progressively while emitting output batches, instead of retaining all state until all batches have been produced.
We could originally use EmitTo::First for this, but it is too expensive because removing the first groups requires shifting/copying the remaining state and renumbering group indices. We then explored organizing the state into blocks in this PR, but that introduces substantial implementation complexity.
So perhaps a simpler alternative is to add EmitTo::Last. Removing the last groups would not require shifting the remaining state or changing its group indices, while still allowing us to release state progressively during final output.
One caveat is that reducing Rust-visible Vec capacity does not necessarily mean RSS is released immediately. That depends on the allocator: jemalloc may split and purge the trailing pages of a large allocation, while mimalloc may retain the underlying allocation until it becomes significantly smaller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Impl intermeidate result blocked approach sketch [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2829974829
This one is still blocked on following prs:
- [ ] #15830
- [ ] #15851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermeidate result blocked approach sketch [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2834179862
This pr may be ready to review now, it is a formal version of #11943 (have proved the idea promising, 10%~20% faster) with some improvements:
- Almost don't lead any regression when `blocked groups approach` is disabled (#11943 will lead slight regression due to the row level index parsing even for `flat index`)
- Better testing with the aggreagation fuzzer (I think the testing is enough to ensure correctness, so I currently set the optimization enabled by default)
- Reusing more codes
cc @Dandandan @alamb @jayzhan211 @2010YOUY01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Impl intermeidate result blocked approach sketch [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2823851220
Bench result of clickbench, it can see no regression when `blocked_groups` disabled (still disable in all queries now due to only support blocked in very few `group values` and `accumulator`).
```
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β main β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 0.63ms β 0.58ms β +1.09x faster β
β QQuery 1 β 75.15ms β 77.55ms β no change β
β QQuery 2 β 157.19ms β 157.03ms β no change β
β QQuery 3 β 177.31ms β 171.35ms β no change β
β QQuery 4 β 1531.13ms β 1533.92ms β no change β
β QQuery 5 β 1329.11ms β 1323.23ms β no change β
β QQuery 6 β 0.64ms β 0.66ms β no change β
β QQuery 7 β 86.57ms β 83.64ms β no change β
β QQuery 8 β 1628.14ms β 1655.81ms β no change β
β QQuery 9 β 1803.07ms β 1889.13ms β no change β
β QQuery 10 β 500.49ms β 491.90ms β no change β
β QQuery 11 β 548.08ms β 553.46ms β no change β
β QQuery 12 β 1504.42ms β 1524.19ms β no change β
β QQuery 13 β 2483.15ms β 2523.10ms β no change β
β QQuery 14 β 1548.59ms β 1570.04ms β no change β
β QQuery 15 β 1791.77ms β 1836.97ms β no change β
β QQuery 16 β 3288.87ms β 3334.03ms β no change β
β QQuery 17 β 3099.75ms β 3114.60ms β no change β
β QQuery 18 β 7042.36ms β 7067.90ms β no change β
β QQuery 19 β 152.89ms β 151.47ms β no change β
β QQuery 20 β 2598.19ms β 2628.23ms β no change β
β QQuery 21 β 3176.48ms β 3183.66ms β no change β
β QQuery 22 β 5965.76ms β 5923.81ms β no change β
β QQuery 23 β 20526.87ms β 20259.01ms β no change β
β QQuery 24 β 894.85ms β 904.43ms β no change β
β QQuery 25 β 727.93ms β 748.53ms β no change β
β QQuery 26 β 1053.03ms β 1050.49ms β no change β
β QQuery 27 β 3689.54ms β 3780.86ms β no change β
β QQuery 28 β 22522.83ms β 22292.86ms β no change β
β QQuery 29 β 903.40ms β 907.97ms β no change β
β QQuery 30 β 1535.05ms β 1553.24ms β no change β
β QQuery 31 β 1756.57ms β 1786.77ms β no change β
β QQuery 32 β 7871.46ms β 7820.37ms β no change β
β QQuery 33 β 7257.54ms β 7342.05ms β no change β
β QQuery 34 β 7327.72ms β 7357.44ms β no change β
β QQuery 35 β 2186.67ms β 2208.35ms β no change β
β QQuery 36 β 140.08ms β 138.37ms β no change β
β QQuery 37 β 80.67ms β 81.03ms β no change β
β QQuery 38 β 139.33ms β 136.52ms β no change β
β QQuery 39 β 209.62ms β 214.95ms β no change β
β QQuery 40 β 65.74ms β 64.55ms β no change β
β QQuery 41 β 61.53ms β 64.67ms β 1.05x slower β
β QQuery 42 β 55.64ms β 58.14ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848554248
> how is the benchmark triggered and can we run clickbench extended too?
>
> upd: I didn't find improvement for extended query locally
Also for the new added one?
```
SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072857296
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -43,8 +46,8 @@ where
T: ArrowPrimitiveType + Send,
F: Fn(&mut T::Native, T::Native) + Send + Sync,
{
- /// values per group, stored as the native type
- values: Vec<T::Native>,
+ /// Values per group, stored as the native type
Review Comment:
Addressed.
I have defined `Blocks` in a dedicated module `blocks.rs`, and simplify all related codes using it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2852827157
> It is really suprised that it show slower?
This is my only concern too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075154369
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,20 +94,27 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
Review Comment:
> yes, i mean block size. I would expect something like batchsize (4-8k), or maybe even bigger to have lower overhead? Did you run some experiments?
Yes, I try it.
Now I set `block_size = batch_size`.
I try the smaller `batch_size` like 1024, and `this pr` show improvement compared to `main`.
It is due to `this pr` can also eliminating the call of `Array::slice`, which is non-trivial due to the computation of null count. Detail can see: https://github.com/apache/arrow-rs/pull/6155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075240973
##########
datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs:
##########
@@ -81,28 +85,52 @@ hash_float!(f16, f32, f64);
pub struct GroupValuesPrimitive<T: ArrowPrimitiveType> {
/// The data type of the output array
data_type: DataType,
+
/// Stores the group index based on the hash of its value
///
/// We don't store the hashes as hashing fixed width primitives
/// is fast enough for this not to benefit performance
- map: HashTable<usize>,
+ map: HashTable<(u64, u64)>,
+
/// The group index of the null value if any
- null_group: Option<usize>,
+ null_group: Option<u64>,
+
/// The values for each group index
- values: Vec<T::Native>,
+ values: Vec<Vec<T::Native>>,
Review Comment:
Could we use
```suggestion
values: Vec<Box<[T::Native]>>,
```
here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2877460331
Plan to sort out codes and make it ready again today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890443367
π€ `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing intermeidate-result-blocked-approach (600318f348908f3aa3a80ddc3c8cb6d9c76deabc) to 07fe23f03dcb53b2715d3f3b3e0c476d54202c00 [diff](https://github.com/apache/datafusion/compare/07fe23f03dcb53b2715d3f3b3e0c476d54202c00..600318f348908f3aa3a80ddc3c8cb6d9c76deabc)
Benchmarks: clickbench_1 tpch_mem
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890527702
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_1.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 0.64ms β 0.61ms β +1.06x faster β
β QQuery 1 β 73.32ms β 77.79ms β 1.06x slower β
β QQuery 2 β 116.11ms β 121.93ms β 1.05x slower β
β QQuery 3 β 127.64ms β 128.08ms β no change β
β QQuery 4 β 650.79ms β 627.02ms β no change β
β QQuery 5 β 862.07ms β 864.44ms β no change β
β QQuery 6 β 0.76ms β 0.71ms β +1.07x faster β
β QQuery 7 β 89.64ms β 96.33ms β 1.07x slower β
β QQuery 8 β 945.07ms β 945.95ms β no change β
β QQuery 9 β 1243.07ms β 1248.13ms β no change β
β QQuery 10 β 294.76ms β 302.60ms β no change β
β QQuery 11 β 327.74ms β 342.77ms β no change β
β QQuery 12 β 876.40ms β 901.09ms β no change β
β QQuery 13 β 1301.86ms β 1359.55ms β no change β
β QQuery 14 β 864.23ms β 850.71ms β no change β
β QQuery 15 β 845.06ms β 852.93ms β no change β
β QQuery 16 β 1750.37ms β 1757.44ms β no change β
β QQuery 17 β 1605.74ms β 1621.73ms β no change β
β QQuery 18 β 3082.30ms β 3123.52ms β no change β
β QQuery 19 β 124.11ms β 124.83ms β no change β
β QQuery 20 β 1179.74ms β 1168.95ms β no change β
β QQuery 21 β 1413.11ms β 1400.95ms β no change β
β QQuery 22 β 2537.54ms β 2451.09ms β no change β
β QQuery 23 β 8489.89ms β 8532.97ms β no change β
β QQuery 24 β 511.43ms β 505.62ms β no change β
β QQuery 25 β 427.56ms β 425.53ms β no change β
β QQuery 26 β 577.57ms β 577.86ms β no change β
β QQuery 27 β 1707.08ms β 1709.85ms β no change β
β QQuery 28 β 12952.06ms β 13194.62ms β no change β
β QQuery 29 β 573.95ms β 570.25ms β no change β
β QQuery 30 β 846.22ms β 855.72ms β no change β
β QQuery 31 β 903.90ms β 896.13ms β no change β
β QQuery 32 β 2682.87ms β 2680.35ms β no change β
β QQuery 33 β 3413.44ms β 3409.35ms β no change β
β QQuery 34 β 3453.88ms β 3441.08ms β no change β
β QQuery 35 β 1287.13ms β 1278.56ms β no change β
β QQuery 36 β 175.07ms β 178.36ms β no change β
β QQuery 37 β 104.82ms β 103.96ms β no change β
β QQuery 38 β 174.73ms β 172.98ms β no change β
β QQuery 39 β 261.50ms β 258.02ms β no change β
β QQuery 40 β 87.21ms β 89.21ms β no change β
β QQuery 41 β 86.33ms β 85.57ms β no change β
β QQuery 42 β 82.95ms β 79.43ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 59111.65ms β
β Total Time (intermeidate-result-blocked-approach) β 59414.57ms β
β Average Time (HEAD) β 1374.69ms β
β Average Time (intermeidate-result-blocked-approach) β 1381.73ms β
β Queries Faster β 2 β
β Queries Slower β 3 β
β Queries with No Change β 38 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
ββββββββββββββββ³βββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 119.85ms β 123.00ms β no change β
β QQuery 2 β 22.89ms β 24.18ms β 1.06x slower β
β QQuery 3 β 34.42ms β 34.48ms β no change β
β QQuery 4 β 20.45ms β 20.75ms β no change β
β QQuery 5 β 54.12ms β 55.55ms β no change β
β QQuery 6 β 12.15ms β 12.01ms β no change β
β QQuery 7 β 104.41ms β 107.42ms β no change β
β QQuery 8 β 25.93ms β 27.42ms β 1.06x slower β
β QQuery 9 β 61.78ms β 63.54ms β no change β
β QQuery 10 β 58.64ms β 57.95ms β no change β
β QQuery 11 β 13.10ms β 13.86ms β 1.06x slower β
β QQuery 12 β 43.69ms β 46.48ms β 1.06x slower β
β QQuery 13 β 29.74ms β 30.87ms β no change β
β QQuery 14 β 10.18ms β 10.02ms β no change β
β QQuery 15 β 25.08ms β 27.64ms β 1.10x slower β
β QQuery 16 β 23.21ms β 22.51ms β no change β
β QQuery 17 β 101.75ms β 103.30ms β no change β
β QQuery 18 β 248.41ms β 244.84ms β no change β
β QQuery 19 β 28.58ms β 26.58ms β +1.08x faster β
β QQuery 20 β 39.04ms β 39.30ms β no change β
β QQuery 21 β 173.27ms β 172.60ms β no change β
β QQuery 22 β 17.97ms β 17.79ms β no change β
ββββββββββββββββ΄βββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1268.64ms β
β Total Time (intermeidate-result-blocked-approach) β 1282.10ms β
β Average Time (HEAD) β 57.67ms β
β Average Time (intermeidate-result-blocked-approach) β 58.28ms β
β Queries Faster β 1 β
β Queries Slower β 5 β
β Queries with No Change β 16 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890589301
> I think still need some time to evaluate what benefits can this brings
I think I am somewhat lost with the current state. Your comment on https://github.com/apache/datafusion/pull/15591#issuecomment-2823308363 states
> I add a query in extened.sql, the blocked approach can get a obvious improvement as expected.
>
> I have confidence it can even improve more in some other queries according to poc https://github.com/apache/datafusion/pull/11943
And the most recent benchmark run https://github.com/apache/datafusion/pull/15591#issuecomment-2890539606 seems to confirm this finding:
> ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1916.94ms β 1866.52ms β no change β
β QQuery 1 β 709.88ms β 673.73ms β +1.05x faster β
β QQuery 2 β 1482.93ms β 1391.57ms β +1.07x faster β
β QQuery 3 β 701.88ms β 721.01ms β no change β
β QQuery 4 β 1494.91ms β 1494.55ms β no change β
β QQuery 5 β 15627.54ms β 15418.84ms β no change β
β QQuery 6 β 2070.08ms β 2042.39ms β no change β
β QQuery 7 β 2108.54ms β 1952.66ms β +1.08x faster β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
However, some of the results on shorter running queries (few groups) shows perhaps a slowdown: https://github.com/apache/datafusion/pull/15591#issuecomment-2890527702
From those numbers, is it a fair assessment that the blocked approach improves performance when there is a large number of intermediate groups, but does not when there is a small number of groups?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Impl intermeidate result blocked approach sketch [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2785561617
> Hi @Rachelint I think I have a alternative proposal that seems relatively easy to implement. I'll share it with you once I have some time to validate the design (probably this evening).
Really thanks. This design in pr indeed introduces quite a few code changes...
I tried to not modify anythings about `GroupAccumulator` firstly:
- Only implement the `blocked logic` in `GroupValues`
- Then we reorder the `input batch` according to their `block index` got from `GroupValues`
- Apply `input batch` to related `GroupAccumulator` using slice
- And when we found the `new block is needed`, create a new `GroupAccumulator`
But I found this way will introduce too many extra cost...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2917574821
Thanks for all your help @Rachelint and congratulations on the new job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3379306551
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4343630821
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4343604373)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4343604373-1908-2hjmh 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (47b69765e44dbac7ec4173315b64ca1d55f259dc) to 8f033e4 (merge-base) [diff](https://github.com/apache/datafusion/compare/8f033e411aa4e097a146718e61d90897fe6e9c3d..47b69765e44dbac7ec4173315b64ca1d55f259dc) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3347692645
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
Review Comment:
> should there also be a transition to `ProducingBlocks` here instead of `ProducingOutput` if enable_blocked_groups is set?
Yes, it should be `ProducingBlocks` finally.
`ProducingOutput` here is due to that: this pr still not apply the blocked approach into spilling path, I think make it in future prs may be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3347787317
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
Review Comment:
> The problem isn't that we're not transitioning to ProducingBlocks early (this can be done via the `skip partial agg feature`), the problem is that:
>
> * emitting a large RecordBatch releases the memory reservation occupied by this batch, but the memory is not freed, it is used in the ProducingOutput state while producing small output batches
> * ProducingBlocks slices the large RecordBatch into smaller ones, but calling `get_array_memory_size` on the small record batch returns the memory of the entire original large RecordBatch, so spilling is broken for the parent operator (e.g. repartitioning)
Yes... what's worse is when dump the slicing `sub-batch`, it actually dump all the whole bytes of the parent batch it slicing from.
And when loading back, it actually load back `num_sub_batch` * `size of parent batch` bytes...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4275998373
run benchmark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276059580
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4276055952)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4276055952-1548-ccn4x 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (a68716b1b99123fef41259899c0c67adda870dbb) to dc973cc (merge-base) [diff](https://github.com/apache/datafusion/compare/dc973cc9e67e3519ec4bc5bd15962e2692debb3e..a68716b1b99123fef41259899c0c67adda870dbb) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3080550313
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ collections::VecDeque,
+ fmt::Debug,
+ iter,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct Blocks<B: Block> {
+ inner: VecDeque<B>,
Review Comment:
I think it would be nice to avoid the `VecDeque` as I believe it is relatively slow to index (because of the `%`).
I think we can use a start offset instead during pop (and increase it), replace the block with an empty one to "pop" it and reclaim the memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3084249604
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ collections::VecDeque,
+ fmt::Debug,
+ iter,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct Blocks<B: Block> {
+ inner: VecDeque<B>,
Review Comment:
Ok, but I think the `Vec` approach is relatively simple as well?
Not to pin you down, but I think when it will be used more it is problably coming up later anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4781064392
> > We can run benchmarks with memory limits to force spilling if that helps
>
> @adriangb hello, is it possible to authorize me to trigger benchmark through bot? I am trying to improve aggregation performance recently.
I also sent @adriangb a direct message as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2074192086
##########
datafusion/core/tests/fuzz_cases/aggregation_fuzzer/context_generator.rs:
##########
@@ -146,11 +147,14 @@ impl SessionContextGenerator {
(provider, false)
};
+ let enable_aggregation_blocked_groups = rng.gen_bool(0.5);
Review Comment:
I wonder if there is any value to testing the old code path (`enable_aggregation_blocked_groups = false`) if our goal is to remove it eventually.
I recommend only testing with the flag set to the default value
##########
datafusion/expr-common/src/groups_accumulator.rs:
##########
@@ -39,6 +43,9 @@ impl EmitTo {
/// remaining values in `v`.
///
/// This avoids copying if Self::All
+ ///
+ /// NOTICE: only support emit strategies: `Self::All` and `Self::First`
Review Comment:
```suggestion
/// NOTICE: only support emit strategies: `Self::All` and `Self::First`
/// Will call `panic` if called with `Self::NextBlock`
```
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
@@ -982,6 +1099,9 @@ impl GroupedHashAggregateStream {
&& self.update_memory_reservation().is_err()
{
assert_ne!(self.mode, AggregateMode::Partial);
+ // TODO: support spilling when blocked group optimization is on
Review Comment:
I think spilling with blocks is actually likely to be much better performing anyways (as we can spill each block potentially in parallel, for example)
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:
##########
@@ -212,25 +234,261 @@ impl NullState {
///
/// resets the internal state appropriately
pub fn build(&mut self, emit_to: EmitTo) -> NullBuffer {
- let nulls: BooleanBuffer = self.seen_values.finish();
+ self.seen_values.emit(emit_to)
+ }
+}
- let nulls = match emit_to {
- EmitTo::All => nulls,
- EmitTo::First(n) => {
- // split off the first N values in seen_values
- //
- // TODO make this more efficient rather than two
- // copies and bitwise manipulation
- let first_n_null: BooleanBuffer = nulls.iter().take(n).collect();
- // reset the existing seen buffer
- for seen in nulls.iter().skip(n) {
- self.seen_values.append(seen);
+/// Adapter for supporting dynamic dispatching of [`FlatNullState`] and [`BlockedNullState`].
+/// For performance, the cost of batch-level dynamic dispatching is acceptable.
+#[derive(Debug)]
+pub enum NullStateAdapter {
+ Flat(FlatNullState),
+ Blocked(BlockedNullState),
+}
+
+impl NullStateAdapter {
+ pub fn new(block_size: Option<usize>) -> Self {
+ if let Some(blk_size) = block_size {
+ Self::Blocked(BlockedNullState::new(blk_size))
+ } else {
+ Self::Flat(FlatNullState::new())
+ }
+ }
+
+ pub fn accumulate<T, F>(
+ &mut self,
+ group_indices: &[usize],
+ values: &PrimitiveArray<T>,
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ value_fn: F,
+ ) where
+ T: ArrowPrimitiveType + Send,
+ F: FnMut(u32, u64, T::Native) + Send,
+ {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.accumulate(
+ group_indices,
+ values,
+ opt_filter,
+ total_num_groups,
+ value_fn,
+ ),
+ NullStateAdapter::Blocked(null_state) => null_state.accumulate(
+ group_indices,
+ values,
+ opt_filter,
+ total_num_groups,
+ value_fn,
+ ),
+ }
+ }
+
+ pub fn accumulate_boolean<F>(
+ &mut self,
+ group_indices: &[usize],
+ values: &BooleanArray,
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ value_fn: F,
+ ) where
+ F: FnMut(u32, u64, bool) + Send,
+ {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.accumulate_boolean(
+ group_indices,
+ values,
+ opt_filter,
+ total_num_groups,
+ value_fn,
+ ),
+ NullStateAdapter::Blocked(null_state) => null_state.accumulate_boolean(
+ group_indices,
+ values,
+ opt_filter,
+ total_num_groups,
+ value_fn,
+ ),
+ }
+ }
+
+ pub fn build(&mut self, emit_to: EmitTo) -> NullBuffer {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.build(emit_to),
+ NullStateAdapter::Blocked(null_state) => null_state.build(emit_to),
+ }
+ }
+
+ pub fn size(&self) -> usize {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.size(),
+ NullStateAdapter::Blocked(null_state) => null_state.size(),
+ }
+ }
+
+ /// Clone and build a single [`BooleanBuffer`] from `seen_values`,
+ /// only used for testing.
+ #[cfg(test)]
+ fn build_cloned_seen_values(&self) -> BooleanBuffer {
+ match self {
+ NullStateAdapter::Flat(null_state) => {
+ null_state.seen_values[0].finish_cloned()
+ }
+ NullStateAdapter::Blocked(null_state) => {
+ let mut return_builder = BooleanBufferBuilder::new(0);
+ let num_blocks = null_state.seen_values.len();
+ for blk_idx in 0..num_blocks {
+ let builder = &null_state.seen_values[blk_idx];
+ for idx in 0..builder.len() {
+ return_builder.append(builder.get_bit(idx));
+ }
+ }
+ return_builder.finish()
+ }
+ }
+ }
+
+ #[cfg(test)]
+ fn build_all_in_once(&mut self) -> NullBuffer {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.build(EmitTo::All),
+ NullStateAdapter::Blocked(null_state) => {
+ let mut return_builder = BooleanBufferBuilder::new(0);
+ let num_blocks = null_state.seen_values.len();
+ for _ in 0..num_blocks {
+ let blocked_nulls = null_state.build(EmitTo::NextBlock);
+ for bit in blocked_nulls.inner().iter() {
+ return_builder.append(bit);
+ }
}
- first_n_null
+
+ NullBuffer::new(return_builder.finish())
}
+ }
+ }
+}
+
+/// [`NullState`] for `flat groups input`
+///
+/// At first, you may need to see something about `block_id` and `block_offset`
+/// from [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// The `flat groups input` are organized like:
+///
+/// ```text
+/// row_0 group_index_0
+/// row_1 group_index_1
+/// row_2 group_index_2
+/// ...
+/// row_n group_index_n
+/// ```
+///
+/// If `row_x group_index_x` is not filtered(`group_index_x` is seen)
+/// `seen_values[group_index_x]` will be set to `true`.
+///
+/// For `set_bit(block_id, block_offset, value)`, `block_id` is unused,
+/// `block_offset` will be set to `group_index`.
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+pub type FlatNullState = NullState<FlatGroupIndexOperations>;
+
+impl FlatNullState {
+ pub fn new() -> Self {
+ Self::default()
+ }
+}
+
+impl Default for FlatNullState {
+ fn default() -> Self {
+ Self {
+ seen_values: Blocks::new(None),
+ _phantom: PhantomData {},
+ }
+ }
+}
+
+/// [`NullState`] for `blocked groups input`
Review Comment:
I really like how you have `NullState` parameterized now
##########
datafusion/expr-common/src/groups_accumulator.rs:
##########
@@ -250,4 +288,30 @@ pub trait GroupsAccumulator: Send {
/// This function is called once per batch, so it should be `O(n)` to
/// compute, not `O(num_groups)`
fn size(&self) -> usize;
+
+ /// Returns `true` if this accumulator supports blocked groups.
+ fn supports_blocked_groups(&self) -> bool {
+ false
+ }
+
+ /// Alter the block size in the accumulator
+ ///
+ /// If the target block size is `None`, it will use a single big
+ /// block(can think it a `Vec`) to manage the state.
+ ///
+ /// If the target block size` is `Some(blk_size)`, it will try to
+ /// set the block size to `blk_size`, and the try will only success
+ /// when the accumulator has supported blocked mode.
+ ///
+ /// NOTICE: After altering block size, all data in previous will be cleared.
Review Comment:
```suggestion
/// NOTICE: After altering block size, all data in existing accumulators will be cleared.
```
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:
##########
@@ -212,25 +234,261 @@ impl NullState {
///
/// resets the internal state appropriately
pub fn build(&mut self, emit_to: EmitTo) -> NullBuffer {
- let nulls: BooleanBuffer = self.seen_values.finish();
+ self.seen_values.emit(emit_to)
+ }
+}
- let nulls = match emit_to {
- EmitTo::All => nulls,
- EmitTo::First(n) => {
- // split off the first N values in seen_values
- //
- // TODO make this more efficient rather than two
- // copies and bitwise manipulation
- let first_n_null: BooleanBuffer = nulls.iter().take(n).collect();
- // reset the existing seen buffer
- for seen in nulls.iter().skip(n) {
- self.seen_values.append(seen);
+/// Adapter for supporting dynamic dispatching of [`FlatNullState`] and [`BlockedNullState`].
+/// For performance, the cost of batch-level dynamic dispatching is acceptable.
+#[derive(Debug)]
+pub enum NullStateAdapter {
+ Flat(FlatNullState),
+ Blocked(BlockedNullState),
+}
+
+impl NullStateAdapter {
+ pub fn new(block_size: Option<usize>) -> Self {
+ if let Some(blk_size) = block_size {
+ Self::Blocked(BlockedNullState::new(blk_size))
+ } else {
+ Self::Flat(FlatNullState::new())
+ }
+ }
+
+ pub fn accumulate<T, F>(
+ &mut self,
+ group_indices: &[usize],
+ values: &PrimitiveArray<T>,
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ value_fn: F,
+ ) where
+ T: ArrowPrimitiveType + Send,
+ F: FnMut(u32, u64, T::Native) + Send,
+ {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.accumulate(
+ group_indices,
+ values,
+ opt_filter,
+ total_num_groups,
+ value_fn,
+ ),
+ NullStateAdapter::Blocked(null_state) => null_state.accumulate(
+ group_indices,
+ values,
+ opt_filter,
+ total_num_groups,
+ value_fn,
+ ),
+ }
+ }
+
+ pub fn accumulate_boolean<F>(
+ &mut self,
+ group_indices: &[usize],
+ values: &BooleanArray,
+ opt_filter: Option<&BooleanArray>,
+ total_num_groups: usize,
+ value_fn: F,
+ ) where
+ F: FnMut(u32, u64, bool) + Send,
+ {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.accumulate_boolean(
+ group_indices,
+ values,
+ opt_filter,
+ total_num_groups,
+ value_fn,
+ ),
+ NullStateAdapter::Blocked(null_state) => null_state.accumulate_boolean(
+ group_indices,
+ values,
+ opt_filter,
+ total_num_groups,
+ value_fn,
+ ),
+ }
+ }
+
+ pub fn build(&mut self, emit_to: EmitTo) -> NullBuffer {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.build(emit_to),
+ NullStateAdapter::Blocked(null_state) => null_state.build(emit_to),
+ }
+ }
+
+ pub fn size(&self) -> usize {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.size(),
+ NullStateAdapter::Blocked(null_state) => null_state.size(),
+ }
+ }
+
+ /// Clone and build a single [`BooleanBuffer`] from `seen_values`,
+ /// only used for testing.
+ #[cfg(test)]
+ fn build_cloned_seen_values(&self) -> BooleanBuffer {
+ match self {
+ NullStateAdapter::Flat(null_state) => {
+ null_state.seen_values[0].finish_cloned()
+ }
+ NullStateAdapter::Blocked(null_state) => {
+ let mut return_builder = BooleanBufferBuilder::new(0);
+ let num_blocks = null_state.seen_values.len();
+ for blk_idx in 0..num_blocks {
+ let builder = &null_state.seen_values[blk_idx];
+ for idx in 0..builder.len() {
+ return_builder.append(builder.get_bit(idx));
+ }
+ }
+ return_builder.finish()
+ }
+ }
+ }
+
+ #[cfg(test)]
+ fn build_all_in_once(&mut self) -> NullBuffer {
+ match self {
+ NullStateAdapter::Flat(null_state) => null_state.build(EmitTo::All),
+ NullStateAdapter::Blocked(null_state) => {
+ let mut return_builder = BooleanBufferBuilder::new(0);
+ let num_blocks = null_state.seen_values.len();
+ for _ in 0..num_blocks {
+ let blocked_nulls = null_state.build(EmitTo::NextBlock);
+ for bit in blocked_nulls.inner().iter() {
+ return_builder.append(bit);
+ }
}
- first_n_null
+
+ NullBuffer::new(return_builder.finish())
}
+ }
+ }
+}
+
+/// [`NullState`] for `flat groups input`
+///
+/// At first, you may need to see something about `block_id` and `block_offset`
+/// from [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// The `flat groups input` are organized like:
+///
+/// ```text
+/// row_0 group_index_0
+/// row_1 group_index_1
+/// row_2 group_index_2
+/// ...
+/// row_n group_index_n
+/// ```
+///
+/// If `row_x group_index_x` is not filtered(`group_index_x` is seen)
+/// `seen_values[group_index_x]` will be set to `true`.
+///
+/// For `set_bit(block_id, block_offset, value)`, `block_id` is unused,
+/// `block_offset` will be set to `group_index`.
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+pub type FlatNullState = NullState<FlatGroupIndexOperations>;
+
+impl FlatNullState {
+ pub fn new() -> Self {
+ Self::default()
+ }
+}
+
+impl Default for FlatNullState {
+ fn default() -> Self {
+ Self {
+ seen_values: Blocks::new(None),
+ _phantom: PhantomData {},
+ }
+ }
+}
+
+/// [`NullState`] for `blocked groups input`
+///
+/// At first, you may need to see something about `block_id` and `block_offset`
+/// from [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// The `flat groups input` are organized like:
+///
+/// ```text
+/// row_0 (block_id_0, block_offset_0)
+/// row_1 (block_id_1, block_offset_1)
+/// row_2 (block_id_1, block_offset_1)
Review Comment:
Should this have a different block offsets?
```suggestion
/// row_1 (block_id_1, block_offset_0)
/// row_2 (block_id_1, block_offset_1)
```
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,305 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ collections::VecDeque,
+ fmt::Debug,
+ iter,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
Review Comment:
β€οΈ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2850575408
Thanks @Rachelint -- I will give this PR another good look today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2852844732
> > It is really suprised that it show slower?
>
> This is my only concern too.
Yes, I think we should find out the reason, it is really suprised and it actually show reproducable improvement in my local
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848548914
π€ `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing intermeidate-result-blocked-approach (ff9c3adc4523e7df83ccccee635679926f083dda) to 74dc4196858784d7872b21bbfc97edc564e47c5e [diff](https://github.com/apache/datafusion/compare/74dc4196858784d7872b21bbfc97edc564e47c5e..ff9c3adc4523e7df83ccccee635679926f083dda)
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848554286
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_extended.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1921.22ms β 1898.23ms β no change β
β QQuery 1 β 657.26ms β 694.92ms β 1.06x slower β
β QQuery 2 β 1398.94ms β 1419.32ms β no change β
β QQuery 3 β 713.54ms β 717.51ms β no change β
β QQuery 4 β 1498.02ms β 1511.57ms β no change β
β QQuery 5 β 15341.63ms β 15578.65ms β no change β
β QQuery 6 β 2029.05ms β 2049.81ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 23559.67ms β
β Total Time (intermeidate-result-blocked-approach) β 23870.00ms β
β Average Time (HEAD) β 3365.67ms β
β Average Time (intermeidate-result-blocked-approach) β 3410.00ms β
β Queries Faster β 0 β
β Queries Slower β 1 β
β Queries with No Change β 6 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark clickbench_partitioned.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 2.32ms β 2.31ms β no change β
β QQuery 1 β 36.90ms β 38.43ms β no change β
β QQuery 2 β 91.60ms β 93.54ms β no change β
β QQuery 3 β 100.24ms β 99.54ms β no change β
β QQuery 4 β 777.83ms β 801.76ms β no change β
β QQuery 5 β 860.11ms β 850.06ms β no change β
β QQuery 6 β 2.10ms β 2.13ms β no change β
β QQuery 7 β 41.49ms β 43.64ms β 1.05x slower β
β QQuery 8 β 925.11ms β 877.46ms β +1.05x faster β
β QQuery 9 β 1201.13ms β 1193.97ms β no change β
β QQuery 10 β 262.49ms β 270.08ms β no change β
β QQuery 11 β 302.47ms β 313.89ms β no change β
β QQuery 12 β 902.20ms β 956.22ms β 1.06x slower β
β QQuery 13 β 1325.06ms β 1388.51ms β no change β
β QQuery 14 β 826.08ms β 851.33ms β no change β
β QQuery 15 β 1026.88ms β 1061.12ms β no change β
β QQuery 16 β 1720.69ms β 1731.12ms β no change β
β QQuery 17 β 1582.12ms β 1599.34ms β no change β
β QQuery 18 β 3066.54ms β 3254.59ms β 1.06x slower β
β QQuery 19 β 84.76ms β 82.92ms β no change β
β QQuery 20 β 1116.50ms β 1178.43ms β 1.06x slower β
β QQuery 21 β 1315.33ms β 1318.33ms β no change β
β QQuery 22 β 2190.24ms β 2170.50ms β no change β
β QQuery 23 β 8290.39ms β 8370.27ms β no change β
β QQuery 24 β 470.87ms β 486.49ms β no change β
β QQuery 25 β 388.04ms β 399.37ms β no change β
β QQuery 26 β 527.09ms β 542.36ms β no change β
β QQuery 27 β 1686.91ms β 1727.53ms β no change β
β QQuery 28 β 12444.07ms β 12631.97ms β no change β
β QQuery 29 β 525.46ms β 534.61ms β no change β
β QQuery 30 β 798.93ms β 822.60ms β no change β
β QQuery 31 β 829.79ms β 870.36ms β no change β
β QQuery 32 β 2594.87ms β 2731.68ms β 1.05x slower β
β QQuery 33 β 3320.81ms β 3380.18ms β no change β
β QQuery 34 β 3357.44ms β 3387.27ms β no change β
β QQuery 35 β 1290.00ms β 1282.24ms β no change β
β QQuery 36 β 130.81ms β 126.75ms β no change β
β QQuery 37 β 57.49ms β 58.36ms β no change β
β QQuery 38 β 127.97ms β 128.85ms β no change β
β QQuery 39 β 204.48ms β 207.11ms β no change β
β QQuery 40 β 50.31ms β 48.89ms β no change β
β QQuery 41 β 48.25ms β 44.18ms β +1.09x faster β
β QQuery 42 β 39.38ms β 38.45ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 56943.53ms β
β Total Time (intermeidate-result-blocked-approach) β 57998.74ms β
β Average Time (HEAD) β 1324.27ms β
β Average Time (intermeidate-result-blocked-approach) β 1348.81ms β
β Queries Faster β 2 β
β Queries Slower β 5 β
β Queries with No Change β 36 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
ββββββββββββββββ³βββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 124.78ms β 130.74ms β no change β
β QQuery 2 β 24.32ms β 23.65ms β no change β
β QQuery 3 β 35.37ms β 35.46ms β no change β
β QQuery 4 β 20.87ms β 20.87ms β no change β
β QQuery 5 β 54.85ms β 57.66ms β 1.05x slower β
β QQuery 6 β 12.21ms β 12.64ms β no change β
β QQuery 7 β 99.22ms β 102.82ms β no change β
β QQuery 8 β 26.11ms β 25.50ms β no change β
β QQuery 9 β 63.11ms β 64.15ms β no change β
β QQuery 10 β 57.98ms β 59.17ms β no change β
β QQuery 11 β 13.04ms β 13.20ms β no change β
β QQuery 12 β 43.72ms β 44.49ms β no change β
β QQuery 13 β 28.90ms β 29.45ms β no change β
β QQuery 14 β 9.78ms β 10.58ms β 1.08x slower β
β QQuery 15 β 26.05ms β 25.35ms β no change β
β QQuery 16 β 23.45ms β 23.66ms β no change β
β QQuery 17 β 95.93ms β 97.78ms β no change β
β QQuery 18 β 240.22ms β 238.96ms β no change β
β QQuery 19 β 27.56ms β 28.63ms β no change β
β QQuery 20 β 38.46ms β 38.79ms β no change β
β QQuery 21 β 173.00ms β 168.56ms β no change β
β QQuery 22 β 18.15ms β 17.81ms β no change β
ββββββββββββββββ΄βββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1257.09ms β
β Total Time (intermeidate-result-blocked-approach) β 1269.90ms β
β Average Time (HEAD) β 57.14ms β
β Average Time (intermeidate-result-blocked-approach) β 57.72ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 20 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075117009
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,20 +94,27 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
Review Comment:
yes, i mean block size. I would expect something like batchsize (4-8k), or maybe even bigger to have lower overhead? Did you run some experiments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2854306764
> π€: Benchmark completed
@alamb I think the faster queries here mainly led by the hash reusing optimization.
I have submitted a new pr here:
https://github.com/apache/datafusion/pull/15962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2863608739
BTW, I also simplified codes although not help to performance.
like `NullState`, I found we actually don't need to introduce blocked approach for it(even will lead to slight regression if we do so).
I remove the related codes to make it nearly unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890549663
> * We perform few large memory allocation in `flat`, but will much more small memory allocations in `blocked`
> * The memory no continuous anymore, I think it may be not so friendly for cpu(like cache prefetch?)
I wonder what happens if we make it more like at least 1 million or 1MiB (e.g. one) so the effect on cache is smaller?
We could optimize a growing strategy for the first allocated `Vec` if memory usage / overhead of first block is a concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890475454
> @alamb Hello, is it ok to trigger the benchmark again?
Will do
> @alamb @Dandandan It is sorry that, I still working on other some new ideas about this pr, so switch to draft again.
No worries -- I am sorry I have not given it as much attention as I would like. Is there anything else I can do to help (like implement one of the grouped memory examples)? I am about to file a ticket to start organizing the follow on work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890562106
> BTW, I am confused about why so many `page_fault` in the `blocked accumualte`.
Me too -- I looked at the flamegraph you provided and I agree it seems like almost half the allocation time is spent with pagefaults / zeroing memory. However, I can't tell if that is because there is slowness with the underlying Vec that wasn't initialized or if there is something else going on.
> * But it maybe not really help much for performance (performance improve mainly due to removal of usage of expansive `slice`) currently.
Yes, that was my understanding -- that blocked aggregation would only help performance when the number of intermediate groups was large (which forced additional memory allocations)
> But inspired by the `batch_size` based memory allocation, I am thinking can we have some ways to reuse memory? And I am trying it today.
I suspect you already know this, but I think you can get back the original Vec from an array via
1. `PrimitiveArray::into_parts()` --> get a `ScalarBuffer`
2. `ScalarBuffer::into_inner()` --> get a `Buffer`
3. `[Buffer::into_vec()](https://docs.rs/arrow/latest/arrow/buffer/struct.Buffer.html#method.into_vec)`
However, in the high cardinality case, I am not sure there are buffers to reuse during aggregation (the buffers are all held until the output is needed, and then once output is needed they don't get re-created)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890700369
> Yes, in current implementation, it only help to performance with large amount of intermediate groups, because the slice will be called many many time and the cost become unacceptable.
And for query with only small groups, it nearly make no difference.
Yeah I think that was expected.
I think we should try to minimize the impact of this on low-cardinality cases (e.g. make sure they fit in one array, minimize the overhead of blocks)...
> So after experiement, I think single vector + resizing is efficient enough actually...
Yeah it is quite efficient, although problematic for large inputs
* Offset out of bounds for utf8 / binary data.
* Overallocation due to exponential allocation strategy
So even with roughly the same performance I think we should still strive to make the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3153036085
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3154660341
> > It is unfortunate we never figured out how to get this over the line π’
> > Thank you for the effort anyways @Rachelint
>
> It is sorry... but actually I still want to continue push it forward... However it is too busy recent few months...
Maybe if/when you are able to return to it with a fresh set of eyes after a break we'll make progress
No worries at all -- I totally understand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #15591: Intermediate result blocked approach to aggregation memory management
URL: https://github.com/apache/datafusion/pull/15591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3749856254
> any ETA you are targeting for making it ready for review?
I plan to make it this week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3830479590
> Hi @Rachelint any update on this?
Continue working today... A bit busy this week, and sorry for delay for the pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4738890798
And before splitting, I will continue to make and prove the refactoring of `GroupValues` and `GroupAccumulator` will not lead to regression in this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4746951932
> I suggest not trying to parallelize steps 1 and 2, as they will likely conflict with each other. Step 3 should be highly parallelizable...
Make sense.
> The performance seems to be a nearly solved issue...
Yes, and I think it seems make slight difference to performance after experiment before.
https://github.com/apache/datafusion/pull/15591#issuecomment-2890524255
And I am paying effort to make it able to lead not regression when we disable the feature(regression happend due to we need to performance two index op even disabling).
https://github.com/apache/datafusion/pull/15591#issuecomment-4343775983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4785188060
run benchmarks clickbench_partitioned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4335043404
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4335018412)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4335018412-1866-rzm7g 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (a818cd8923a8b19ce6befad53e0b052aa47be9bf) to 22bb4e6 (merge-base) [diff](https://github.com/apache/datafusion/compare/22bb4e6b752c7a62b677d94a63bcf08b68e8d5ec..a818cd8923a8b19ce6befad53e0b052aa47be9bf) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3169360091
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
Review Comment:
Could be outside of this PR's scope, but another question I have is why isn't there a transition to `ProducingBlocks` as soon as we have a batch ready. Instead, all the input is consumed first before producing the first batch (assuming there's no group ordering)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5424238812
> > my implementation ditched the emit to first and emit all and instead only have emit next block, since it will simplify implementations to not have enums and a lot of generics like this PR have to support both
> > The problem with that is that this is breaking change that cannot be done incrementally, like, all the helpers can be created in different pr, but changing to blocked implementation cannot be done incremently.
>
> Based on my previous experiments, I agree with using only `EmitTo::NextBlock`, and I also think it would be challenging to roll this out incrementally.
>
> I can participate asynchronously by reviewing PoC PRs. Given the complexity of the implementation, I think weβll need to look at the actual code to make meaningful progress.
Maybe we can add an `EmitTo::NextBlock` to the enum (and left the others), but made all existing aggregators / group key storage error with unimplemented at first? And then we slowly migrated things over?
I realize it will be hard to do this incrementally, but I think it would be easier than trying to do it all at once
What I suggest is:
1. We push to finish the existing grouping refactor (so we can delete the old hash stream code and focus on one copy at a time)
2. In parallel, prototype what an EmitTo::NextBlock would look like using the new hash aggregate streams (e.g. https://github.com/apache/datafusion/blob/9a3d35351aaa91a4d22d6cb9dc4ce61c76059dc2/datafusion/physical-plan/src/aggregates/mod.rs#L1233-L1279)
Maybe we could focus on just one stream type (e.g. `PartialHashAggregateStream` or `FinalHashAggregateStream`) and figure out how EmitTo::NextBloc could work there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5333397682
> In the very hot loop of doing aggregation and jumping in memory you are adding another memory jump
I think this is the key observation -- I think there is no way we can add the extra indirection and not slow down performance for small aggregates.
For higher cardinality, I think it may not be as bad as we are going to be doing lots of random memory acesses anyways as the table doesn't fit in the processor caches.
So I am guessing what we will need is some way to use direct indexing for small hash tables, but when the table grows above a certain size switch to 2 part indexes
But that is just a theory and I have not tested it yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5377900320
@rluvaton @alamb @Dandandan
Iβm actually more concerned about the implementation complexity, especially for complex typesβthe work required just to extract the trait was already quite painful.
Iβm somewhat less concerned about the performance impact. Based on the benchmarks that have been run throughout this PR, the regression appears relatively modest, ranging from roughly neutral to around 10%.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848561993
π€ `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing intermeidate-result-blocked-approach (ff9c3adc4523e7df83ccccee635679926f083dda) to 74dc4196858784d7872b21bbfc97edc564e47c5e [diff](https://github.com/apache/datafusion/compare/74dc4196858784d7872b21bbfc97edc564e47c5e..ff9c3adc4523e7df83ccccee635679926f083dda)
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848571591
> Given we have past evidence this approach will work I think we could merge it before the queries really sped up. My big concern is that we have a plan to eventually avoid both single and multi blocked management
The mainly hard point for removing single management is the `sorted aggregation`:
- We need `Emit::First(exactly n)` for `sorted aggregation`
- Supporting `Emit::First(exactly n)` in blocked management is actually too expansive...
Actually I encounter the same problem in #12996 ... I think maybe it is a common problem about how to support `sorted aggregation`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2854295196
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_extended.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1940.88ms β 1976.66ms β no change β
β QQuery 1 β 760.85ms β 766.84ms β no change β
β QQuery 2 β 1501.65ms β 1542.77ms β no change β
β QQuery 3 β 719.82ms β 727.87ms β no change β
β QQuery 4 β 1484.50ms β 1494.48ms β no change β
β QQuery 5 β 15426.35ms β 15206.88ms β no change β
β QQuery 6 β 2069.76ms β 2128.21ms β no change β
β QQuery 7 β 2708.23ms β 2071.61ms β +1.31x faster β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 26612.04ms β
β Total Time (intermeidate-result-blocked-approach) β 25915.33ms β
β Average Time (HEAD) β 3326.50ms β
β Average Time (intermeidate-result-blocked-approach) β 3239.42ms β
β Queries Faster β 1 β
β Queries Slower β 0 β
β Queries with No Change β 7 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark clickbench_partitioned.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 2.25ms β 2.20ms β no change β
β QQuery 1 β 36.40ms β 39.03ms β 1.07x slower β
β QQuery 2 β 90.29ms β 91.70ms β no change β
β QQuery 3 β 99.54ms β 102.47ms β no change β
β QQuery 4 β 803.59ms β 611.45ms β +1.31x faster β
β QQuery 5 β 874.76ms β 860.26ms β no change β
β QQuery 6 β 2.29ms β 2.29ms β no change β
β QQuery 7 β 43.79ms β 45.26ms β no change β
β QQuery 8 β 902.84ms β 913.05ms β no change β
β QQuery 9 β 1166.55ms β 1218.38ms β no change β
β QQuery 10 β 265.93ms β 276.09ms β no change β
β QQuery 11 β 307.22ms β 315.94ms β no change β
β QQuery 12 β 929.85ms β 927.87ms β no change β
β QQuery 13 β 1362.99ms β 1364.08ms β no change β
β QQuery 14 β 831.28ms β 855.27ms β no change β
β QQuery 15 β 1027.92ms β 841.50ms β +1.22x faster β
β QQuery 16 β 1764.73ms β 1738.44ms β no change β
β QQuery 17 β 1595.54ms β 1615.47ms β no change β
β QQuery 18 β 3163.56ms β 3090.69ms β no change β
β QQuery 19 β 83.33ms β 84.19ms β no change β
β QQuery 20 β 1148.16ms β 1163.96ms β no change β
β QQuery 21 β 1335.22ms β 1362.75ms β no change β
β QQuery 22 β 2173.36ms β 2251.51ms β no change β
β QQuery 23 β 8305.24ms β 8570.07ms β no change β
β QQuery 24 β 481.58ms β 478.64ms β no change β
β QQuery 25 β 400.16ms β 404.96ms β no change β
β QQuery 26 β 542.87ms β 547.19ms β no change β
β QQuery 27 β 1569.96ms β 1610.78ms β no change β
β QQuery 28 β 12626.33ms β 12723.51ms β no change β
β QQuery 29 β 544.45ms β 527.61ms β no change β
β QQuery 30 β 820.33ms β 839.81ms β no change β
β QQuery 31 β 856.68ms β 896.19ms β no change β
β QQuery 32 β 2686.42ms β 2661.76ms β no change β
β QQuery 33 β 3404.03ms β 3380.42ms β no change β
β QQuery 34 β 3384.26ms β 3372.65ms β no change β
β QQuery 35 β 1279.60ms β 1305.74ms β no change β
β QQuery 36 β 127.62ms β 129.44ms β no change β
β QQuery 37 β 58.72ms β 57.92ms β no change β
β QQuery 38 β 127.53ms β 127.19ms β no change β
β QQuery 39 β 200.45ms β 206.16ms β no change β
β QQuery 40 β 52.11ms β 50.59ms β no change β
β QQuery 41 β 45.71ms β 48.66ms β 1.06x slower β
β QQuery 42 β 40.79ms β 40.32ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 57566.23ms β
β Total Time (intermeidate-result-blocked-approach) β 57753.47ms β
β Average Time (HEAD) β 1338.75ms β
β Average Time (intermeidate-result-blocked-approach) β 1343.10ms β
β Queries Faster β 2 β
β Queries Slower β 2 β
β Queries with No Change β 39 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
ββββββββββββββββ³βββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 123.23ms β 128.90ms β no change β
β QQuery 2 β 24.21ms β 23.25ms β no change β
β QQuery 3 β 34.68ms β 34.89ms β no change β
β QQuery 4 β 20.63ms β 20.65ms β no change β
β QQuery 5 β 55.82ms β 55.13ms β no change β
β QQuery 6 β 12.26ms β 12.21ms β no change β
β QQuery 7 β 104.79ms β 105.61ms β no change β
β QQuery 8 β 26.97ms β 26.69ms β no change β
β QQuery 9 β 61.07ms β 61.85ms β no change β
β QQuery 10 β 56.94ms β 57.98ms β no change β
β QQuery 11 β 12.90ms β 12.63ms β no change β
β QQuery 12 β 46.09ms β 44.05ms β no change β
β QQuery 13 β 29.68ms β 30.13ms β no change β
β QQuery 14 β 10.47ms β 10.19ms β no change β
β QQuery 15 β 26.18ms β 25.38ms β no change β
β QQuery 16 β 23.65ms β 23.74ms β no change β
β QQuery 17 β 97.76ms β 101.06ms β no change β
β QQuery 18 β 243.09ms β 240.17ms β no change β
β QQuery 19 β 27.98ms β 27.06ms β no change β
β QQuery 20 β 39.30ms β 39.33ms β no change β
β QQuery 21 β 169.96ms β 166.55ms β no change β
β QQuery 22 β 18.31ms β 18.76ms β no change β
ββββββββββββββββ΄βββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1265.96ms β
β Total Time (intermeidate-result-blocked-approach) β 1266.21ms β
β Average Time (HEAD) β 57.54ms β
β Average Time (intermeidate-result-blocked-approach) β 57.56ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 22 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072992809
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
@@ -982,6 +1099,9 @@ impl GroupedHashAggregateStream {
&& self.update_memory_reservation().is_err()
{
assert_ne!(self.mode, AggregateMode::Partial);
+ // TODO: support spilling when blocked group optimization is on
Review Comment:
Yes, I will file it.
Actually there are also some other problems in spilling, and I will fix it together when working on supporting blocked management in spilling path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2854199522
> > π€: Benchmark completed
> > Details
>
> It is really suprised that it show slower?
FWIW the machine I am running this benchmark on has 16 not very good cores:
```
cat /proc/cpuinfo
...
processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) CPU @ 3.10GHz
stepping : 7
microcode : 0xffffffff
cpu MHz : 3100.320
cache size : 25344 KB
physical id : 0
siblings : 16
core id : 7
cpu cores : 8
apicid : 15
initial apicid : 15
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
bugs : spectre_v1 spectre_v2 spec_store_bypass swapgs taa mmio_stale_data retbleed eibrs_pbrsb bhi
bogomips : 6200.64
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072859602
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs:
##########
@@ -507,3 +509,157 @@ pub(crate) fn slice_and_maybe_filter(
Ok(sliced_arrays)
}
}
+
+// ===============================================
+// Useful tools for group index
+// ===============================================
+
+/// Operations about group index parsing
+///
+/// There are mainly 2 `group index` needing parsing: `flat` and `blocked`.
+///
+/// # Flat group index
+/// `flat group index` format is like:
+///
+/// ```text
+/// | block_offset(64bit) |
+/// ```
+///
+/// It is used in `flat GroupValues/GroupAccumulator`, only a single block
+/// exists, so its `block_id` is always 0, and use all 64 bits to store the
+/// `block offset`.
+///
+/// # Blocked group index
+/// `blocked group index` format is like:
+///
+/// ```text
+/// | block_id(32bit) | block_offset(32bit)
+/// ```
+///
+/// It is used in `blocked GroupValues/GroupAccumulator`, multiple blocks
+/// exist, and we use high 32 bits to store `block_id`, and low 32 bit to
+/// store `block_offset`.
+///
+/// The `get_block_offset` method requires to return `block_offset` as u64,
+/// that is for compatible for `flat group index`'s parsing.
+///
+pub trait GroupIndexOperations: Debug {
+ fn pack_index(block_id: u32, block_offset: u64) -> u64;
+
+ fn get_block_id(packed_index: u64) -> u32;
+
+ fn get_block_offset(packed_index: u64) -> u64;
+}
+
+#[derive(Debug)]
+pub struct BlockedGroupIndexOperations;
+
+impl GroupIndexOperations for BlockedGroupIndexOperations {
+ fn pack_index(block_id: u32, block_offset: u64) -> u64 {
+ ((block_id as u64) << 32) | block_offset
+ }
+
+ fn get_block_id(packed_index: u64) -> u32 {
+ (packed_index >> 32) as u32
+ }
+
+ fn get_block_offset(packed_index: u64) -> u64 {
+ (packed_index as u32) as u64
+ }
+}
+
+#[derive(Debug)]
+pub struct FlatGroupIndexOperations;
+
+impl GroupIndexOperations for FlatGroupIndexOperations {
+ fn pack_index(_block_id: u32, block_offset: u64) -> u64 {
+ block_offset
+ }
+
+ fn get_block_id(_packed_index: u64) -> u32 {
+ 0
+ }
+
+ fn get_block_offset(packed_index: u64) -> u64 {
+ packed_index
+ }
+}
+
+// ===============================================
+// Useful tools for block
+// ===============================================
+pub(crate) fn ensure_room_enough_for_blocks<B, F>(
+ blocks: &mut VecDeque<B>,
+ total_num_groups: usize,
+ block_size: usize,
+ new_block: F,
+ default_value: B::T,
+) where
+ B: Block,
+ F: Fn(usize) -> B,
+{
+ // For resize, we need to:
+ // 1. Ensure the blks are enough first
+ // 2. and then ensure slots in blks are enough
+ let (mut cur_blk_idx, exist_slots) = if !blocks.is_empty() {
+ let cur_blk_idx = blocks.len() - 1;
+ let exist_slots = (blocks.len() - 1) * block_size + blocks.back().unwrap().len();
+
+ (cur_blk_idx, exist_slots)
+ } else {
+ (0, 0)
+ };
+
+ // No new groups, don't need to expand, just return
+ if exist_slots >= total_num_groups {
+ return;
+ }
+
+ // 1. Ensure blks are enough
+ let exist_blks = blocks.len();
+ let new_blks = total_num_groups.div_ceil(block_size) - exist_blks;
+ if new_blks > 0 {
+ for _ in 0..new_blks {
+ let block = new_block(block_size);
+ blocks.push_back(block);
+ }
+ }
+
+ // 2. Ensure slots are enough
+ let mut new_slots = total_num_groups - exist_slots;
+
+ // 2.1 Only fill current blk if it may be already enough
+ let cur_blk_rest_slots = block_size - blocks[cur_blk_idx].len();
+ if cur_blk_rest_slots >= new_slots {
+ blocks[cur_blk_idx].fill_default_value(new_slots, default_value.clone());
+ return;
+ }
+
+ // 2.2 Fill current blk to full
+ blocks[cur_blk_idx].fill_default_value(cur_blk_rest_slots, default_value.clone());
+ new_slots -= cur_blk_rest_slots;
+
+ // 2.3 Fill complete blks
+ let complete_blks = new_slots / block_size;
+ for _ in 0..complete_blks {
+ cur_blk_idx += 1;
+ blocks[cur_blk_idx].fill_default_value(block_size, default_value.clone());
+ }
+
+ // 2.4 Fill last blk if needed
+ let rest_slots = new_slots % block_size;
+ if rest_slots > 0 {
+ blocks
+ .back_mut()
+ .unwrap()
+ .fill_default_value(rest_slots, default_value);
+ }
+}
+
+pub(crate) trait Block {
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2852381080
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_extended.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1941.28ms β 2043.89ms β 1.05x slower β
β QQuery 1 β 772.84ms β 712.97ms β +1.08x faster β
β QQuery 2 β 1522.71ms β 1443.21ms β +1.06x faster β
β QQuery 3 β 719.60ms β 764.04ms β 1.06x slower β
β QQuery 4 β 1468.51ms β 1528.06ms β no change β
β QQuery 5 β 15673.72ms β 15342.97ms β no change β
β QQuery 6 β 2080.95ms β 2074.26ms β no change β
β QQuery 7 β 2533.15ms β 3035.39ms β 1.20x slower β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 26712.76ms β
β Total Time (intermeidate-result-blocked-approach) β 26944.80ms β
β Average Time (HEAD) β 3339.09ms β
β Average Time (intermeidate-result-blocked-approach) β 3368.10ms β
β Queries Faster β 2 β
β Queries Slower β 3 β
β Queries with No Change β 3 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark clickbench_partitioned.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 2.40ms β 2.62ms β 1.09x slower β
β QQuery 1 β 36.42ms β 39.04ms β 1.07x slower β
β QQuery 2 β 89.04ms β 93.59ms β 1.05x slower β
β QQuery 3 β 98.34ms β 97.56ms β no change β
β QQuery 4 β 798.63ms β 793.86ms β no change β
β QQuery 5 β 857.58ms β 838.99ms β no change β
β QQuery 6 β 2.29ms β 2.20ms β no change β
β QQuery 7 β 43.03ms β 43.70ms β no change β
β QQuery 8 β 926.38ms β 940.06ms β no change β
β QQuery 9 β 1200.58ms β 1264.19ms β 1.05x slower β
β QQuery 10 β 266.44ms β 271.84ms β no change β
β QQuery 11 β 312.69ms β 308.31ms β no change β
β QQuery 12 β 921.14ms β 915.99ms β no change β
β QQuery 13 β 1246.19ms β 1360.62ms β 1.09x slower β
β QQuery 14 β 861.03ms β 849.35ms β no change β
β QQuery 15 β 1031.14ms β 1065.46ms β no change β
β QQuery 16 β 1724.44ms β 1734.01ms β no change β
β QQuery 17 β 1600.58ms β 1619.84ms β no change β
β QQuery 18 β 3083.17ms β 3112.74ms β no change β
β QQuery 19 β 86.39ms β 84.08ms β no change β
β QQuery 20 β 1152.51ms β 1142.67ms β no change β
β QQuery 21 β 1340.71ms β 1335.70ms β no change β
β QQuery 22 β 2206.78ms β 2189.52ms β no change β
β QQuery 23 β 8437.51ms β 8423.58ms β no change β
β QQuery 24 β 482.26ms β 471.91ms β no change β
β QQuery 25 β 389.71ms β 384.09ms β no change β
β QQuery 26 β 539.21ms β 533.60ms β no change β
β QQuery 27 β 1582.07ms β 1608.14ms β no change β
β QQuery 28 β 12725.70ms β 12648.73ms β no change β
β QQuery 29 β 544.35ms β 533.90ms β no change β
β QQuery 30 β 823.56ms β 829.97ms β no change β
β QQuery 31 β 877.33ms β 864.93ms β no change β
β QQuery 32 β 2690.58ms β 2714.19ms β no change β
β QQuery 33 β 3366.34ms β 3387.57ms β no change β
β QQuery 34 β 3379.91ms β 3396.79ms β no change β
β QQuery 35 β 1285.04ms β 1307.48ms β no change β
β QQuery 36 β 127.50ms β 133.65ms β no change β
β QQuery 37 β 58.70ms β 57.55ms β no change β
β QQuery 38 β 126.60ms β 128.91ms β no change β
β QQuery 39 β 204.31ms β 201.60ms β no change β
β QQuery 40 β 50.93ms β 48.51ms β no change β
β QQuery 41 β 46.66ms β 50.48ms β 1.08x slower β
β QQuery 42 β 41.58ms β 40.14ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 57667.73ms β
β Total Time (intermeidate-result-blocked-approach) β 57871.62ms β
β Average Time (HEAD) β 1341.11ms β
β Average Time (intermeidate-result-blocked-approach) β 1345.85ms β
β Queries Faster β 0 β
β Queries Slower β 6 β
β Queries with No Change β 37 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
ββββββββββββββββ³βββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 123.50ms β 131.97ms β 1.07x slower β
β QQuery 2 β 23.84ms β 24.18ms β no change β
β QQuery 3 β 35.12ms β 34.43ms β no change β
β QQuery 4 β 20.83ms β 21.25ms β no change β
β QQuery 5 β 56.08ms β 55.91ms β no change β
β QQuery 6 β 12.46ms β 16.00ms β 1.28x slower β
β QQuery 7 β 102.73ms β 101.72ms β no change β
β QQuery 8 β 26.09ms β 25.78ms β no change β
β QQuery 9 β 61.80ms β 63.79ms β no change β
β QQuery 10 β 58.06ms β 58.04ms β no change β
β QQuery 11 β 13.35ms β 13.26ms β no change β
β QQuery 12 β 43.76ms β 46.30ms β 1.06x slower β
β QQuery 13 β 28.83ms β 29.74ms β no change β
β QQuery 14 β 10.44ms β 10.29ms β no change β
β QQuery 15 β 25.31ms β 25.77ms β no change β
β QQuery 16 β 24.81ms β 22.53ms β +1.10x faster β
β QQuery 17 β 97.61ms β 107.73ms β 1.10x slower β
β QQuery 18 β 238.30ms β 244.27ms β no change β
β QQuery 19 β 27.80ms β 28.20ms β no change β
β QQuery 20 β 38.27ms β 39.68ms β no change β
β QQuery 21 β 168.61ms β 171.59ms β no change β
β QQuery 22 β 17.91ms β 19.24ms β 1.07x slower β
ββββββββββββββββ΄βββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1255.49ms β
β Total Time (intermeidate-result-blocked-approach) β 1291.66ms β
β Average Time (HEAD) β 57.07ms β
β Average Time (intermeidate-result-blocked-approach) β 58.71ms β
β Queries Faster β 1 β
β Queries Slower β 5 β
β Queries with No Change β 16 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2888023128
> π€: Benchmark completed
> Details
>
> ```
> Comparing HEAD and intermeidate-result-blocked-approach
> --------------------
> Benchmark clickbench_extended.json
> --------------------
> ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
> β Query β HEAD β intermeidate-result-blocked-approach β Change β
> β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β QQuery 0 β 1952.39ms β 2165.55ms β 1.11x slower β
> β QQuery 1 β 725.35ms β 776.75ms β 1.07x slower β
> β QQuery 2 β 1474.77ms β 1525.72ms β no change β
> β QQuery 3 β 720.28ms β 733.30ms β no change β
> β QQuery 4 β 1491.93ms β 1549.46ms β no change β
> β QQuery 5 β 15503.37ms β 15748.81ms β no change β
> β QQuery 6 β 2118.87ms β 2106.19ms β no change β
> β QQuery 7 β 2215.33ms β 2189.80ms β no change β
> ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
> β Benchmark Summary β β
> β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β Total Time (HEAD) β 26202.29ms β
> β Total Time (intermeidate-result-blocked-approach) β 26795.58ms β
> β Average Time (HEAD) β 3275.29ms β
> β Average Time (intermeidate-result-blocked-approach) β 3349.45ms β
> β Queries Faster β 0 β
> β Queries Slower β 2 β
> β Queries with No Change β 6 β
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
> --------------------
> Benchmark clickbench_partitioned.json
> --------------------
> ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
> β Query β HEAD β intermeidate-result-blocked-approach β Change β
> β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β QQuery 0 β 2.42ms β 2.74ms β 1.13x slower β
> β QQuery 1 β 36.72ms β 39.03ms β 1.06x slower β
> β QQuery 2 β 91.22ms β 95.52ms β no change β
> β QQuery 3 β 101.22ms β 101.34ms β no change β
> β QQuery 4 β 638.98ms β 639.74ms β no change β
> β QQuery 5 β 906.48ms β 875.22ms β no change β
> β QQuery 6 β 2.31ms β 2.46ms β 1.06x slower β
> β QQuery 7 β 43.69ms β 43.48ms β no change β
> β QQuery 8 β 960.12ms β 947.47ms β no change β
> β QQuery 9 β 1290.26ms β 1263.99ms β no change β
> β QQuery 10 β 273.55ms β 273.89ms β no change β
> β QQuery 11 β 306.30ms β 319.30ms β no change β
> β QQuery 12 β 969.05ms β 954.35ms β no change β
> β QQuery 13 β 1311.18ms β 1265.49ms β no change β
> β QQuery 14 β 891.85ms β 867.59ms β no change β
> β QQuery 15 β 860.97ms β 883.83ms β no change β
> β QQuery 16 β 1764.37ms β 1767.56ms β no change β
> β QQuery 17 β 1653.48ms β 1634.83ms β no change β
> β QQuery 18 β 3151.14ms β 3177.39ms β no change β
> β QQuery 19 β 86.57ms β 83.49ms β no change β
> β QQuery 20 β 1164.58ms β 1161.57ms β no change β
> β QQuery 21 β 1380.48ms β 1337.39ms β no change β
> β QQuery 22 β 2295.06ms β 2227.17ms β no change β
> β QQuery 23 β 8437.28ms β 8619.52ms β no change β
> β QQuery 24 β 487.41ms β 475.78ms β no change β
> β QQuery 25 β 403.06ms β 401.45ms β no change β
> β QQuery 26 β 537.58ms β 540.40ms β no change β
> β QQuery 27 β 1608.99ms β 1597.83ms β no change β
> β QQuery 28 β 12616.94ms β 12274.70ms β no change β
> β QQuery 29 β 538.94ms β 547.25ms β no change β
> β QQuery 30 β 849.96ms β 846.92ms β no change β
> β QQuery 31 β 879.77ms β 888.86ms β no change β
> β QQuery 32 β 2719.39ms β 2737.35ms β no change β
> β QQuery 33 β 3459.61ms β 3454.83ms β no change β
> β QQuery 34 β 3471.82ms β 3530.09ms β no change β
> β QQuery 35 β 1327.72ms β 1326.17ms β no change β
> β QQuery 36 β 128.38ms β 128.36ms β no change β
> β QQuery 37 β 59.04ms β 59.49ms β no change β
> β QQuery 38 β 128.01ms β 125.30ms β no change β
> β QQuery 39 β 211.26ms β 201.66ms β no change β
> β QQuery 40 β 49.25ms β 49.61ms β no change β
> β QQuery 41 β 48.96ms β 47.59ms β no change β
> β QQuery 42 β 40.31ms β 40.19ms β no change β
> ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
> β Benchmark Summary β β
> β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β Total Time (HEAD) β 58185.65ms β
> β Total Time (intermeidate-result-blocked-approach) β 57858.18ms β
> β Average Time (HEAD) β 1353.15ms β
> β Average Time (intermeidate-result-blocked-approach) β 1345.54ms β
> β Queries Faster β 0 β
> β Queries Slower β 3 β
> β Queries with No Change β 40 β
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
> --------------------
> Benchmark tpch_mem_sf1.json
> --------------------
> ββββββββββββββββ³βββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
> β Query β HEAD β intermeidate-result-blocked-approach β Change β
> β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β QQuery 1 β 124.65ms β 125.09ms β no change β
> β QQuery 2 β 23.82ms β 23.97ms β no change β
> β QQuery 3 β 36.41ms β 36.11ms β no change β
> β QQuery 4 β 21.12ms β 20.25ms β no change β
> β QQuery 5 β 57.07ms β 55.34ms β no change β
> β QQuery 6 β 12.75ms β 12.51ms β no change β
> β QQuery 7 β 109.32ms β 103.95ms β no change β
> β QQuery 8 β 26.94ms β 29.50ms β 1.10x slower β
> β QQuery 9 β 63.71ms β 65.97ms β no change β
> β QQuery 10 β 59.22ms β 58.71ms β no change β
> β QQuery 11 β 13.22ms β 12.89ms β no change β
> β QQuery 12 β 43.95ms β 45.96ms β no change β
> β QQuery 13 β 30.27ms β 31.22ms β no change β
> β QQuery 14 β 10.31ms β 10.51ms β no change β
> β QQuery 15 β 25.33ms β 26.91ms β 1.06x slower β
> β QQuery 16 β 24.44ms β 22.87ms β +1.07x faster β
> β QQuery 17 β 102.19ms β 104.36ms β no change β
> β QQuery 18 β 250.78ms β 254.73ms β no change β
> β QQuery 19 β 26.81ms β 28.56ms β 1.07x slower β
> β QQuery 20 β 41.12ms β 42.04ms β no change β
> β QQuery 21 β 170.36ms β 173.00ms β no change β
> β QQuery 22 β 17.60ms β 17.06ms β no change β
> ββββββββββββββββ΄βββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
> β Benchmark Summary β β
> β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β Total Time (HEAD) β 1291.37ms β
> β Total Time (intermeidate-result-blocked-approach) β 1301.51ms β
> β Average Time (HEAD) β 58.70ms β
> β Average Time (intermeidate-result-blocked-approach) β 59.16ms β
> β Queries Faster β 1 β
> β Queries Slower β 3 β
> β Queries with No Change β 18 β
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
> ```
It is suprised that no obvious improvement in the testing machine...
Can I know somethings about the testing, such as how many `target_partitions` when running?
I am also considering how to get the same enviroment, and understand more about what happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890775426
> We make the block size large enough
For the first block, we still perform resizing at firstly
But after it grow large enough, we switch to blocked approach?
Yes - exactly!
Alternatively maybe we even can switch from flat to block approach after reaching a certain threshold (for example 1 million groups) to avoid overhead of the block approach on lower cardinality cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2907769450
I wonder what the plan is for this PR?
From what I understand, it currently improves performance for aggregates with large numbers of groups, but (slightly) slows down aggregates for smaller numbers of groups. I think this is due to accessing group storage via two indirections (block index / actual index)
It seems like the [proposal](https://github.com/apache/datafusion/pull/15591#issuecomment-2890775426) is to have some sort of adaptive structure that uses one part indexes for small numbers of groups and then switches to two part indexes for larger numbers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2910945348
> > My biggest concern is if we can get more obvious improvement to make the change worthy...
>
> As I understood the way to get a bigger improvement would be to implement the chunked approach for more group storage / aggregates so that more queries in our benchmarks (like ClickBench) could use the new code path
>
> Though of course that would make this PR even bigger
>
> We could also make a "POC" type PR with some more implementation to prove out the performance and then break it into smaller pieces for review π€
Yes, I am trying to implement it for `count`, and see if some improvements in `q4` and `q15`.
But according to the flamegraph, I found all such queries' bottleneck is actually `hashtable`. I think performance improvement from this pr will be not obvious before we overcome `hashtable`(I am experimenting about clickhouse like hashtable).
I think the benefit from this one currently is:
- Part of the better aggregation spilling (I will file a issue to discuss it maybe tonight)
- Lower memory usage due to faster freeing batch by batch
- Can make really high cardinality query around 1.1 faster
Do you think it still worthy continuing to push forward for above benefits?
I plan to sort out codes tonight, and we just make a simplest implementation in first stage (now is a bit complex due to some experiment about continue improving performance)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2910963654
> Part of the better aggregation spilling (I will file a issue to discuss it maybe tonight)
Without having a full understanding of this PR (I have just been following the conversation because the change is exciting) my 2Β’ is: for us memory management is currently one of the biggest thorns with DataFusion. It is quite hard at the moment to run with a fixed memory budget given the mix of exceeding memory through under accounting and refusing to run operators that can't spill / try to claim more memory than they actually use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072414793
##########
benchmarks/queries/clickbench/extended.sql:
##########
@@ -5,3 +5,4 @@ SELECT "SocialSourceNetworkID", "RegionID", COUNT(*), AVG("Age"), AVG("ParamPric
SELECT "ClientIP", "WatchID", COUNT(*) c, MIN("ResponseStartTiming") tmin, MEDIAN("ResponseStartTiming") tmed, MAX("ResponseStartTiming") tmax FROM hits WHERE "JavaEnable" = 0 GROUP BY "ClientIP", "WatchID" HAVING c > 1 ORDER BY tmed DESC LIMIT 10;
SELECT "ClientIP", "WatchID", COUNT(*) c, MIN("ResponseStartTiming") tmin, APPROX_PERCENTILE_CONT("ResponseStartTiming", 0.95) tp95, MAX("ResponseStartTiming") tmax FROM 'hits' WHERE "JavaEnable" = 0 GROUP BY "ClientIP", "WatchID" HAVING c > 1 ORDER BY tp95 DESC LIMIT 10;
SELECT COUNT(*) AS ShareCount FROM hits WHERE "IsMobile" = 1 AND "MobilePhoneModel" LIKE 'iPhone%' AND "SocialAction" = 'share' AND "SocialSourceNetworkID" IN (5, 12) AND "ClientTimeZone" BETWEEN -5 AND 5 AND regexp_match("Referer", '\/campaign\/(spring|summer)_promo') IS NOT NULL AND CASE WHEN split_part(split_part("URL", 'resolution=', 2), '&', 1) ~ '^\d+$' THEN split_part(split_part("URL", 'resolution=', 2), '&', 1)::INT ELSE 0 END > 1920 AND levenshtein(CAST("UTMSource" AS STRING), CAST("UTMCampaign" AS STRING)) < 3;
+SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Review Comment:
Fixed with #15936
##########
benchmarks/queries/clickbench/extended.sql:
##########
@@ -5,3 +5,4 @@ SELECT "SocialSourceNetworkID", "RegionID", COUNT(*), AVG("Age"), AVG("ParamPric
SELECT "ClientIP", "WatchID", COUNT(*) c, MIN("ResponseStartTiming") tmin, MEDIAN("ResponseStartTiming") tmed, MAX("ResponseStartTiming") tmax FROM hits WHERE "JavaEnable" = 0 GROUP BY "ClientIP", "WatchID" HAVING c > 1 ORDER BY tmed DESC LIMIT 10;
SELECT "ClientIP", "WatchID", COUNT(*) c, MIN("ResponseStartTiming") tmin, APPROX_PERCENTILE_CONT("ResponseStartTiming", 0.95) tp95, MAX("ResponseStartTiming") tmax FROM 'hits' WHERE "JavaEnable" = 0 GROUP BY "ClientIP", "WatchID" HAVING c > 1 ORDER BY tp95 DESC LIMIT 10;
SELECT COUNT(*) AS ShareCount FROM hits WHERE "IsMobile" = 1 AND "MobilePhoneModel" LIKE 'iPhone%' AND "SocialAction" = 'share' AND "SocialSourceNetworkID" IN (5, 12) AND "ClientTimeZone" BETWEEN -5 AND 5 AND regexp_match("Referer", '\/campaign\/(spring|summer)_promo') IS NOT NULL AND CASE WHEN split_part(split_part("URL", 'resolution=', 2), '&', 1) ~ '^\d+$' THEN split_part(split_part("URL", 'resolution=', 2), '&', 1)::INT ELSE 0 END > 1920 AND levenshtein(CAST("UTMSource" AS STRING), CAST("UTMCampaign" AS STRING)) < 3;
+SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Review Comment:
Address with #15936
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072417977
##########
datafusion/expr-common/src/groups_accumulator.rs:
##########
@@ -250,4 +288,30 @@ pub trait GroupsAccumulator: Send {
/// This function is called once per batch, so it should be `O(n)` to
/// compute, not `O(num_groups)`
fn size(&self) -> usize;
+
+ /// Returns `true` if this accumulator supports blocked groups.
+ fn supports_blocked_groups(&self) -> bool {
+ false
+ }
+
+ /// Alter the block size in the accumulator
+ ///
+ /// If the target block size is `None`, it will use a single big
+ /// block(can think it a `Vec`) to manage the state.
+ ///
+ /// If the target block size` is `Some(blk_size)`, it will try to
+ /// set the block size to `blk_size`, and the try will only success
+ /// when the accumulator has supported blocked mode.
+ ///
+ /// NOTICE: After altering block size, all data in previous will be cleared.
Review Comment:
Yes, it will be usually used in `load back + merge` step in spilling:
- Emit the rest blocks at first
- Then switch to `flat mode` and perform sorted aggregation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3347793267
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,387 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ fmt::Debug,
+ iter, mem,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// Internally uses a `Vec<B>` with a `start` offset to track the first active
+/// block. When blocks are popped via [`Self::pop_block`], the block is swapped
+/// out in O(1) using `mem::replace` and the `start` cursor advances, avoiding
+/// the O(n) shift cost of `Vec::remove(0)`.
Review Comment:
This avoids the O(n) shift but it doesn't free the memory, I'm wondering whether a VecDeque would work better here, so inner: VecDeque<B> instead of Vec<B>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4273979782
ci passed and no conflict again, rest things before ready:
- improve tests
- improve `Blocks`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276103930
For clickbench:
```
β QQuery 4 β 294.74 / 301.67 Β±9.80 / 321.09 ms β 367.14 / 374.88 Β±7.17 / 385.68 ms β 1.24x slower β
β QQuery 15 β 358.58 / 373.13 Β±17.99 / 408.00 ms β 419.77 / 441.75 Β±20.14 / 471.01 ms β 1.18x slower β
```
These look like some regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4785204145
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4785188060)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4785188060-647-xvzht 6.12.68+ #1 SMP Sat May 2 07:49:07 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (58691671a13824a41949d3270f163ac0bf7d49d3) to a27f030 (merge-base) [diff](https://github.com/apache/datafusion/compare/a27f030d5829a4460e87dbb8d2a6382c8f9ecd4f..58691671a13824a41949d3270f163ac0bf7d49d3) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4781701805
done in https://github.com/adriangb/datafusion-benchmarking/pull/16
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4750775371
> Yes, and actually I think it make few difference to performance after experiment before (some steps are improved like removing slice of record batch, removing Vec resizing, and some steps are regressed like we need to perform 2 index op, and finally near to no difference will be made), and just a better memory management approach.
I disagree, since the memory management is directly tied to performance via the spilling mechanism when running with memory limits configured. See https://github.com/apache/datafusion/issues/22526#issuecomment-4568611822
The memory overaccounting issues caused by the current design of hash aggregation have a real performance impact in the downstream operators which are either:
* forced to spill prematurely
* outright fail because they don't have enough memory, see https://github.com/apache/datafusion/issues/22861
So I believe the new "blocked" approach will have significant performance improvements in production-like workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4738875697
> > Marking as a draft as I don't think this one is ready to merge quite yet and I am trying to clean up the review / merge queue
>
> Yes, and I think the whole feature will be suitable to push forward after the aggregation refactoring.
>
> Howerver, actually to parts are included in this:
>
> * One part is about refactoring `GroupValues` and `GroupAccumulator`
> * The other part is about applying the blocked logic in aggreagating
>
> How about we split this pr into twos or mores? And push forward the part one (`GroupValues` and `GroupAccumulator`) in parallel with the aggregation refactoring?
@alamb @2010YOUY01 @ariel-miculas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4343768069
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4343604373)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 7.00 / 7.68 Β±0.86 / 9.38 ms β 6.89 / 7.34 Β±0.80 / 8.94 ms β no change β
β QQuery 2 β 147.94 / 148.31 Β±0.33 / 148.87 ms β 150.27 / 150.81 Β±0.35 / 151.34 ms β no change β
β QQuery 3 β 113.77 / 114.52 Β±1.04 / 116.44 ms β 112.57 / 114.06 Β±0.93 / 115.49 ms β no change β
β QQuery 4 β 1272.35 / 1289.93 Β±9.47 / 1297.51 ms β 1279.06 / 1287.61 Β±7.47 / 1300.99 ms β no change β
β QQuery 5 β 172.36 / 173.77 Β±1.11 / 175.21 ms β 172.17 / 173.72 Β±0.99 / 175.27 ms β no change β
β QQuery 6 β 132.86 / 135.21 Β±1.96 / 138.81 ms β 134.27 / 135.56 Β±1.48 / 138.38 ms β no change β
β QQuery 7 β 331.97 / 336.06 Β±2.26 / 338.52 ms β 329.35 / 332.45 Β±2.18 / 334.35 ms β no change β
β QQuery 8 β 114.21 / 115.08 Β±0.89 / 116.78 ms β 113.60 / 114.44 Β±0.68 / 115.53 ms β no change β
β QQuery 9 β 96.72 / 100.31 Β±2.88 / 103.05 ms β 96.54 / 100.04 Β±4.64 / 108.63 ms β no change β
β QQuery 10 β 103.15 / 103.86 Β±0.59 / 104.90 ms β 102.04 / 102.96 Β±0.66 / 103.86 ms β no change β
β QQuery 11 β 879.95 / 890.92 Β±6.51 / 898.80 ms β 873.33 / 885.27 Β±7.12 / 891.83 ms β no change β
β QQuery 12 β 43.37 / 44.06 Β±0.50 / 44.85 ms β 43.27 / 44.66 Β±1.27 / 46.61 ms β no change β
β QQuery 13 β 387.44 / 390.62 Β±3.05 / 394.81 ms β 385.70 / 390.07 Β±2.94 / 394.16 ms β no change β
β QQuery 14 β 1035.29 / 1041.95 Β±3.73 / 1046.72 ms β 1033.68 / 1036.02 Β±2.80 / 1041.51 ms β no change β
β QQuery 15 β 15.13 / 15.70 Β±0.31 / 16.08 ms β 14.94 / 15.76 Β±0.70 / 16.86 ms β no change β
β QQuery 16 β 7.61 / 7.75 Β±0.20 / 8.14 ms β 7.29 / 7.45 Β±0.23 / 7.90 ms β no change β
β QQuery 17 β 221.34 / 222.78 Β±1.01 / 224.21 ms β 218.09 / 221.31 Β±1.63 / 222.40 ms β no change β
β QQuery 18 β 126.29 / 127.09 Β±0.73 / 128.19 ms β 125.42 / 126.53 Β±0.80 / 127.90 ms β no change β
β QQuery 19 β 154.15 / 155.01 Β±1.18 / 157.36 ms β 152.93 / 153.92 Β±0.66 / 155.00 ms β no change β
β QQuery 20 β 13.28 / 13.64 Β±0.38 / 14.33 ms β 12.95 / 13.23 Β±0.25 / 13.60 ms β no change β
β QQuery 21 β 19.38 / 19.71 Β±0.36 / 20.38 ms β 19.09 / 19.69 Β±1.00 / 21.69 ms β no change β
β QQuery 22 β 474.17 / 481.32 Β±6.53 / 492.97 ms β 471.26 / 477.65 Β±5.56 / 484.10 ms β no change β
β QQuery 23 β 1089.14 / 1095.14 Β±4.66 / 1102.99 ms β 1083.49 / 1087.13 Β±4.69 / 1095.88 ms β no change β
β QQuery 24 β 713.12 / 715.91 Β±3.47 / 722.73 ms β 700.60 / 708.53 Β±5.74 / 715.25 ms β no change β
β QQuery 25 β 332.76 / 335.08 Β±1.62 / 337.59 ms β 329.47 / 332.35 Β±1.89 / 334.65 ms β no change β
β QQuery 26 β 78.59 / 78.95 Β±0.29 / 79.45 ms β 77.25 / 77.97 Β±0.45 / 78.54 ms β no change β
β QQuery 27 β 7.01 / 7.29 Β±0.28 / 7.75 ms β 7.02 / 7.13 Β±0.18 / 7.49 ms β no change β
β QQuery 28 β 149.11 / 150.10 Β±1.31 / 152.60 ms β 146.72 / 148.75 Β±1.73 / 151.64 ms β no change β
β QQuery 29 β 272.26 / 275.71 Β±4.54 / 284.70 ms β 270.94 / 273.02 Β±1.80 / 275.81 ms β no change β
β QQuery 30 β 41.51 / 42.07 Β±0.32 / 42.45 ms β 40.69 / 41.52 Β±0.58 / 42.22 ms β no change β
β QQuery 31 β 165.98 / 167.33 Β±0.91 / 168.55 ms β 164.89 / 168.19 Β±2.68 / 172.95 ms β no change β
β QQuery 32 β 13.52 / 13.81 Β±0.34 / 14.48 ms β 13.34 / 13.53 Β±0.15 / 13.75 ms β no change β
β QQuery 33 β 139.05 / 141.23 Β±2.13 / 144.38 ms β 138.72 / 141.17 Β±2.31 / 145.35 ms β no change β
β QQuery 34 β 7.13 / 7.32 Β±0.26 / 7.83 ms β 7.14 / 7.27 Β±0.14 / 7.55 ms β no change β
β QQuery 35 β 101.18 / 102.82 Β±1.08 / 104.58 ms β 100.00 / 100.60 Β±0.46 / 101.39 ms β no change β
β QQuery 36 β 7.13 / 7.26 Β±0.07 / 7.35 ms β 6.76 / 6.95 Β±0.11 / 7.11 ms β no change β
β QQuery 37 β 8.46 / 8.54 Β±0.08 / 8.67 ms β 8.37 / 9.39 Β±1.46 / 12.28 ms β 1.10x slower β
β QQuery 38 β 86.54 / 87.63 Β±1.02 / 88.94 ms β 85.31 / 87.11 Β±2.36 / 91.64 ms β no change β
β QQuery 39 β 118.80 / 122.53 Β±4.97 / 132.15 ms β 115.77 / 119.89 Β±4.67 / 128.90 ms β no change β
β QQuery 40 β 102.34 / 108.67 Β±3.60 / 113.16 ms β 104.20 / 105.78 Β±2.27 / 110.21 ms β no change β
β QQuery 41 β 14.60 / 14.78 Β±0.18 / 15.10 ms β 14.45 / 14.70 Β±0.26 / 15.20 ms β no change β
β QQuery 42 β 106.33 / 107.75 Β±1.30 / 109.51 ms β 106.93 / 109.37 Β±2.43 / 113.47 ms β no change β
β QQuery 43 β 5.90 / 6.05 Β±0.17 / 6.37 ms β 5.64 / 5.76 Β±0.11 / 5.96 ms β no change β
β QQuery 44 β 11.01 / 12.56 Β±2.68 / 17.91 ms β 11.00 / 11.20 Β±0.11 / 11.33 ms β +1.12x faster β
β QQuery 45 β 48.47 / 48.99 Β±0.44 / 49.61 ms β 48.87 / 49.13 Β±0.17 / 49.36 ms β no change β
β QQuery 46 β 8.73 / 8.86 Β±0.14 / 9.11 ms β 8.40 / 8.53 Β±0.16 / 8.84 ms β no change β
β QQuery 47 β 690.52 / 697.24 Β±7.48 / 707.46 ms β 684.67 / 694.47 Β±7.61 / 707.87 ms β no change β
β QQuery 48 β 273.79 / 276.80 Β±2.35 / 280.10 ms β 274.36 / 278.26 Β±3.19 / 281.88 ms β no change β
β QQuery 49 β 249.77 / 250.38 Β±0.71 / 251.77 ms β 247.13 / 248.86 Β±2.02 / 251.68 ms β no change β
β QQuery 50 β 203.92 / 208.89 Β±5.40 / 219.00 ms β 201.63 / 206.69 Β±5.96 / 218.05 ms β no change β
β QQuery 51 β 176.48 / 178.47 Β±1.64 / 180.90 ms β 176.92 / 178.69 Β±1.80 / 181.94 ms β no change β
β QQuery 52 β 107.06 / 108.70 Β±2.30 / 113.19 ms β 106.91 / 108.43 Β±1.72 / 111.76 ms β no change β
β QQuery 53 β 102.10 / 102.45 Β±0.36 / 103.10 ms β 102.69 / 103.13 Β±0.32 / 103.67 ms β no change β
β QQuery 54 β 145.32 / 146.85 Β±2.23 / 151.21 ms β 145.03 / 147.76 Β±2.82 / 153.09 ms β no change β
β QQuery 55 β 105.91 / 106.72 Β±0.85 / 107.97 ms β 105.78 / 106.75 Β±0.73 / 107.71 ms β no change β
β QQuery 56 β 139.36 / 139.99 Β±0.65 / 140.91 ms β 139.93 / 141.00 Β±1.03 / 142.26 ms β no change β
β QQuery 57 β 166.40 / 167.91 Β±1.65 / 171.12 ms β 166.59 / 168.71 Β±2.07 / 172.68 ms β no change β
β QQuery 58 β 265.07 / 267.43 Β±1.33 / 268.84 ms β 264.50 / 265.44 Β±0.66 / 266.41 ms β no change β
β QQuery 59 β 196.64 / 198.55 Β±2.36 / 203.09 ms β 195.43 / 197.95 Β±1.49 / 199.77 ms β no change β
β QQuery 60 β 141.00 / 142.00 Β±0.81 / 142.88 ms β 140.10 / 141.38 Β±1.09 / 143.22 ms β no change β
β QQuery 61 β 13.62 / 13.75 Β±0.20 / 14.15 ms β 13.34 / 13.47 Β±0.20 / 13.87 ms β no change β
β QQuery 62 β 864.82 / 871.16 Β±3.80 / 875.97 ms β 857.81 / 863.66 Β±5.99 / 873.88 ms β no change β
β QQuery 63 β 102.51 / 105.46 Β±3.63 / 112.11 ms β 101.81 / 103.52 Β±1.47 / 105.73 ms β no change β
β QQuery 64 β 660.64 / 666.47 Β±3.88 / 672.36 ms β 655.08 / 661.50 Β±5.23 / 670.81 ms β no change β
β QQuery 65 β 245.34 / 247.38 Β±3.24 / 253.82 ms β 242.15 / 247.07 Β±3.18 / 252.16 ms β no change β
β QQuery 66 β 215.12 / 224.49 Β±12.64 / 249.50 ms β 214.34 / 223.14 Β±10.22 / 241.20 ms β no change β
β QQuery 67 β 294.46 / 308.03 Β±15.10 / 333.37 ms β 290.38 / 299.10 Β±7.70 / 311.26 ms β no change β
β QQuery 68 β 8.90 / 9.09 Β±0.28 / 9.64 ms β 8.79 / 9.02 Β±0.21 / 9.42 ms β no change β
β QQuery 69 β 99.38 / 102.65 Β±3.24 / 108.88 ms β 96.89 / 97.59 Β±0.42 / 98.16 ms β no change β
β QQuery 70 β 317.32 / 329.13 Β±9.17 / 341.99 ms β 311.39 / 320.77 Β±7.32 / 330.29 ms β no change β
β QQuery 71 β 135.54 / 136.94 Β±1.38 / 139.50 ms β 132.97 / 135.82 Β±3.75 / 143.24 ms β no change β
β QQuery 72 β 593.07 / 598.81 Β±3.20 / 602.94 ms β 578.99 / 589.68 Β±8.80 / 605.61 ms β no change β
β QQuery 73 β 6.90 / 7.08 Β±0.24 / 7.56 ms β 6.68 / 6.86 Β±0.18 / 7.17 ms β no change β
β QQuery 74 β 553.70 / 564.78 Β±6.99 / 575.31 ms β 545.71 / 563.15 Β±15.99 / 590.98 ms β no change β
β QQuery 75 β 268.27 / 271.04 Β±2.47 / 275.31 ms β 264.18 / 267.29 Β±2.90 / 271.55 ms β no change β
β QQuery 76 β 129.49 / 131.12 Β±2.25 / 135.56 ms β 128.38 / 128.83 Β±0.45 / 129.65 ms β no change β
β QQuery 77 β 187.39 / 188.10 Β±0.48 / 188.64 ms β 186.79 / 189.13 Β±3.07 / 194.97 ms β no change β
β QQuery 78 β 330.92 / 332.48 Β±1.11 / 334.23 ms β 326.54 / 329.54 Β±2.05 / 332.11 ms β no change β
β QQuery 79 β 227.68 / 230.77 Β±2.20 / 233.58 ms β 227.07 / 231.02 Β±4.71 / 240.09 ms β no change β
β QQuery 80 β 318.82 / 320.86 Β±1.69 / 322.93 ms β 315.83 / 319.58 Β±2.99 / 324.81 ms β no change β
β QQuery 81 β 26.08 / 26.50 Β±0.48 / 27.25 ms β 25.85 / 26.24 Β±0.21 / 26.44 ms β no change β
β QQuery 82 β 38.99 / 39.36 Β±0.31 / 39.86 ms β 38.54 / 38.96 Β±0.44 / 39.70 ms β no change β
β QQuery 83 β 37.32 / 38.16 Β±1.13 / 40.36 ms β 36.67 / 36.79 Β±0.13 / 37.04 ms β no change β
β QQuery 84 β 46.19 / 46.41 Β±0.16 / 46.66 ms β 46.41 / 46.52 Β±0.13 / 46.76 ms β no change β
β QQuery 85 β 142.18 / 143.91 Β±2.31 / 148.47 ms β 139.51 / 142.40 Β±2.73 / 147.38 ms β no change β
β QQuery 86 β 37.65 / 38.19 Β±0.38 / 38.82 ms β 37.09 / 37.48 Β±0.61 / 38.70 ms β no change β
β QQuery 87 β 3.59 / 3.71 Β±0.16 / 4.03 ms β 3.50 / 3.62 Β±0.15 / 3.92 ms β no change β
β QQuery 88 β 100.75 / 101.06 Β±0.22 / 101.39 ms β 99.48 / 100.82 Β±1.74 / 104.25 ms β no change β
β QQuery 89 β 116.12 / 119.59 Β±4.85 / 128.95 ms β 116.53 / 119.39 Β±4.81 / 128.95 ms β no change β
β QQuery 90 β 22.93 / 23.33 Β±0.23 / 23.55 ms β 22.84 / 23.70 Β±0.47 / 24.14 ms β no change β
β QQuery 91 β 58.43 / 59.35 Β±0.74 / 60.49 ms β 58.12 / 59.01 Β±0.59 / 59.96 ms β no change β
β QQuery 92 β 56.87 / 58.00 Β±1.82 / 61.60 ms β 56.48 / 57.02 Β±0.45 / 57.80 ms β no change β
β QQuery 93 β 179.52 / 180.80 Β±1.01 / 182.64 ms β 177.44 / 181.30 Β±3.77 / 188.27 ms β no change β
β QQuery 94 β 60.88 / 62.00 Β±1.38 / 64.66 ms β 60.72 / 61.06 Β±0.23 / 61.39 ms β no change β
β QQuery 95 β 126.06 / 126.85 Β±0.62 / 127.95 ms β 125.09 / 126.08 Β±0.85 / 127.60 ms β no change β
β QQuery 96 β 70.70 / 72.68 Β±1.40 / 75.04 ms β 68.74 / 69.52 Β±0.56 / 70.17 ms β no change β
β QQuery 97 β 116.86 / 118.08 Β±1.01 / 119.28 ms β 116.88 / 118.33 Β±1.68 / 121.58 ms β no change β
β QQuery 98 β 150.71 / 152.14 Β±1.19 / 153.54 ms β 148.32 / 150.96 Β±2.47 / 155.34 ms β no change β
β QQuery 99 β 10729.03 / 10820.23 Β±69.00 / 10906.36 ms β 10684.37 / 10794.78 Β±110.72 / 10962.14 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 30484.02ms β
β Total Time (intermeidate-result-blocked-approach) β 30326.75ms β
β Average Time (HEAD) β 307.92ms β
β Average Time (intermeidate-result-blocked-approach) β 306.33ms β
β Queries Faster β 1 β
β Queries Slower β 1 β
β Queries with No Change β 97 β
β Queries with Failure β 0 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.4 GiB |
| Avg memory | 5.7 GiB |
| CPU user | 242.0s |
| CPU sys | 7.7s |
| Peak spill | 0 B |
**tpcds β branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 6.4 GiB |
| Avg memory | 5.7 GiB |
| CPU user | 240.7s |
| CPU sys | 7.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5413135326
> @2010YOUY01 So I started implementing locally blocked for multi group by and was able to implement it with something I'm comfortable, but the added helpers are where the complexity exists, those helpers are also seems useful to be in arrow
>
> my implementation ditched the emit to first and emit all and instead only have emit next block, since it will simplify implementations to not have enums and a lot of generics like this PR have to support both
>
> The problem with that is that this is breaking change that cannot be done incrementally, like, all the helpers can be created in different pr, but changing to blocked implementation cannot be done incremently.
>
> I would like to jump on a call with you two and discuss this further.
I am available typically most days US Eastern time. In terms of aligning time zones, Early morning is good for me (like 5 or 5:30 AM Eastern time), though I am not sure what timezone @2010YOUY01 or @Rachelint are in or are up for a call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5409636846
@2010YOUY01 So I started implementing locally blocked for multi group by and was able to implement it with something I'm comfortable, but the added helpers are where the complexity exists, those helpers are also seems useful to be in arrow
my implementation ditched the emit to first and emit all and instead only have emit next block, since it will simplify implementations to not have enums and a lot of generics like this PR have to support both
The problem with that is that this is breaking change that cannot be done incrementally, like, all the helpers can be created in different pr, but changing to blocked implementation cannot be done incremently.
I would like to jump on a call with you two and discuss this further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3796078674
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/block_store/blocked.rs:
##########
@@ -0,0 +1,436 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Blocked [`BlockStore`] implementation.
+
+use std::{
+ fmt::Debug,
+ mem,
+ ops::{Index, IndexMut},
+};
+
+use crate::aggregate::groups_accumulator::block_store::{Block, BlockStore};
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// Blocks are popped in FIFO order by keeping a cursor into `inner`. Popping a
+/// block swaps it out in O(1) using `mem::take` and advances the cursor,
+/// avoiding the O(n) shift cost of `Vec::remove(0)`.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct BlockedBlockStore<B: Block> {
Review Comment:
I've tried to copy the code `ArrowBytesMap` to `BlockedArrowBytesMap` where the `Block` is:
```rust
#[derive(Debug)]
struct BufferAndOffsetBlock<O: OffsetSizeTrait> {
/// In progress arrow `Buffer` containing all values
buffer: BufferBuilder<u8>,
/// Offsets into `buffer` for each distinct value. These offsets as used
/// directly to create the final `GenericBinaryArray`. The `i`th string is
/// stored in the range `offsets[i]..offsets[i+1]` in `buffer`. Null values
/// are stored as a zero length string.
offsets: Vec<O>,
}
impl<O: OffsetSizeTrait> Default for BufferAndOffsetBlock<O> {
fn default() -> Self {
Self::new(INITIAL_BUFFER_CAPACITY)
}
}
impl<O: OffsetSizeTrait> Block for BufferAndOffsetBlock<O> {
type T = Vec<u8>;
fn new(capacity: usize) -> Self {
Self {
buffer: BufferBuilder::new(capacity),
offsets: vec![O::default()], // first offset is always 0
}
}
fn fill_default_value(&mut self, fill_len: usize, default_value: Self::T) {
for _ in 0..fill_len {
self.buffer.append_slice(&default_value);
}
let mut last_offset = *self.offsets.last().unwrap();
let len_in_bytes = O::usize_as(default_value.len());
for _ in 0..fill_len {
self.offsets.push(last_offset + len_in_bytes);
last_offset = last_offset + len_in_bytes;
}
}
fn len(&self) -> usize {
self.offsets.len() - 1
}
}
```
but then I realized the `fill_default_value` will not serve any benefit when called from resize since I will have to update the offsets (zero cost, O(1)) and add to the buffer - expensive since it requires shifting the bytes (unless the default value is 0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3080550313
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ collections::VecDeque,
+ fmt::Debug,
+ iter,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct Blocks<B: Block> {
+ inner: VecDeque<B>,
Review Comment:
I think it would be nice to avoid the `VecDeque` as I believe it is relatively slow to index (because of the `%`).
I think we can use a start offset instead during pop (and increment it), replace the block with an empty one to "pop" it and reclaim the memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3084020685
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ collections::VecDeque,
+ fmt::Debug,
+ iter,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct Blocks<B: Block> {
+ inner: VecDeque<B>,
Review Comment:
Yes, it is better to use `Vec<T>` and I tried it when I still see this a performance improvement feature.
However, after many tries, I found it actually can't help dafafusion run faster (it is only something can help to better memory management)... And I finally switch to use `VecDeque` for simplicity...
The experiments can be saw in this archived branch:
https://github.com/Rachelint/arrow-datafusion/compare/intermeidate-result-blocked-approach-bak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4334373967
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4334353557)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4334353557-1860-dvb85 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (a84d71cf9cc8161aca8398c329749d00137965d3) to bbf67d9 (merge-base) [diff](https://github.com/apache/datafusion/compare/bbf67d999c0dcbaf47956233a5d8e78458c13ffa..a84d71cf9cc8161aca8398c329749d00137965d3) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4334409210
Some build problems introduced by count distinct pr, fixing.
And after that, the pr seems can be ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2852744312
> π€: Benchmark completed
> Details
It is really suprised that it show slower?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075086531
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,20 +94,27 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
Review Comment:
this seems rather small? won't larger values have lower overhead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermeidate result blocked approach sketch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2847864538
Starting to check this out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075198294
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,20 +94,27 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
Review Comment:
I found the hash compute for primitive is actually non-trivial...
And in high cardinality case, the duplicated hash compute led by rehash is really expansive!
I am experimenting about also save hash in hashtable like multi group by. Actually I have found improvement in the new added query in `extended.sql`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2888275782
> we could enforce blocksize being a power of two so we can avoid the expensive div and mod operations?
Yes, I am trying to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2887367750
> @alamb @Dandandan Hi, this pr is ready again now.
I will do so as soon as possible (hopefully later today but probably more like Sunday morning as I am away Saturday / tomorrow morning)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2888827881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890507769
> @alamb @Dandandan It is sorry that, I still working on other some new ideas about this pr, so switch to draft again.
No worries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2907882330
> I wonder what the plan is for this PR?
>
> From what I understand, it currently improves performance for aggregates with large numbers of groups, but (slightly) slows down aggregates for smaller numbers of groups. I think this is due to accessing group storage via two indirections (block index / actual index)
>
> It seems like the [proposal](https://github.com/apache/datafusion/pull/15591#issuecomment-2890775426) is to have some sort of adaptive structure that uses one part indexes for small numbers of groups and then switches to two part indexes for larger numbers.
I am experimenting if something we can do basing on this pr to improve performance more, like [memory reuse] (https://github.com/apache/datafusion/pull/15591#issuecomment-2890663260). Actually #16135 is part of the attempt.
My biggest concern is if we can get more obvious improvement to make the change worthy...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Impl intermeidate result blocked approach sketch [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2785518608
Hi @Rachelint I think I have a alternative proposal that seems relatively easy to implement.
I'll share it with you once I have some time to validate the design (probably this evening).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072857296
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -43,8 +46,8 @@ where
T: ArrowPrimitiveType + Send,
F: Fn(&mut T::Native, T::Native) + Send + Sync,
{
- /// values per group, stored as the native type
- values: Vec<T::Native>,
+ /// Values per group, stored as the native type
Review Comment:
Addressed.
I have defined `Blocks` in a dedicated moduel `blocks.rs`, and simplify all related codes using it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848551267
> > π€: Benchmark completed
>
> (I am surprised this PR didn't yield better results, I am reruning now to see if the results are reproducable
I am back from holiday, and continue to work on this today.
Because for simplicity for the initial pr, I just:
- Implement the blocked version `GroupValuesPrimitive` for `GroupValues`
- and `PrimitiveGroupsAccumulator` for `GroupAccumulator`
And actually still don't have such a simple query can show the improvement now...
And for showing the improvement, I add an new query in `clickbench/extended.sql`:
```sql
SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072440427
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:
##########
@@ -212,7 +229,66 @@ impl NullState {
///
/// resets the internal state appropriately
pub fn build(&mut self, emit_to: EmitTo) -> NullBuffer {
- let nulls: BooleanBuffer = self.seen_values.finish();
+ self.seen_values.emit(emit_to)
+ }
+}
+
+/// Structure marking if accumulating groups are seen at least one
+pub trait SeenValues: Default + Debug + Send {
+ fn resize(&mut self, total_num_groups: usize, default_value: bool);
+
+ fn set_bit(&mut self, block_id: u32, block_offset: u64, value: bool);
+
+ fn emit(&mut self, emit_to: EmitTo) -> NullBuffer;
+
+ fn capacity(&self) -> usize;
+}
+
+/// [`SeenValues`] for `flat groups input`
+///
+/// The `flat groups input` are organized like:
+///
+/// ```text
+/// row_0 group_index_0
+/// row_1 group_index_1
+/// row_2 group_index_2
+/// ...
+/// row_n group_index_n
+/// ```
+///
+/// If `row_x group_index_x` is not filtered(`group_index_x` is seen)
+/// `seen_values[group_index_x]` will be set to `true`.
+///
+/// For `set_bit(block_id, block_offset, value)`, `block_id` is unused,
Review Comment:
Addressed.
I add related comments in `GroupsAccumulator::supports_blocked_groups` and `GroupValues::supports_blocked_groups`, also link it in `HashAggregateStream`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072440207
##########
datafusion/expr-common/src/groups_accumulator.rs:
##########
@@ -17,29 +17,52 @@
//! Vectorized [`GroupsAccumulator`]
+use std::collections::VecDeque;
+
use arrow::array::{ArrayRef, BooleanArray};
-use datafusion_common::{not_impl_err, Result};
+use datafusion_common::{not_impl_err, DataFusionError, Result};
/// Describes how many rows should be emitted during grouping.
#[derive(Debug, Clone, Copy)]
pub enum EmitTo {
- /// Emit all groups
+ /// Emit all groups, will clear all existing group indexes
All,
/// Emit only the first `n` groups and shift all existing group
/// indexes down by `n`.
///
/// For example, if `n=10`, group_index `0, 1, ... 9` are emitted
/// and group indexes `10, 11, 12, ...` become `0, 1, 2, ...`.
First(usize),
+ /// Emit next block in the blocked managed groups
+ ///
+ /// Similar as `Emit::All`, will also clear all existing group indexes
+ NextBlock,
}
impl EmitTo {
+ /// Remove and return `needed values` from `values`.
+ pub fn take_needed<T>(
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072440252
##########
datafusion/functions-aggregate/src/average.rs:
##########
@@ -667,8 +668,8 @@ where
partial_counts,
opt_filter,
total_num_groups,
- |group_index, partial_count| {
- self.counts[group_index] += partial_count;
+ |_, group_index, partial_count| {
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach sketch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072019575
##########
datafusion/common/src/config.rs:
##########
@@ -405,6 +405,18 @@ config_namespace! {
/// in joins can reduce memory usage when joining large
/// tables with a highly-selective join filter, but is also slightly slower.
pub enforce_batch_size_in_joins: bool, default = false
+
Review Comment:
Do you expect users will ever disable this feature? Or does this setting exist as an "escape" valve in case we hit a problem with the new behavior and want to go back?
##########
benchmarks/queries/clickbench/extended.sql:
##########
@@ -5,3 +5,4 @@ SELECT "SocialSourceNetworkID", "RegionID", COUNT(*), AVG("Age"), AVG("ParamPric
SELECT "ClientIP", "WatchID", COUNT(*) c, MIN("ResponseStartTiming") tmin, MEDIAN("ResponseStartTiming") tmed, MAX("ResponseStartTiming") tmax FROM hits WHERE "JavaEnable" = 0 GROUP BY "ClientIP", "WatchID" HAVING c > 1 ORDER BY tmed DESC LIMIT 10;
SELECT "ClientIP", "WatchID", COUNT(*) c, MIN("ResponseStartTiming") tmin, APPROX_PERCENTILE_CONT("ResponseStartTiming", 0.95) tp95, MAX("ResponseStartTiming") tmax FROM 'hits' WHERE "JavaEnable" = 0 GROUP BY "ClientIP", "WatchID" HAVING c > 1 ORDER BY tp95 DESC LIMIT 10;
SELECT COUNT(*) AS ShareCount FROM hits WHERE "IsMobile" = 1 AND "MobilePhoneModel" LIKE 'iPhone%' AND "SocialAction" = 'share' AND "SocialSourceNetworkID" IN (5, 12) AND "ClientTimeZone" BETWEEN -5 AND 5 AND regexp_match("Referer", '\/campaign\/(spring|summer)_promo') IS NOT NULL AND CASE WHEN split_part(split_part("URL", 'resolution=', 2), '&', 1) ~ '^\d+$' THEN split_part(split_part("URL", 'resolution=', 2), '&', 1)::INT ELSE 0 END > 1920 AND levenshtein(CAST("UTMSource" AS STRING), CAST("UTMCampaign" AS STRING)) < 3;
+SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Review Comment:
If we are going to add a new query to the extended benchmarks, can we please also document the query here?
https://github.com/apache/datafusion/tree/main/benchmarks/queries/clickbench#extended-queries
?
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -53,10 +56,20 @@ where
starting_value: T::Native,
/// Track nulls in the input / filters
- null_state: NullState,
+ null_state: NullStateAdapter,
/// Function that computes the primitive result
prim_fn: F,
+
+ /// Block size of current `GroupAccumulator` if exist:
+ /// - If `None`, it means block optimization is disabled,
+ /// all `group values`` will be stored in a single `Vec`
+ ///
+ /// - If `Some(blk_size)`, it means block optimization is enabled,
Review Comment:
I wonder what you think about not keeping the code around to support both paths, but instead always used the blocked approach? It seems to me like in the long term if we try to support both paths it will be hard to ensure they are all tested and thus will result in subtle bugs. Keeping as few combinations as possible seems more maintainable to me
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -198,4 +232,28 @@ where
fn size(&self) -> usize {
self.values.capacity() * size_of::<T::Native>() + self.null_state.size()
}
+
+ fn supports_blocked_groups(&self) -> bool {
+ true
+ }
+
+ fn alter_block_size(&mut self, block_size: Option<usize>) -> Result<()> {
+ self.values.clear();
+ self.null_state = NullStateAdapter::new(block_size);
+ self.block_size = block_size;
+
+ Ok(())
+ }
+}
+
+impl<N: ArrowNativeTypeOp> Block for Vec<N> {
Review Comment:
this might make sense to put in the same module of as the definition of `Block` so it was easier to find
##########
datafusion/expr-common/src/groups_accumulator.rs:
##########
@@ -17,29 +17,52 @@
//! Vectorized [`GroupsAccumulator`]
+use std::collections::VecDeque;
+
use arrow::array::{ArrayRef, BooleanArray};
-use datafusion_common::{not_impl_err, Result};
+use datafusion_common::{not_impl_err, DataFusionError, Result};
/// Describes how many rows should be emitted during grouping.
#[derive(Debug, Clone, Copy)]
pub enum EmitTo {
- /// Emit all groups
+ /// Emit all groups, will clear all existing group indexes
All,
/// Emit only the first `n` groups and shift all existing group
/// indexes down by `n`.
///
/// For example, if `n=10`, group_index `0, 1, ... 9` are emitted
/// and group indexes `10, 11, 12, ...` become `0, 1, 2, ...`.
First(usize),
+ /// Emit next block in the blocked managed groups
+ ///
+ /// Similar as `Emit::All`, will also clear all existing group indexes
+ NextBlock,
}
impl EmitTo {
+ /// Remove and return `needed values` from `values`.
+ pub fn take_needed<T>(
Review Comment:
Can you document what the `is_blocked_groups` parameter means too?
##########
datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs:
##########
@@ -81,28 +85,49 @@ hash_float!(f16, f32, f64);
pub struct GroupValuesPrimitive<T: ArrowPrimitiveType> {
/// The data type of the output array
data_type: DataType,
+
/// Stores the group index based on the hash of its value
///
/// We don't store the hashes as hashing fixed width primitives
/// is fast enough for this not to benefit performance
- map: HashTable<usize>,
+ map: HashTable<u64>,
+
/// The group index of the null value if any
- null_group: Option<usize>,
+ null_group: Option<u64>,
+
/// The values for each group index
- values: Vec<T::Native>,
+ values: VecDeque<Vec<T::Native>>,
+
/// The random state used to generate hashes
random_state: RandomState,
+
+ /// Block size of current `GroupValues` if exist:
+ /// - If `None`, it means block optimization is disabled,
+ /// all `group values`` will be stored in a single `Vec`
+ ///
+ /// - If `Some(blk_size)`, it means block optimization is enabled,
+ /// `group values` will be stored in multiple `Vec`s, and each
+ /// `Vec` if of `blk_size` len, and we call it a `block`
+ ///
+ block_size: Option<usize>,
Review Comment:
See my comment above about making a `Blocks` struct which I think would avoid some non trivial duplication
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -43,8 +46,8 @@ where
T: ArrowPrimitiveType + Send,
F: Fn(&mut T::Native, T::Native) + Send + Sync,
{
- /// values per group, stored as the native type
- values: Vec<T::Native>,
+ /// Values per group, stored as the native type
Review Comment:
Something I noticed several times was `VecDeqeue<B:BlocK>` being used -- did you consider making a special struct for this? Something like
```rust
struct Blocks<B: Block> {
inner: VecDeque<Block>
}
impl Blocks<B: Block> {
// resize filling all to default value
fn resize(&mut self, num_values: usize, default_value: Block::T) { .. }
fn ensure_room_enough_for_blocks(...)
}
```
That would make it easier to understand and document how lists of blocks are managed and would make it easier to read code that used it. It would also make `ensure_room_enough_for_blocks` more discoverable
You might also be able to encapsulate the `block_size` parameter in `Blocks` which would make migrating other GroupsAccumulators easier
##########
datafusion/common/src/config.rs:
##########
@@ -405,6 +405,18 @@ config_namespace! {
/// in joins can reduce memory usage when joining large
/// tables with a highly-selective join filter, but is also slightly slower.
pub enforce_batch_size_in_joins: bool, default = false
+
+ /// Should DataFusion use the the blocked approach to manage the groups
+ /// values and their related states in accumulators.
+ /// By default, the blocked approach will be used. And the blocked approach
+ /// allocates capacity for the block based on a predefined block size firstly.
+ /// When the block reaches its limit, we allocate a new block (also with
+ /// the same predefined block size based capacity) instead of expanding
+ /// the current one and copying the data.
+ /// If setting this flag to `false`, will fall-back to use the single approach,
+ /// values are managed within a single large block(can think of it as a Vec).
+ /// As this block grows, it often triggers numerous copies, resulting in poor performance.
Review Comment:
Here is a suggestion that focuses more on the external facing results rather than the internal implementation
```suggestion
/// Should DataFusion use a blocked approach to manage grouping state.
/// By default, the blocked approach is used which
/// allocates capacity based on a predefined block size firstly.
/// When the block reaches its limit, we allocate a new block (also with
/// the same predefined block size based capacity) instead of expanding
/// the current one and copying the data.
/// If `false`, a single allocation approach is used, where
/// values are managed within a single large memory block.
/// As this block grows, it often triggers numerous copies, resulting in poor performance.
```
##########
datafusion/functions-aggregate/src/correlation.rs:
##########
@@ -448,6 +448,9 @@ impl GroupsAccumulator for CorrelationGroupsAccumulator {
let n = match emit_to {
EmitTo::All => self.count.len(),
EmitTo::First(n) => n,
+ EmitTo::NextBlock => {
+ unreachable!("this accumulator still not support blocked groups")
Review Comment:
Instead of `unreachable!` which will panic, I think it would make more sense to return an internal error here to signal it is not expected:
```suggestion
internal_err!("correlation does not support blocked groups")
```
Similarly below
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
@@ -982,6 +1099,9 @@ impl GroupedHashAggregateStream {
&& self.update_memory_reservation().is_err()
{
assert_ne!(self.mode, AggregateMode::Partial);
+ // TODO: support spilling when blocked group optimization is on
Review Comment:
We may want to file a ticket to track this -- but I think in general figuring out how to handle spilling for hashing in a better way is worth considering so maybe this particular task would become irrelevant
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs:
##########
@@ -507,3 +509,157 @@ pub(crate) fn slice_and_maybe_filter(
Ok(sliced_arrays)
}
}
+
+// ===============================================
+// Useful tools for group index
+// ===============================================
+
+/// Operations about group index parsing
+///
+/// There are mainly 2 `group index` needing parsing: `flat` and `blocked`.
+///
+/// # Flat group index
+/// `flat group index` format is like:
+///
+/// ```text
+/// | block_offset(64bit) |
+/// ```
+///
+/// It is used in `flat GroupValues/GroupAccumulator`, only a single block
+/// exists, so its `block_id` is always 0, and use all 64 bits to store the
+/// `block offset`.
+///
+/// # Blocked group index
+/// `blocked group index` format is like:
+///
+/// ```text
+/// | block_id(32bit) | block_offset(32bit)
+/// ```
+///
+/// It is used in `blocked GroupValues/GroupAccumulator`, multiple blocks
+/// exist, and we use high 32 bits to store `block_id`, and low 32 bit to
+/// store `block_offset`.
+///
+/// The `get_block_offset` method requires to return `block_offset` as u64,
+/// that is for compatible for `flat group index`'s parsing.
+///
+pub trait GroupIndexOperations: Debug {
Review Comment:
Maybe it would make sense to pull this into a new module
- `datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/operations.rs` perhaps
##########
datafusion/functions-aggregate/src/average.rs:
##########
@@ -667,8 +668,8 @@ where
partial_counts,
opt_filter,
total_num_groups,
- |group_index, partial_count| {
- self.counts[group_index] += partial_count;
+ |_, group_index, partial_count| {
Review Comment:
I think it would help to have a comment here explaining why the block index was ignored (because this accumulator only supports a single block)
##########
datafusion/expr-common/src/groups_accumulator.rs:
##########
@@ -250,4 +288,30 @@ pub trait GroupsAccumulator: Send {
/// This function is called once per batch, so it should be `O(n)` to
/// compute, not `O(num_groups)`
fn size(&self) -> usize;
+
+ /// Returns `true` if this accumulator supports blocked groups.
+ fn supports_blocked_groups(&self) -> bool {
+ false
+ }
+
+ /// Alter the block size in the accumulator
+ ///
+ /// If the target block size is `None`, it will use a single big
+ /// block(can think it a `Vec`) to manage the state.
+ ///
+ /// If the target block size` is `Some(blk_size)`, it will try to
+ /// set the block size to `blk_size`, and the try will only success
+ /// when the accumulator has supported blocked mode.
+ ///
+ /// NOTICE: After altering block size, all data in previous will be cleared.
Review Comment:
Does this mean that all existing accumulators will be cleared?
##########
datafusion/physical-plan/Cargo.toml:
##########
@@ -51,6 +51,7 @@ datafusion-common = { workspace = true, default-features = true }
datafusion-common-runtime = { workspace = true, default-features = true }
datafusion-execution = { workspace = true }
datafusion-expr = { workspace = true }
+datafusion-functions-aggregate-common = { workspace = true }
Review Comment:
I think this new dependency is ok as it doesn't use any specific aggregate implementation which we are trying to avoid
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs:
##########
@@ -507,3 +509,157 @@ pub(crate) fn slice_and_maybe_filter(
Ok(sliced_arrays)
}
}
+
+// ===============================================
+// Useful tools for group index
+// ===============================================
+
+/// Operations about group index parsing
+///
+/// There are mainly 2 `group index` needing parsing: `flat` and `blocked`.
+///
+/// # Flat group index
+/// `flat group index` format is like:
+///
+/// ```text
+/// | block_offset(64bit) |
+/// ```
+///
+/// It is used in `flat GroupValues/GroupAccumulator`, only a single block
+/// exists, so its `block_id` is always 0, and use all 64 bits to store the
+/// `block offset`.
+///
+/// # Blocked group index
+/// `blocked group index` format is like:
+///
+/// ```text
+/// | block_id(32bit) | block_offset(32bit)
+/// ```
+///
+/// It is used in `blocked GroupValues/GroupAccumulator`, multiple blocks
+/// exist, and we use high 32 bits to store `block_id`, and low 32 bit to
+/// store `block_offset`.
+///
+/// The `get_block_offset` method requires to return `block_offset` as u64,
+/// that is for compatible for `flat group index`'s parsing.
+///
+pub trait GroupIndexOperations: Debug {
+ fn pack_index(block_id: u32, block_offset: u64) -> u64;
+
+ fn get_block_id(packed_index: u64) -> u32;
+
+ fn get_block_offset(packed_index: u64) -> u64;
+}
+
+#[derive(Debug)]
+pub struct BlockedGroupIndexOperations;
+
+impl GroupIndexOperations for BlockedGroupIndexOperations {
+ fn pack_index(block_id: u32, block_offset: u64) -> u64 {
+ ((block_id as u64) << 32) | block_offset
+ }
+
+ fn get_block_id(packed_index: u64) -> u32 {
+ (packed_index >> 32) as u32
+ }
+
+ fn get_block_offset(packed_index: u64) -> u64 {
+ (packed_index as u32) as u64
+ }
+}
+
+#[derive(Debug)]
+pub struct FlatGroupIndexOperations;
+
+impl GroupIndexOperations for FlatGroupIndexOperations {
+ fn pack_index(_block_id: u32, block_offset: u64) -> u64 {
+ block_offset
+ }
+
+ fn get_block_id(_packed_index: u64) -> u32 {
+ 0
+ }
+
+ fn get_block_offset(packed_index: u64) -> u64 {
+ packed_index
+ }
+}
+
+// ===============================================
+// Useful tools for block
+// ===============================================
+pub(crate) fn ensure_room_enough_for_blocks<B, F>(
+ blocks: &mut VecDeque<B>,
+ total_num_groups: usize,
+ block_size: usize,
+ new_block: F,
+ default_value: B::T,
+) where
+ B: Block,
+ F: Fn(usize) -> B,
+{
+ // For resize, we need to:
+ // 1. Ensure the blks are enough first
+ // 2. and then ensure slots in blks are enough
+ let (mut cur_blk_idx, exist_slots) = if !blocks.is_empty() {
+ let cur_blk_idx = blocks.len() - 1;
+ let exist_slots = (blocks.len() - 1) * block_size + blocks.back().unwrap().len();
+
+ (cur_blk_idx, exist_slots)
+ } else {
+ (0, 0)
+ };
+
+ // No new groups, don't need to expand, just return
+ if exist_slots >= total_num_groups {
+ return;
+ }
+
+ // 1. Ensure blks are enough
+ let exist_blks = blocks.len();
+ let new_blks = total_num_groups.div_ceil(block_size) - exist_blks;
+ if new_blks > 0 {
+ for _ in 0..new_blks {
+ let block = new_block(block_size);
+ blocks.push_back(block);
+ }
+ }
+
+ // 2. Ensure slots are enough
+ let mut new_slots = total_num_groups - exist_slots;
+
+ // 2.1 Only fill current blk if it may be already enough
+ let cur_blk_rest_slots = block_size - blocks[cur_blk_idx].len();
+ if cur_blk_rest_slots >= new_slots {
+ blocks[cur_blk_idx].fill_default_value(new_slots, default_value.clone());
+ return;
+ }
+
+ // 2.2 Fill current blk to full
+ blocks[cur_blk_idx].fill_default_value(cur_blk_rest_slots, default_value.clone());
+ new_slots -= cur_blk_rest_slots;
+
+ // 2.3 Fill complete blks
+ let complete_blks = new_slots / block_size;
+ for _ in 0..complete_blks {
+ cur_blk_idx += 1;
+ blocks[cur_blk_idx].fill_default_value(block_size, default_value.clone());
+ }
+
+ // 2.4 Fill last blk if needed
+ let rest_slots = new_slots % block_size;
+ if rest_slots > 0 {
+ blocks
+ .back_mut()
+ .unwrap()
+ .fill_default_value(rest_slots, default_value);
+ }
+}
+
+pub(crate) trait Block {
Review Comment:
I think it would be good to add some comments here describing what the trait is for
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:
##########
@@ -212,7 +229,66 @@ impl NullState {
///
/// resets the internal state appropriately
pub fn build(&mut self, emit_to: EmitTo) -> NullBuffer {
- let nulls: BooleanBuffer = self.seen_values.finish();
+ self.seen_values.emit(emit_to)
+ }
+}
+
+/// Structure marking if accumulating groups are seen at least one
+pub trait SeenValues: Default + Debug + Send {
+ fn resize(&mut self, total_num_groups: usize, default_value: bool);
+
+ fn set_bit(&mut self, block_id: u32, block_offset: u64, value: bool);
+
+ fn emit(&mut self, emit_to: EmitTo) -> NullBuffer;
+
+ fn capacity(&self) -> usize;
+}
+
+/// [`SeenValues`] for `flat groups input`
+///
+/// The `flat groups input` are organized like:
+///
+/// ```text
+/// row_0 group_index_0
+/// row_1 group_index_1
+/// row_2 group_index_2
+/// ...
+/// row_n group_index_n
+/// ```
+///
+/// If `row_x group_index_x` is not filtered(`group_index_x` is seen)
+/// `seen_values[group_index_x]` will be set to `true`.
+///
+/// For `set_bit(block_id, block_offset, value)`, `block_id` is unused,
Review Comment:
Something I didn't see documented anywhere was what `block_id` and `block_offset` meant -- maybe we could add something on the HashAggregateStream π€
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276095992
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4276055952)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.21 / 4.50 Β±6.43 / 17.37 ms β 1.20 / 4.46 Β±6.36 / 17.19 ms β no change β
β QQuery 1 β 14.12 / 14.68 Β±0.31 / 15.03 ms β 14.13 / 14.68 Β±0.29 / 14.94 ms β no change β
β QQuery 2 β 44.01 / 44.52 Β±0.30 / 44.79 ms β 45.03 / 45.31 Β±0.40 / 46.09 ms β no change β
β QQuery 3 β 44.15 / 45.42 Β±1.29 / 47.57 ms β 44.51 / 46.98 Β±1.45 / 48.80 ms β no change β
β QQuery 4 β 294.74 / 301.67 Β±9.80 / 321.09 ms β 367.14 / 374.88 Β±7.17 / 385.68 ms β 1.24x slower β
β QQuery 5 β 345.54 / 349.64 Β±2.64 / 353.57 ms β 346.92 / 352.98 Β±4.75 / 358.08 ms β no change β
β QQuery 6 β 5.88 / 6.58 Β±0.42 / 7.13 ms β 5.96 / 9.31 Β±3.29 / 14.27 ms β 1.41x slower β
β QQuery 7 β 16.86 / 17.39 Β±0.40 / 17.89 ms β 17.24 / 17.99 Β±0.77 / 19.45 ms β no change β
β QQuery 8 β 412.54 / 420.67 Β±4.81 / 426.00 ms β 422.41 / 439.77 Β±10.14 / 453.22 ms β no change β
β QQuery 9 β 651.79 / 672.54 Β±12.42 / 686.69 ms β 698.72 / 712.77 Β±9.32 / 723.81 ms β 1.06x slower β
β QQuery 10 β 94.34 / 96.64 Β±3.38 / 103.28 ms β 95.64 / 97.99 Β±3.36 / 104.63 ms β no change β
β QQuery 11 β 107.00 / 108.71 Β±1.31 / 110.14 ms β 107.78 / 109.36 Β±1.54 / 112.15 ms β no change β
β QQuery 12 β 341.81 / 346.09 Β±3.18 / 350.39 ms β 353.63 / 361.21 Β±6.34 / 372.47 ms β no change β
β QQuery 13 β 459.64 / 466.33 Β±5.80 / 475.57 ms β 461.89 / 482.61 Β±15.33 / 502.49 ms β no change β
β QQuery 14 β 350.52 / 354.24 Β±3.72 / 360.43 ms β 354.29 / 358.51 Β±3.71 / 364.87 ms β no change β
β QQuery 15 β 358.58 / 373.13 Β±17.99 / 408.00 ms β 419.77 / 441.75 Β±20.14 / 471.01 ms β 1.18x slower β
β QQuery 16 β 717.58 / 738.10 Β±18.91 / 763.78 ms β 714.12 / 737.50 Β±24.14 / 781.33 ms β no change β
β QQuery 17 β 711.45 / 730.77 Β±17.08 / 760.26 ms β 709.25 / 720.89 Β±8.15 / 731.18 ms β no change β
β QQuery 18 β 1446.88 / 1492.48 Β±35.03 / 1537.37 ms β 1438.10 / 1492.83 Β±36.51 / 1551.26 ms β no change β
β QQuery 19 β 35.71 / 54.65 Β±34.76 / 124.10 ms β 35.83 / 62.61 Β±31.16 / 106.95 ms β 1.15x slower β
β QQuery 20 β 718.19 / 735.83 Β±17.04 / 757.31 ms β 722.66 / 738.47 Β±23.97 / 785.69 ms β no change β
β QQuery 21 β 762.49 / 766.02 Β±3.71 / 772.94 ms β 760.19 / 769.58 Β±9.08 / 786.29 ms β no change β
β QQuery 22 β 1142.54 / 1149.77 Β±7.36 / 1163.89 ms β 1132.74 / 1135.29 Β±1.98 / 1138.65 ms β no change β
β QQuery 23 β 3143.16 / 3164.71 Β±18.18 / 3193.88 ms β 3102.82 / 3127.41 Β±21.13 / 3162.71 ms β no change β
β QQuery 24 β 102.88 / 104.34 Β±1.14 / 105.89 ms β 101.73 / 103.78 Β±1.98 / 107.26 ms β no change β
β QQuery 25 β 139.96 / 141.59 Β±1.89 / 144.50 ms β 139.37 / 140.63 Β±0.89 / 141.79 ms β no change β
β QQuery 26 β 102.28 / 104.35 Β±1.13 / 105.25 ms β 100.86 / 104.87 Β±2.27 / 107.92 ms β no change β
β QQuery 27 β 850.87 / 854.05 Β±2.35 / 857.15 ms β 853.12 / 854.65 Β±1.92 / 858.35 ms β no change β
β QQuery 28 β 3254.86 / 3301.26 Β±30.19 / 3337.06 ms β 3241.25 / 3281.59 Β±20.70 / 3297.95 ms β no change β
β QQuery 29 β 52.28 / 83.30 Β±49.60 / 181.05 ms β 50.87 / 54.86 Β±5.80 / 66.30 ms β +1.52x faster β
β QQuery 30 β 359.61 / 372.13 Β±9.33 / 388.09 ms β 367.06 / 371.13 Β±3.59 / 377.61 ms β no change β
β QQuery 31 β 389.74 / 394.17 Β±4.60 / 401.91 ms β 367.74 / 380.88 Β±7.09 / 387.06 ms β no change β
β QQuery 32 β 1183.58 / 1273.13 Β±58.99 / 1359.97 ms β 1057.41 / 1074.65 Β±16.39 / 1104.98 ms β +1.18x faster β
β QQuery 33 β 1462.15 / 1496.83 Β±35.88 / 1564.06 ms β 1448.29 / 1463.96 Β±10.26 / 1478.79 ms β no change β
β QQuery 34 β 1446.01 / 1473.12 Β±22.66 / 1514.56 ms β 1458.88 / 1465.49 Β±4.43 / 1471.04 ms β no change β
β QQuery 35 β 400.71 / 406.46 Β±4.71 / 414.08 ms β 451.55 / 458.48 Β±5.51 / 467.61 ms β 1.13x slower β
β QQuery 36 β 112.72 / 120.63 Β±4.31 / 124.70 ms β 117.19 / 122.19 Β±3.10 / 125.18 ms β no change β
β QQuery 37 β 49.16 / 49.97 Β±0.84 / 51.46 ms β 48.95 / 51.30 Β±2.06 / 54.29 ms β no change β
β QQuery 38 β 76.74 / 78.06 Β±1.07 / 79.83 ms β 74.92 / 77.43 Β±1.97 / 79.38 ms β no change β
β QQuery 39 β 213.50 / 219.18 Β±5.42 / 228.18 ms β 216.12 / 223.26 Β±5.44 / 232.43 ms β no change β
β QQuery 40 β 25.46 / 27.04 Β±1.93 / 30.79 ms β 24.30 / 25.60 Β±1.43 / 28.23 ms β +1.06x faster β
β QQuery 41 β 20.21 / 21.50 Β±1.29 / 23.78 ms β 19.23 / 20.60 Β±1.65 / 23.78 ms β no change β
β QQuery 42 β 19.37 / 20.61 Β±0.89 / 22.06 ms β 19.77 / 22.35 Β±1.87 / 25.24 ms β 1.08x slower β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 22996.80ms β
β Total Time (intermeidate-result-blocked-approach) β 22952.85ms β
β Average Time (HEAD) β 534.81ms β
β Average Time (intermeidate-result-blocked-approach) β 533.79ms β
β Queries Faster β 3 β
β Queries Slower β 7 β
β Queries with No Change β 33 β
β Queries with Failure β 0 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 116.2s |
| Peak memory | 42.3 GiB |
| Avg memory | 28.9 GiB |
| CPU user | 1083.8s |
| CPU sys | 98.9s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 115.9s |
| Peak memory | 42.3 GiB |
| Avg memory | 30.5 GiB |
| CPU user | 1093.9s |
| CPU sys | 85.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alchemist51 (via GitHub)" <gi...@apache.org>.
alchemist51 commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2726532346
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:
##########
@@ -37,12 +47,10 @@ use datafusion_expr_common::groups_accumulator::EmitTo;
#[derive(Debug)]
pub enum SeenValues {
/// All groups seen so far have seen at least one non-null value
- All {
Review Comment:
probably unintended change
##########
datafusion/common/src/config.rs:
##########
@@ -647,6 +647,16 @@ config_namespace! {
/// the remote end point.
pub objectstore_writer_buffer_size: usize, default = 10 * 1024 * 1024
+ /// Should DataFusion use a blocked approach to manage grouping state.
+ /// By default, the blocked approach is used which
+ /// allocates capacity based on a predefined block size firstly.
+ /// When the block reaches its limit, we allocate a new block (also with
+ /// the same predefined block size based capacity) instead of expanding
+ /// the current one and copying the data.
+ /// If `false`, a single allocation approach is used, where
+ /// values are managed within a single large memory block.
+ /// As this block grows, it often triggers numerous copies, resulting in poor performance.
+ pub enable_aggregation_blocked_groups: bool, default = true
Review Comment:
It should be false by default for now?
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:
##########
@@ -121,26 +136,29 @@ pub struct NullState {
///
/// If `seen_values` is `SeenValues::All`, all groups have seen at least one non null value
seen_values: SeenValues,
-}
-impl Default for NullState {
- fn default() -> Self {
- Self::new()
- }
+ /// Size of one seen values block, can be None if only desire single block
+ block_size: Option<usize>,
+
+ /// phantom data for required type `<O>`
+ _phantom: PhantomData<O>,
}
-impl NullState {
- pub fn new() -> Self {
+impl<O: GroupIndexOperations> NullState<O> {
+ /// Create a new `NullState`
+ pub fn new(block_size: Option<usize>) -> Self {
Self {
seen_values: SeenValues::All { num_values: 0 },
+ block_size,
+ _phantom: PhantomData,
Review Comment:
Didn't quite get the phantom usage
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:
##########
@@ -60,27 +68,34 @@ impl SeenValues {
///
/// The builder is then ensured to have at least `total_num_groups` length,
/// with any new entries initialized to false.
- fn get_builder(&mut self, total_num_groups: usize) -> &mut BooleanBufferBuilder {
- match self {
+ fn get_big_enough_builder(
+ &mut self,
+ total_num_groups: usize,
+ block_size: Option<usize>,
+ ) -> &mut Blocks<BooleanBufferBuilder> {
+ // If `self` is `SeenValues::All`, transition it to `SeenValues::Some` with `num_values trues` firstly,
+ // then return mutable reference to the builder.
+ // If `self` is `SeenValues::Some`, just directly return mutable reference to the builder.
+ let new_block = |block_size: Option<usize>| {
Review Comment:
should we move it to SeenValues::All match call rather?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3749255055
My personal suggestion here is to try and find some way to incrementally move this forward (as we were discussing elsewhere). Our ability to replace the entire group by hash machinery in one go will be very hard, both to review, as well as to track down potential regressions
So I personally recommend focusing on some core usecases first (e.g. single column hash aggregates for Primtive and String/StringViews) and see what you can get working.
We can then slowly add more functionality (multi column grouping, more types, spilling, etc) incrementally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4335145434
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4335018412)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 40.72 / 41.57 Β±1.06 / 43.63 ms β 40.37 / 41.16 Β±1.00 / 43.12 ms β no change β
β QQuery 2 β 20.72 / 21.51 Β±0.62 / 22.55 ms β 21.04 / 21.34 Β±0.25 / 21.62 ms β no change β
β QQuery 3 β 38.35 / 40.09 Β±1.37 / 41.87 ms β 38.43 / 39.34 Β±1.06 / 41.26 ms β no change β
β QQuery 4 β 18.38 / 18.58 Β±0.20 / 18.95 ms β 18.27 / 18.60 Β±0.20 / 18.86 ms β no change β
β QQuery 5 β 47.70 / 50.67 Β±2.39 / 53.39 ms β 48.15 / 50.52 Β±1.69 / 52.22 ms β no change β
β QQuery 6 β 17.39 / 17.67 Β±0.19 / 17.90 ms β 17.48 / 17.60 Β±0.11 / 17.78 ms β no change β
β QQuery 7 β 54.37 / 55.31 Β±0.66 / 56.38 ms β 53.61 / 55.88 Β±1.27 / 57.34 ms β no change β
β QQuery 8 β 48.41 / 48.93 Β±0.52 / 49.84 ms β 48.46 / 48.71 Β±0.27 / 49.18 ms β no change β
β QQuery 9 β 53.68 / 55.26 Β±1.43 / 57.73 ms β 54.43 / 54.68 Β±0.32 / 55.29 ms β no change β
β QQuery 10 β 65.52 / 66.01 Β±0.36 / 66.56 ms β 65.57 / 66.14 Β±0.63 / 67.30 ms β no change β
β QQuery 11 β 14.25 / 14.47 Β±0.12 / 14.60 ms β 13.93 / 14.20 Β±0.19 / 14.40 ms β no change β
β QQuery 12 β 28.14 / 28.25 Β±0.10 / 28.39 ms β 27.65 / 28.07 Β±0.34 / 28.64 ms β no change β
β QQuery 13 β 37.26 / 38.28 Β±0.65 / 39.13 ms β 37.99 / 38.44 Β±0.55 / 39.43 ms β no change β
β QQuery 14 β 27.86 / 28.24 Β±0.21 / 28.44 ms β 28.04 / 28.13 Β±0.06 / 28.19 ms β no change β
β QQuery 15 β 33.52 / 33.86 Β±0.20 / 34.06 ms β 33.62 / 34.41 Β±0.83 / 35.81 ms β no change β
β QQuery 16 β 15.25 / 15.41 Β±0.13 / 15.65 ms β 15.33 / 15.41 Β±0.07 / 15.50 ms β no change β
β QQuery 17 β 77.75 / 78.94 Β±0.78 / 80.14 ms β 82.48 / 83.06 Β±0.48 / 83.70 ms β 1.05x slower β
β QQuery 18 β 75.24 / 76.41 Β±0.99 / 78.24 ms β 77.38 / 78.68 Β±0.95 / 80.02 ms β no change β
β QQuery 19 β 38.18 / 38.52 Β±0.31 / 39.03 ms β 37.65 / 37.98 Β±0.24 / 38.36 ms β no change β
β QQuery 20 β 39.62 / 40.29 Β±0.58 / 41.38 ms β 39.81 / 41.08 Β±2.08 / 45.23 ms β no change β
β QQuery 21 β 62.67 / 64.84 Β±1.62 / 67.72 ms β 63.10 / 64.93 Β±0.98 / 66.01 ms β no change β
β QQuery 22 β 23.22 / 23.76 Β±0.46 / 24.54 ms β 23.86 / 24.50 Β±0.63 / 25.61 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 896.89ms β
β Total Time (intermeidate-result-blocked-approach) β 902.85ms β
β Average Time (HEAD) β 40.77ms β
β Average Time (intermeidate-result-blocked-approach) β 41.04ms β
β Queries Faster β 0 β
β Queries Slower β 1 β
β Queries with No Change β 21 β
β Queries with Failure β 0 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 5.8 GiB |
| Avg memory | 5.0 GiB |
| CPU user | 33.9s |
| CPU sys | 2.4s |
| Peak spill | 0 B |
**tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 5.7 GiB |
| Avg memory | 5.0 GiB |
| CPU user | 34.2s |
| CPU sys | 2.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4343626246
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4343604373)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4343604373-1909-n4jjp 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (47b69765e44dbac7ec4173315b64ca1d55f259dc) to 8f033e4 (merge-base) [diff](https://github.com/apache/datafusion/compare/8f033e411aa4e097a146718e61d90897fe6e9c3d..47b69765e44dbac7ec4173315b64ca1d55f259dc) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4764001581
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v54.0.0 (current)
error: running cargo-doc on crate 'datafusion' failed with output:
-----
Compiling proc-macro2 v1.0.106
Compiling unicode-ident v1.0.24
Compiling quote v1.0.45
Compiling libc v0.2.186
Checking cfg-if v1.0.4
Compiling find-msvc-tools v0.1.9
Compiling shlex v2.0.1
Compiling autocfg v1.5.1
Checking memchr v2.8.2
Compiling syn v2.0.118
Compiling libm v0.2.16
Compiling num-traits v0.2.19
Compiling jobserver v0.1.34
Compiling version_check v0.9.5
Compiling cc v1.2.65
Checking bytes v1.12.0
Compiling zerocopy v0.8.52
Compiling serde_core v1.0.228
Checking once_cell v1.21.4
Compiling getrandom v0.3.4
Checking equivalent v1.0.2
Checking itoa v1.0.18
Checking allocator-api2 v0.2.21
Checking foldhash v0.2.0
Checking hashbrown v0.17.1
Compiling zmij v1.0.21
Checking num-integer v0.1.46
Checking indexmap v2.14.0
Compiling serde v1.0.228
Compiling serde_json v1.0.150
Checking num-bigint v0.4.6
Checking iana-time-zone v0.1.65
Checking siphasher v1.0.3
Compiling pkg-config v0.3.33
Compiling synstructure v0.13.2
Checking phf_shared v0.12.1
Checking chrono v0.4.45
Compiling ahash v0.8.12
Compiling chrono-tz v0.10.4
Checking stable_deref_trait v1.2.1
Checking phf v0.12.1
Checking num-complex v0.4.6
Compiling zstd-sys v2.0.16+zstd.1.5.7
Checking pin-project-lite v0.2.17
Checking smallvec v1.15.2
Checking futures-sink v0.3.32
Checking writeable v0.6.3
Compiling object v0.37.3
Compiling zerocopy-derive v0.8.52
Compiling zerofrom-derive v0.1.7
Compiling serde_derive v1.0.228
Checking zerofrom v0.1.8
Compiling yoke-derive v0.8.2
Checking arrow-schema v59.0.0
Checking half v2.7.1
Checking arrow-buffer v59.0.0
Checking yoke v0.8.3
Compiling zerovec-derive v0.11.3
Compiling displaydoc v0.2.6
Checking arrow-data v59.0.0
Checking arrow-array v59.0.0
Checking futures-core v0.3.32
Checking lexical-util v1.0.7
Checking litemap v0.8.2
Checking zerovec v0.11.6
Compiling zstd-safe v7.2.4
Checking zerotrie v0.2.4
Checking utf8_iter v1.0.4
Checking tinystr v0.8.3
Checking icu_locale_core v2.2.0
Checking potential_utf v0.1.5
Compiling icu_properties_data v2.2.0
Checking arrow-select v59.0.0
Compiling icu_normalizer_data v2.2.0
Checking icu_provider v2.2.0
Checking icu_collections v2.2.0
Compiling tokio-macros v2.7.0
Compiling crc32fast v1.5.0
Compiling semver v1.0.28
Checking tokio v1.52.3
Compiling rustc_version v0.4.1
Checking lexical-write-integer v1.0.6
Checking lexical-parse-integer v1.0.6
Checking futures-channel v0.3.32
Compiling futures-macro v0.3.32
Checking futures-task v0.3.32
Checking slab v0.4.12
Checking simd-adler32 v0.3.9
Checking bitflags v2.13.0
Checking adler2 v2.0.1
Compiling parking_lot_core v0.9.12
Checking futures-io v0.3.32
Checking miniz_oxide v0.8.9
Checking futures-util v0.3.32
Compiling ar_archive_writer v0.5.2
Checking lexical-parse-float v1.0.6
Checking lexical-write-float v1.0.6
Checking icu_normalizer v2.2.0
Checking icu_properties v2.2.0
Compiling psm v0.1.31
Compiling flatbuffers v25.12.19
Checking aho-corasick v1.1.4
Checking scopeguard v1.2.0
Checking base64 v0.22.1
Checking ryu v1.0.23
Compiling getrandom v0.4.3
Checking unicode-segmentation v1.13.3
Checking zlib-rs v0.6.4
Checking zstd v0.13.3
Checking unicode-width v0.2.2
Checking regex-syntax v0.8.11
Checking lock_api v0.4.14
Checking comfy-table v7.2.2
Checking idna_adapter v1.2.2
Checking lexical-core v1.0.6
Checking arrow-ord v59.0.0
Compiling stacker v0.1.24
Checking atoi v2.0.0
Checking percent-encoding v2.3.2
Compiling snap v1.1.1
Checking alloc-no-stdlib v2.0.4
Checking twox-hash v2.1.2
Compiling thiserror v2.0.18
Checking lz4_flex v0.13.1
Checking flate2 v1.1.9
Checking alloc-stdlib v0.2.4
Checking arrow-cast v59.0.0
Checking form_urlencoded v1.2.2
Checking idna v1.1.0
Checking regex-automata v0.4.14
Checking futures-executor v0.3.32
Compiling tracing-attributes v0.1.31
Compiling thiserror-impl v2.0.18
Compiling ring v0.17.14
Checking tracing-core v0.1.36
Checking csv-core v0.1.13
Checking either v1.16.0
Compiling paste v1.0.15
Checking simdutf8 v0.1.5
Checking same-file v1.0.6
Checking itertools v0.14.0
Checking walkdir v2.5.0
Checking csv v1.4.0
Checking regex v1.12.4
Checking tracing v0.1.44
Checking futures v0.3.32
Checking arrow-ipc v59.0.0
Checking url v2.5.8
Checking brotli-decompressor v5.0.3
Checking parking_lot v0.12.5
Compiling async-trait v0.1.89
Compiling recursive-proc-macro-impl v0.1.1
Checking http v1.4.2
Checking getrandom v0.2.17
Checking humantime v2.3.0
Checking untrusted v0.9.0
Checking log v0.4.33
Checking recursive v0.1.1
Checking brotli v8.0.4
Checking arrow-csv v59.0.0
Checking object_store v0.13.2
Checking arrow-string v59.0.0
Checking arrow-json v59.0.0
Checking uuid v1.23.3
Checking arrow-row v59.0.0
Checking arrow-arith v59.0.0
Compiling sqlparser_derive v0.5.0
Compiling seq-macro v0.3.6
Checking sqlparser v0.62.0
Checking arrow v59.0.0
Checking hex v0.4.3
Checking typenum v1.20.1
Compiling pin-project-internal v1.1.13
Compiling generic-array v0.14.7
Checking datafusion-doc v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
Checking hybrid-array v0.4.12
Checking pin-project v1.1.13
Checking ppv-lite86 v0.2.21
Checking rand_core v0.9.5
Checking foldhash v0.1.5
Compiling rustix v1.1.4
Compiling crossbeam-utils v0.8.21
Checking parquet v59.0.0
Checking hashbrown v0.15.5
Checking rand_chacha v0.9.0
Checking fixedbitset v0.5.7
Checking linux-raw-sys v0.12.1
Checking petgraph v0.8.3
Checking rand v0.9.4
Checking block-buffer v0.12.1
Checking crypto-common v0.2.2
Checking hashbrown v0.14.5
Checking const-oid v0.10.2
Checking fastrand v2.4.1
Checking tempfile v3.27.0
Compiling datafusion-macros v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
Checking digest v0.11.3
Checking dashmap v6.2.1
Checking block-buffer v0.10.4
Checking crypto-common v0.1.7
Compiling blake3 v1.8.5
Checking cpufeatures v0.3.0
Checking subtle v2.6.1
Checking digest v0.10.7
Checking arrayvec v0.7.7
Checking constant_time_eq v0.4.2
Checking arrayref v0.3.9
Checking blake2 v0.10.6
Checking sha2 v0.11.0
Checking md-5 v0.11.0
Compiling liblzma-sys v0.4.6
Checking libbz2-rs-sys v0.2.5
Checking datafusion-common-runtime v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
Checking compression-core v0.4.32
Checking tokio-util v0.7.18
Checking bzip2 v0.6.1
Checking glob v0.3.3
Compiling bigdecimal v0.4.10
Checking crc-catalog v2.5.0
Compiling heck v0.5.0
Checking crc v3.4.0
Compiling strum_macros v0.28.0
Checking tokio-stream v0.1.18
Checking liblzma v0.4.6
Checking compression-codecs v0.4.38
Checking arrow-avro v59.0.0
Checking datafusion-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
Checking async-compression v0.4.42
Checking datafusion-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr-common)
Checking datafusion-physical-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)
Checking datafusion-functions-aggregate-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common)
Checking datafusion-functions-window-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window-common)
Checking datafusion-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr)
Checking datafusion-physical-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr)
Checking datafusion-execution v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/execution)
Checking datafusion-functions v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)
Checking datafusion-functions-aggregate v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate)
Checking datafusion-functions-window v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window)
Checking datafusion-optimizer v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/optimizer)
Checking datafusion-physical-plan v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)
Checking datafusion-physical-expr-adapter v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-adapter)
error: unexpected closing delimiter: `}`
--> /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/mod.rs:5977:5
|
5952 | -> Result<()> {
| - this delimiter might not be properly closed...
...
5973 | }
| - ...as it matches this but it has different indentation
...
5977 | }
| ^ unexpected closing delimiter
error: could not compile `datafusion-physical-plan` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
-----
error: failed to build rustdoc for crate datafusion v54.0.0
note: this is usually due to a compilation error in the crate,
and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path /home/runner/work/datafusion/datafusion/datafusion/core --features array_expressions,avro,backtrace,bzip2,compression,crypto_expressions,datafusion-datasource-avro,datafusion-datasource-parquet,datafusion-functions-nested,datafusion-sql,datetime_expressions,default,encoding_expressions,extended_tests,flate2,force_hash_collisions,liblzma,math_expressions,nested_expressions,parquet,parquet_encryption,recursive_protection,regex_expressions,serde,sql,sqlparser,string_expressions,unicode_expressions,zstd &&
cargo check &&
cargo doc
Building datafusion-common v54.0.0 (current)
Built [ 31.898s] (current)
Parsing datafusion-common v54.0.0 (current)
Parsed [ 0.058s] (current)
Building datafusion-common v54.0.0 (baseline)
Built [ 31.902s] (baseline)
Parsing datafusion-common v54.0.0 (baseline)
Parsed [ 0.060s] (baseline)
Checking datafusion-common v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.638s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure constructible_struct_adds_field: externally-constructible struct adds field ---
Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field ExecutionOptions.enable_aggregation_blocked_groups in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:723
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 65.858s] datafusion-common
Building datafusion-expr-common v54.0.0 (current)
Built [ 18.344s] (current)
Parsing datafusion-expr-common v54.0.0 (current)
Parsed [ 0.018s] (current)
Building datafusion-expr-common v54.0.0 (baseline)
Built [ 18.238s] (baseline)
Parsing datafusion-expr-common v54.0.0 (baseline)
Parsed [ 0.019s] (baseline)
Checking datafusion-expr-common v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.209s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_variant_added.ron
Failed in:
variant EmitTo:NextBlock in /home/runner/work/datafusion/datafusion/datafusion/expr-common/src/groups_accumulator.rs:37
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 37.757s] datafusion-expr-common
Building datafusion-ffi v54.0.0 (current)
error: running cargo-doc on crate 'datafusion-ffi' failed with output:
-----
Compiling proc-macro2 v1.0.106
Compiling quote v1.0.45
Compiling unicode-ident v1.0.24
Compiling libc v0.2.186
Checking cfg-if v1.0.4
Compiling autocfg v1.5.1
Compiling libm v0.2.16
Checking memchr v2.8.2
Compiling num-traits v0.2.19
Compiling syn v2.0.118
Compiling zerocopy v0.8.52
Checking bytes v1.12.0
Compiling serde_core v1.0.228
Checking allocator-api2 v0.2.21
Checking itoa v1.0.18
Checking foldhash v0.2.0
Checking equivalent v1.0.2
Compiling getrandom v0.3.4
Checking once_cell v1.21.4
Checking hashbrown v0.17.1
Compiling zmij v1.0.21
Checking bitflags v2.13.0
Compiling serde_json v1.0.150
Checking indexmap v2.14.0
Checking num-integer v0.1.46
Checking iana-time-zone v0.1.65
Checking siphasher v1.0.3
Compiling version_check v0.9.5
Checking stable_deref_trait v1.2.1
Checking phf_shared v0.12.1
Checking chrono v0.4.45
Compiling ahash v0.8.12
Checking num-bigint v0.4.6
Compiling chrono-tz v0.10.4
Checking phf v0.12.1
Checking num-complex v0.4.6
Compiling synstructure v0.13.2
Checking arrow-schema v59.0.0
Compiling semver v1.0.28
Compiling rustc_version v0.4.1
Checking pin-project-lite v0.2.17
Compiling shlex v2.0.1
Compiling jobserver v0.1.34
Compiling find-msvc-tools v0.1.9
Compiling cc v1.2.65
Checking futures-sink v0.3.32
Compiling zerocopy-derive v0.8.52
Compiling zerofrom-derive v0.1.7
Checking zerofrom v0.1.8
Compiling yoke-derive v0.8.2
Compiling zerovec-derive v0.11.3
Checking yoke v0.8.3
Compiling displaydoc v0.2.6
Checking zerovec v0.11.6
Checking futures-core v0.3.32
Checking tinystr v0.8.3
Checking writeable v0.6.3
Checking litemap v0.8.2
Compiling pkg-config v0.3.33
Checking smallvec v1.15.2
Checking lexical-util v1.0.7
Checking icu_locale_core v2.2.0
Checking potential_utf v0.1.5
Compiling zstd-sys v2.0.16+zstd.1.5.7
Checking zerotrie v0.2.4
Compiling icu_normalizer_data v2.2.0
Compiling icu_properties_data v2.2.0
Checking utf8_iter v1.0.4
Checking icu_provider v2.2.0
Checking icu_collections v2.2.0
Checking lexical-write-integer v1.0.6
Checking lexical-parse-integer v1.0.6
Checking futures-channel v0.3.32
Compiling futures-macro v0.3.32
Compiling tokio-macros v2.7.0
Checking futures-task v0.3.32
Checking slab v0.4.12
Compiling zstd-safe v7.2.4
Checking futures-io v0.3.32
Compiling parking_lot_core v0.9.12
Checking futures-util v0.3.32
Checking tokio v1.52.3
Checking half v2.7.1
Checking arrow-buffer v59.0.0
Checking arrow-data v59.0.0
Checking arrow-array v59.0.0
Checking lexical-write-float v1.0.6
Checking lexical-parse-float v1.0.6
Checking icu_properties v2.2.0
Checking icu_normalizer v2.2.0
Compiling flatbuffers v25.12.19
Checking arrow-select v59.0.0
Checking aho-corasick v1.1.4
Checking unicode-width v0.2.2
Checking unicode-segmentation v1.13.3
Checking ryu v1.0.23
Checking scopeguard v1.2.0
Checking base64 v0.22.1
Checking regex-syntax v0.8.11
Checking lock_api v0.4.14
Checking comfy-table v7.2.2
Checking arrow-ord v59.0.0
Checking idna_adapter v1.2.2
Checking lexical-core v1.0.6
Checking atoi v2.0.0
Checking twox-hash v2.1.2
Compiling thiserror v2.0.18
Checking alloc-no-stdlib v2.0.4
Compiling getrandom v0.4.3
Checking percent-encoding v2.3.2
Checking form_urlencoded v1.2.2
Checking alloc-stdlib v0.2.4
Checking lz4_flex v0.13.1
Checking arrow-cast v59.0.0
Checking regex-automata v0.4.14
Checking idna v1.1.0
Checking futures-executor v0.3.32
Compiling thiserror-impl v2.0.18
Compiling tracing-attributes v0.1.31
Checking tracing-core v0.1.36
Checking regex v1.12.4
Checking csv-core v0.1.13
Compiling paste v1.0.15
Checking same-file v1.0.6
Checking simdutf8 v0.1.5
Checking either v1.16.0
Compiling snap v1.1.1
Checking tracing v0.1.44
Checking itertools v0.14.0
Checking walkdir v2.5.0
Checking csv v1.4.0
Checking futures v0.3.32
Checking url v2.5.8
Checking brotli-decompressor v5.0.3
Checking parking_lot v0.12.5
Compiling async-trait v0.1.89
Checking http v1.4.2
Compiling anyhow v1.0.102
Checking humantime v2.3.0
Checking zlib-rs v0.6.4
Checking brotli v8.0.4
Checking object_store v0.13.2
Checking flate2 v1.1.9
Checking arrow-csv v59.0.0
Checking arrow-json v59.0.0
Checking arrow-string v59.0.0
Checking arrow-row v59.0.0
Checking arrow-arith v59.0.0
Checking log v0.4.33
Compiling seq-macro v0.3.6
Checking zstd v0.13.3
Checking arrow-ipc v59.0.0
Checking uuid v1.23.3
Compiling pin-project-internal v1.1.13
Compiling crossbeam-utils v0.8.21
Checking datafusion-doc v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
Compiling rustix v1.1.4
Checking ppv-lite86 v0.2.21
Checking parquet v59.0.0
Checking arrow v59.0.0
Checking pin-project v1.1.13
Checking rand_core v0.9.5
Checking linux-raw-sys v0.12.1
Checking foldhash v0.1.5
Compiling prost-derive v0.14.4
Checking hashbrown v0.15.5
Checking rand_chacha v0.9.0
Checking fastrand v2.4.1
Checking fixedbitset v0.5.7
Checking prost v0.14.4
Checking hashbrown v0.14.5
Checking tempfile v3.27.0
Compiling datafusion-macros v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
Checking petgraph v0.8.3
Checking dashmap v6.2.1
Checking rand v0.9.4
Checking hex v0.4.3
Checking datafusion-common-runtime v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
Checking glob v0.3.3
Compiling winnow v1.0.3
Compiling toml_datetime v1.1.1+spec-1.1.0
Compiling stabby-macros v72.1.8
Compiling toml_parser v1.1.2+spec-1.1.0
Compiling rustversion v1.0.22
Checking tokio-util v0.7.18
Compiling stabby-abi v72.1.8
Compiling toml_edit v0.25.12+spec-1.1.0
Checking tokio-stream v0.1.18
Checking sha2-const-stable v0.1.0
Checking datafusion-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
Compiling stabby v72.1.8
Checking libloading v0.9.0
Checking async-ffi v0.5.0
Compiling proc-macro-crate v3.5.0
Checking datafusion-proto-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/proto-common)
Checking datafusion-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr-common)
Checking datafusion-proto-models v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/proto-models)
Checking datafusion-physical-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)
Checking datafusion-functions-window-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window-common)
Checking datafusion-functions-aggregate-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common)
Checking datafusion-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr)
Checking datafusion-physical-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr)
Checking datafusion-execution v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/execution)
Checking datafusion-functions v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)
Checking datafusion-functions-window v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window)
Checking datafusion-functions-aggregate v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate)
Checking datafusion-physical-plan v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)
Checking datafusion-physical-expr-adapter v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-adapter)
error: unexpected closing delimiter: `}`
--> /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/mod.rs:5977:5
|
5952 | -> Result<()> {
| - this delimiter might not be properly closed...
...
5973 | }
| - ...as it matches this but it has different indentation
...
5977 | }
| ^ unexpected closing delimiter
error: could not compile `datafusion-physical-plan` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
-----
error: failed to build rustdoc for crate datafusion-ffi v54.0.0
note: this is usually due to a compilation error in the crate,
and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path /home/runner/work/datafusion/datafusion/datafusion/ffi --features datafusion-functions,datafusion-functions-aggregate,datafusion-functions-table,datafusion-functions-window,integration-tests,tarpaulin_include &&
cargo check &&
cargo doc
Building datafusion-functions-aggregate v54.0.0 (current)
Built [ 29.521s] (current)
Parsing datafusion-functions-aggregate v54.0.0 (current)
Parsed [ 0.044s] (current)
Building datafusion-functions-aggregate v54.0.0 (baseline)
Built [ 29.473s] (baseline)
Parsing datafusion-functions-aggregate v54.0.0 (baseline)
Parsed [ 0.046s] (baseline)
Checking datafusion-functions-aggregate v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.185s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 60.442s] datafusion-functions-aggregate
Building datafusion-functions-aggregate-common v54.0.0 (current)
Built [ 19.580s] (current)
Parsing datafusion-functions-aggregate-common v54.0.0 (current)
Parsed [ 0.022s] (current)
Building datafusion-functions-aggregate-common v54.0.0 (baseline)
Built [ 19.587s] (baseline)
Parsing datafusion-functions-aggregate-common v54.0.0 (baseline)
Parsed [ 0.019s] (baseline)
Checking datafusion-functions-aggregate-common v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.140s] 223 checks: 219 pass, 4 fail, 0 warn, 30 skip
--- failure enum_struct_variant_field_added: pub enum struct variant field added ---
Description:
An enum's exhaustive struct variant has a new field, which has to be included when constructing or matching on this variant.
ref: https://doc.rust-lang.org/reference/attributes/type_system.html#the-non_exhaustive-attribute
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_struct_variant_field_added.ron
Failed in:
field pending_builder of variant SeenValues::All in /home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:113
field builder of variant SeenValues::Some in /home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:116
--- failure enum_struct_variant_field_missing: pub enum struct variant's field removed or renamed ---
Description:
A publicly-visible enum has a struct variant whose field is no longer available under its prior name. It may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_struct_variant_field_missing.ron
Failed in:
field values of variant SeenValues::Some, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/c7e11494a6baec33443333870bb0106fd7408910/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:46
--- failure trait_requires_more_generic_type_params: trait now requires more generic type parameters ---
Description:
A trait now requires more generic type parameters than it used to. Uses of this trait that supplied the previously-required number of generic types will be broken. To fix this, consider supplying default values for newly-added generic types.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-new-parameter-no-default
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/trait_requires_more_generic_type_params.ron
Failed in:
trait NullState (0 -> 2 required generic types) in /home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:305
trait SeenValues (0 -> 1 required generic types) in /home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:99
--- failure type_requires_more_generic_type_params: type now requires more generic type parameters ---
Description:
A type now requires more generic type parameters than it used to. Uses of this type that supplied the previously-required number of generic types will be broken. To fix this, consider supplying default values for newly-added generic types.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-new-parameter-no-default
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/type_requires_more_generic_type_params.ron
Failed in:
Struct NullState (0 -> 2 required generic types) in /home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:305
Enum SeenValues (0 -> 1 required generic types) in /home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/accumulate.rs:99
Summary semver requires new major version: 4 major and 0 minor checks failed
Finished [ 40.145s] datafusion-functions-aggregate-common
Building datafusion-physical-expr v54.0.0 (current)
Built [ 27.855s] (current)
Parsing datafusion-physical-expr v54.0.0 (current)
Parsed [ 0.047s] (current)
Building datafusion-physical-expr v54.0.0 (baseline)
Built [ 28.218s] (baseline)
Parsing datafusion-physical-expr v54.0.0 (baseline)
Parsed [ 0.051s] (baseline)
Checking datafusion-physical-expr v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.334s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 57.454s] datafusion-physical-expr
Building datafusion-physical-plan v54.0.0 (current)
error: running cargo-doc on crate 'datafusion-physical-plan' failed with output:
-----
Compiling proc-macro2 v1.0.106
Compiling quote v1.0.45
Compiling unicode-ident v1.0.24
Compiling libc v0.2.186
Checking cfg-if v1.0.4
Compiling libm v0.2.16
Compiling autocfg v1.5.1
Checking memchr v2.8.2
Compiling num-traits v0.2.19
Compiling syn v2.0.118
Compiling zerocopy v0.8.52
Compiling getrandom v0.3.4
Checking bytes v1.12.0
Checking rand_core v0.9.5
Checking foldhash v0.2.0
Compiling serde_core v1.0.228
Checking once_cell v1.21.4
Checking itoa v1.0.18
Checking allocator-api2 v0.2.21
Checking equivalent v1.0.2
Compiling zmij v1.0.21
Compiling serde_json v1.0.150
Checking hashbrown v0.17.1
Checking num-integer v0.1.46
Checking siphasher v1.0.3
Checking indexmap v2.14.0
Checking iana-time-zone v0.1.65
Compiling version_check v0.9.5
Compiling ahash v0.8.12
Checking chrono v0.4.45
Checking phf_shared v0.12.1
Checking num-bigint v0.4.6
Compiling chrono-tz v0.10.4
Checking phf v0.12.1
Checking num-complex v0.4.6
Checking arrow-schema v59.0.0
Checking lexical-util v1.0.7
Compiling jobserver v0.1.34
Compiling find-msvc-tools v0.1.9
Compiling shlex v2.0.1
Compiling synstructure v0.13.2
Compiling cc v1.2.65
Compiling zerocopy-derive v0.8.52
Compiling pkg-config v0.3.33
Checking pin-project-lite v0.2.17
Compiling zerofrom-derive v0.1.7
Compiling zstd-sys v2.0.16+zstd.1.5.7
Checking lexical-parse-integer v1.0.6
Checking lexical-write-integer v1.0.6
Compiling semver v1.0.28
Compiling rustc_version v0.4.1
Checking lexical-write-float v1.0.6
Checking lexical-parse-float v1.0.6
Checking zerofrom v0.1.8
Compiling yoke-derive v0.8.2
Checking aho-corasick v1.1.4
Checking unicode-width v0.2.2
Compiling zstd-safe v7.2.4
Checking stable_deref_trait v1.2.1
Checking unicode-segmentation v1.13.3
Checking bitflags v2.13.0
Checking ryu v1.0.23
Checking regex-syntax v0.8.11
Checking comfy-table v7.2.2
Checking yoke v0.8.3
Checking lexical-core v1.0.6
Compiling flatbuffers v25.12.19
Checking atoi v2.0.0
Checking base64 v0.22.1
Checking smallvec v1.15.2
Compiling getrandom v0.4.3
Checking ppv-lite86 v0.2.21
Compiling tokio-macros v2.7.0
Checking rand_chacha v0.9.0
Checking regex-automata v0.4.14
Checking rand v0.9.4
Compiling zerovec-derive v0.11.3
Checking rand_distr v0.5.1
Checking half v2.7.1
Checking arrow-buffer v59.0.0
Checking csv-core v0.1.13
Checking twox-hash v2.1.2
Checking lz4_flex v0.13.1
Checking regex v1.12.4
Checking csv v1.4.0
Checking arrow-data v59.0.0
Checking zerovec v0.11.6
Checking tokio v1.52.3
Checking arrow-array v59.0.0
Compiling displaydoc v0.2.6
Checking either v1.16.0
Compiling anyhow v1.0.102
Checking simdutf8 v0.1.5
Checking itertools v0.14.0
Checking arrow-select v59.0.0
Checking arrow-arith v59.0.0
Checking arrow-row v59.0.0
Checking arrow-ord v59.0.0
Checking arrow-string v59.0.0
Checking log v0.4.33
Compiling parking_lot_core v0.9.12
Checking arrow-cast v59.0.0
Checking uuid v1.23.3
Checking scopeguard v1.2.0
Checking lock_api v0.4.14
Checking tinystr v0.8.3
Checking litemap v0.8.2
Checking arrow-csv v59.0.0
Checking arrow-json v59.0.0
Checking writeable v0.6.3
Checking icu_locale_core v2.2.0
Compiling prost-derive v0.14.4
Checking zerotrie v0.2.4
Checking potential_utf v0.1.5
Checking utf8_iter v1.0.4
Compiling icu_properties_data v2.2.0
Compiling icu_normalizer_data v2.2.0
Checking icu_provider v2.2.0
Checking icu_collections v2.2.0
Checking parking_lot v0.12.5
Compiling pin-project-internal v1.1.13
Checking futures-sink v0.3.32
Checking futures-core v0.3.32
Checking futures-channel v0.3.32
Compiling futures-macro v0.3.32
Checking prost v0.14.4
Checking futures-task v0.3.32
Checking pin-project v1.1.13
Checking slab v0.4.12
Checking futures-io v0.3.32
Checking icu_normalizer v2.2.0
Checking icu_properties v2.2.0
Checking futures-util v0.3.32
Compiling async-trait v0.1.89
Checking zstd v0.13.3
Checking arrow-ipc v59.0.0
Checking idna_adapter v1.2.2
Compiling crossbeam-utils v0.8.21
Compiling rustix v1.1.4
Checking percent-encoding v2.3.2
Compiling thiserror v2.0.18
Checking form_urlencoded v1.2.2
Checking idna v1.1.0
Compiling thiserror-impl v2.0.18
Compiling tracing-attributes v0.1.31
Checking arrow v59.0.0
Checking datafusion-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
Checking futures-executor v0.3.32
Checking tracing-core v0.1.36
Checking linux-raw-sys v0.12.1
Checking datafusion-doc v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
Checking same-file v1.0.6
Checking tracing v0.1.44
Checking walkdir v2.5.0
Checking futures v0.3.32
Checking url v2.5.8
Checking http v1.4.2
Checking fastrand v2.4.1
Checking foldhash v0.1.5
Checking humantime v2.3.0
Checking hashbrown v0.14.5
Checking object_store v0.13.2
Checking dashmap v6.2.1
Checking tempfile v3.27.0
Checking hashbrown v0.15.5
Checking fixedbitset v0.5.7
Checking petgraph v0.8.3
Compiling datafusion-macros v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
Checking hex v0.4.3
Checking datafusion-common-runtime v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
Checking datafusion-proto-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/proto-common)
Checking datafusion-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr-common)
Checking datafusion-proto-models v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/proto-models)
Checking datafusion-physical-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)
Checking datafusion-functions-window-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window-common)
Checking datafusion-functions-aggregate-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common)
Checking datafusion-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr)
Checking datafusion-execution v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/execution)
Checking datafusion-physical-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr)
Checking datafusion-functions v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)
Documenting datafusion-physical-plan v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)
error: unexpected closing delimiter: `}`
--> /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/mod.rs:5977:5
|
5952 | -> Result<()> {
| - this delimiter might not be properly closed...
...
5973 | }
| - ...as it matches this but it has different indentation
...
5977 | }
| ^ unexpected closing delimiter
error: could not document `datafusion-physical-plan`
-----
error: failed to build rustdoc for crate datafusion-physical-plan v54.0.0
note: this is usually due to a compilation error in the crate,
and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path /home/runner/work/datafusion/datafusion/datafusion/physical-plan --features force_hash_collisions,proto,test_utils,tokio_coop,tokio_coop_fallback &&
cargo check &&
cargo doc
Building datafusion-sqllogictest v54.0.0 (current)
error: running cargo-doc on crate 'datafusion-sqllogictest' failed with output:
-----
Compiling proc-macro2 v1.0.106
Compiling unicode-ident v1.0.24
Compiling quote v1.0.45
Compiling libc v0.2.186
Checking cfg-if v1.0.4
Checking bytes v1.12.0
Checking memchr v2.8.2
Compiling serde_core v1.0.228
Compiling syn v2.0.118
Compiling autocfg v1.5.1
Compiling jobserver v0.1.34
Compiling shlex v2.0.1
Compiling find-msvc-tools v0.1.9
Compiling cc v1.2.65
Checking itoa v1.0.18
Checking allocator-api2 v0.2.21
Checking equivalent v1.0.2
Checking once_cell v1.21.4
Checking foldhash v0.2.0
Compiling libm v0.2.16
Compiling num-traits v0.2.19
Checking hashbrown v0.17.1
Checking pin-project-lite v0.2.17
Compiling zmij v1.0.21
Checking indexmap v2.14.0
Compiling zerocopy v0.8.52
Checking futures-core v0.3.32
Checking futures-sink v0.3.32
Checking errno v0.3.14
Checking signal-hook-registry v1.4.8
Checking socket2 v0.6.4
Checking mio v1.2.1
Compiling version_check v0.9.5
Compiling serde v1.0.228
Compiling getrandom v0.3.4
Checking num-integer v0.1.46
Checking slab v0.4.12
Compiling serde_json v1.0.150
Checking futures-channel v0.3.32
Checking smallvec v1.15.2
Checking num-bigint v0.4.6
Compiling synstructure v0.13.2
Checking http v1.4.2
Checking futures-io v0.3.32
Checking futures-task v0.3.32
Checking base64 v0.22.1
Checking iana-time-zone v0.1.65
Checking chrono v0.4.45
Checking tracing-core v0.1.36
Compiling zerocopy-derive v0.8.52
Compiling tokio-macros v2.7.0
Checking tokio v1.52.3
Compiling serde_derive v1.0.228
Compiling futures-macro v0.3.32
Checking futures-util v0.3.32
Compiling tracing-attributes v0.1.31
Compiling zerofrom-derive v0.1.7
Checking siphasher v1.0.3
Checking tracing v0.1.44
Compiling yoke-derive v0.8.2
Checking stable_deref_trait v1.2.1
Checking num-complex v0.4.6
Compiling zerovec-derive v0.11.3
Checking zerofrom v0.1.8
Checking getrandom v0.2.17
Compiling pkg-config v0.3.33
Checking half v2.7.1
Checking phf_shared v0.12.1
Compiling displaydoc v0.2.6
Compiling ahash v0.8.12
Compiling thiserror v2.0.18
Checking yoke v0.8.3
Compiling chrono-tz v0.10.4
Checking percent-encoding v2.3.2
Checking zerovec v0.11.6
Checking phf v0.12.1
Checking arrow-buffer v59.0.0
Checking ppv-lite86 v0.2.21
Compiling thiserror-impl v2.0.18
Checking arrow-schema v59.0.0
Compiling ring v0.17.14
Checking arrow-data v59.0.0
Compiling semver v1.0.28
Checking log v0.4.33
Checking rand_core v0.10.1
Compiling getrandom v0.4.3
Checking tinystr v0.8.3
Checking untrusted v0.9.0
Checking litemap v0.8.2
Checking bitflags v2.13.0
Checking writeable v0.6.3
Checking potential_utf v0.1.5
Checking icu_locale_core v2.2.0
Checking zerotrie v0.2.4
Compiling zstd-sys v2.0.16+zstd.1.5.7
Compiling async-trait v0.1.89
Checking rand_core v0.9.5
Compiling icu_normalizer_data v2.2.0
Checking utf8_iter v1.0.4
Compiling icu_properties_data v2.2.0
Checking icu_collections v2.2.0
Checking rand_chacha v0.9.0
Checking icu_provider v2.2.0
Checking rand v0.9.4
Checking aho-corasick v1.1.4
Checking regex-syntax v0.8.11
Checking arrow-array v59.0.0
Compiling zstd-safe v7.2.4
Checking lexical-util v1.0.7
Compiling object v0.37.3
Checking ryu v1.0.23
Checking regex-automata v0.4.14
Checking icu_properties v2.2.0
Checking arrow-select v59.0.0
Checking icu_normalizer v2.2.0
Compiling rustix v1.1.4
Checking idna_adapter v1.2.2
Checking form_urlencoded v1.2.2
Checking either v1.16.0
Checking regex v1.12.4
Compiling crc32fast v1.5.0
Compiling parking_lot_core v0.9.12
Checking typenum v1.20.1
Checking unicode-width v0.2.2
Checking idna v1.1.0
Checking lexical-parse-integer v1.0.6
Checking lexical-write-integer v1.0.6
Compiling rustc_version v0.4.1
Checking futures-executor v0.3.32
Compiling pin-project-internal v1.1.13
Checking simd-adler32 v0.3.9
Checking scopeguard v1.2.0
Checking adler2 v2.0.1
Checking miniz_oxide v0.8.9
Checking lock_api v0.4.14
Checking futures v0.3.32
Compiling flatbuffers v25.12.19
Checking lexical-write-float v1.0.6
Checking pin-project v1.1.13
Checking lexical-parse-float v1.0.6
Checking url v2.5.8
Checking zlib-rs v0.6.4
Checking unicode-segmentation v1.13.3
Compiling ar_archive_writer v0.5.2
Checking comfy-table v7.2.2
Checking lexical-core v1.0.6
Checking itertools v0.14.0
Checking flate2 v1.1.9
Compiling psm v0.1.31
Checking arrow-ord v59.0.0
Checking twox-hash v2.1.2
Checking atoi v2.0.0
Compiling stacker v0.1.24
Checking alloc-no-stdlib v2.0.4
Compiling snap v1.1.1
Checking hex v0.4.3
Checking alloc-stdlib v0.2.4
Checking arrow-cast v59.0.0
Checking lz4_flex v0.13.1
Checking parking_lot v0.12.5
Checking csv-core v0.1.13
Checking humantime v2.3.0
Compiling paste v1.0.15
Checking simdutf8 v0.1.5
Checking same-file v1.0.6
Checking walkdir v2.5.0
Checking csv v1.4.0
Checking brotli-decompressor v5.0.3
Compiling recursive-proc-macro-impl v0.1.1
Checking subtle v2.6.1
Checking arrow-json v59.0.0
Checking recursive v0.1.1
Checking arrow-csv v59.0.0
Checking object_store v0.13.2
Checking brotli v8.0.4
Checking arrow-string v59.0.0
Checking arrow-row v59.0.0
Checking arrow-arith v59.0.0
Checking uuid v1.23.3
Checking tokio-util v0.7.18
Compiling sqlparser_derive v0.5.0
Compiling seq-macro v0.3.6
Checking zstd v0.13.3
Checking arrow-ipc v59.0.0
Checking sqlparser v0.62.0
Checking cpufeatures v0.3.0
Checking hybrid-array v0.4.12
Checking linux-raw-sys v0.12.1
Checking cmov v0.5.4
Checking ctutils v0.4.2
Checking crypto-common v0.2.2
Checking arrow v59.0.0
Checking parquet v59.0.0
Checking block-buffer v0.12.1
Compiling generic-array v0.14.7
Checking const-oid v0.10.2
Checking digest v0.11.3
Checking datafusion-doc v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
Compiling crossbeam-utils v0.8.21
Checking foldhash v0.1.5
Checking crypto-common v0.1.7
Checking hashbrown v0.15.5
Checking block-buffer v0.10.4
Compiling heck v0.5.0
Checking fastrand v2.4.1
Checking fixedbitset v0.5.7
Checking tempfile v3.27.0
Checking petgraph v0.8.3
Checking digest v0.10.7
Checking sha2 v0.11.0
Checking md-5 v0.11.0
Checking hashbrown v0.14.5
Checking dashmap v6.2.1
Compiling blake3 v1.8.5
Compiling datafusion-macros v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
Compiling anyhow v1.0.102
Checking arrayvec v0.7.7
Checking constant_time_eq v0.4.2
Checking arrayref v0.3.9
Checking blake2 v0.10.6
Compiling liblzma-sys v0.4.6
Checking libbz2-rs-sys v0.2.5
Checking bzip2 v0.6.1
Checking datafusion-common-runtime v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
Checking compression-core v0.4.32
Compiling prost-derive v0.14.4
Compiling httparse v1.10.1
Checking http-body v1.0.1
Checking glob v0.3.3
Checking tower-service v0.3.3
Checking fnv v1.0.7
Checking try-lock v0.2.5
Checking atomic-waker v1.1.2
Checking h2 v0.4.15
Checking want v0.3.1
Checking tokio-stream v0.1.18
Checking zeroize v1.9.0
Checking httpdate v1.0.3
Checking hyper v1.10.1
Checking rustls-pki-types v1.14.1
Checking datafusion-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
Compiling prost v0.14.4
Checking hyper-util v0.1.20
Checking http-body-util v0.1.3
Checking tower-layer v0.3.3
Compiling rustls v0.23.40
Compiling prettyplease v0.2.37
Checking sync_wrapper v1.0.2
Compiling prost-types v0.14.4
Checking liblzma v0.4.6
Checking compression-codecs v0.4.38
Checking datafusion-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr-common)
Checking async-compression v0.4.42
Checking rustls-webpki v0.103.13
Compiling serde_derive_internals v0.29.1
Checking datafusion-physical-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)
Checking mime v0.3.17
Compiling schemars v0.8.22
Compiling schemars_derive v0.8.22
Compiling hashbrown v0.16.1
Checking datafusion-functions-window-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window-common)
Checking datafusion-functions-aggregate-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common)
Checking axum-core v0.5.6
Checking datafusion-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr)
Checking tower v0.5.3
Compiling dyn-clone v1.0.20
Checking matchit v0.8.4
Compiling multimap v0.10.1
Checking axum v0.8.9
Compiling prost-build v0.14.4
Compiling regress v0.10.5
Checking datafusion-physical-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr)
Checking datafusion-execution v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/execution)
Compiling pbjson-build v0.8.0
Checking datafusion-functions v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)
Checking hyper-timeout v0.5.2
Checking chacha20 v0.10.0
Compiling strsim v0.11.1
Compiling portable-atomic v1.13.1
Compiling ident_case v1.0.1
Checking rand v0.10.1
Compiling darling_core v0.23.0
Compiling typify-impl v0.5.0
Checking tonic v0.14.6
Compiling serde_tokenstream v0.2.3
Checking ureq-proto v0.6.0
Compiling bigdecimal v0.4.10
Checking crc-catalog v2.5.0
Checking utf8parse v0.2.2
Checking utf8-zero v0.8.1
Compiling bollard-buildkit-proto v0.7.0
Checking tinyvec_macros v0.1.1
Checking tinyvec v1.11.0
Checking datafusion-physical-plan v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)
error: unexpected closing delimiter: `}`
--> /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/mod.rs:5977:5
|
5952 | -> Result<()> {
| - this delimiter might not be properly closed...
...
5973 | }
| - ...as it matches this but it has different indentation
...
5977 | }
| ^ unexpected closing delimiter
error: could not compile `datafusion-physical-plan` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
-----
error: failed to build rustdoc for crate datafusion-sqllogictest v54.0.0
note: this is usually due to a compilation error in the crate,
and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
cargo new --lib example &&
cd example &&
echo '[workspace]' >> Cargo.toml &&
cargo add --path /home/runner/work/datafusion/datafusion/datafusion/sqllogictest --features avro,backtrace,bytes,chrono,datafusion-substrait,parquet_encryption,postgres,postgres-types,substrait,testcontainers-modules,tokio-postgres &&
cargo check &&
cargo doc
error: aborting due to failure to build rustdoc for crate datafusion v54.0.0
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3347807763
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
Review Comment:
Sorry, I'm not sure what you mean by `dump` in this context, could you please explain?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3748931416
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/block_store/vec_block_store.rs:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`BlockStore`] wrapper specialized for stores whose blocks are `Vec<T>`.
+
+use std::fmt::Debug;
+use std::iter;
+use std::marker::PhantomData;
+use std::ops::{Index, IndexMut};
+
+use datafusion_common::utils::split_vec_min_alloc;
+use datafusion_common::{Result, internal_datafusion_err, internal_err};
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+use crate::aggregate::groups_accumulator::block_store::{Block, BlockStore};
+
+/// Thin wrapper around a [`BlockStore<Vec<T>>`] that adds vector-specific
+/// emit semantics on top of the generic block store API.
+///
+/// Emitting `EmitTo::First` requires vector-specific split semantics, so it
+/// lives on this wrapper rather than on [`BlockStore`] itself. The wrapper
+/// re-exposes every [`BlockStore`] method by delegation, plus an [`emit`]
+/// method implemented purely via [`BlockStore::push_block`] /
+/// [`BlockStore::pop_block`] so it works uniformly over flat and blocked
+/// storage.
+///
+/// [`emit`]: VecBlockStore::emit
+#[derive(Debug)]
+pub struct VecBlockStore<T, S>
+where
+ T: Clone + Debug,
+ S: BlockStore<Vec<T>>,
+{
+ inner: S,
+ _phantom: PhantomData<T>,
+}
+
+impl<T, S> VecBlockStore<T, S>
+where
+ T: Clone + Debug,
+ S: BlockStore<Vec<T>>,
+{
+ /// Wrap an existing block store.
+ pub fn new(inner: S) -> Self {
+ Self {
+ inner,
+ _phantom: PhantomData,
+ }
+ }
+
+ // ---- BlockStore method delegation ----------------------------------
+ pub fn reserve_blocks(&mut self) {
+ self.inner.reserve_blocks();
+ }
+
+ pub fn resize(&mut self, total_num_groups: usize, default_value: T) {
+ self.inner.resize(total_num_groups, default_value);
+ }
+
+ pub fn num_blocks(&self) -> usize {
+ self.inner.num_blocks()
+ }
+
+ pub fn block_size(&self) -> Option<usize> {
+ self.inner.block_size()
+ }
+
+ pub fn clear(&mut self) {
+ self.inner.clear();
+ }
+
+ // ---- Emit ----------------------------------------------------------
+
+ /// Emit values according to `emit_to`, expressed only in terms of
+ /// [`BlockStore::push_block`] and [`BlockStore::pop_block`].
+ ///
+ /// - [`EmitTo::All`]: drains every block via repeated `pop_block` and
+ /// concatenates the results into a single `Vec<T>`.
+ /// - [`EmitTo::First`]`(n)`: pops the first block, splits off the first
+ /// `n` values, and pushes the remainder back. Only meaningful when the
+ /// first block holds at least `n` values (true for flat storage); the
+ /// call returns an internal error otherwise.
+ /// - [`EmitTo::NextBlock`]: pops a single block.
+ pub fn emit(&mut self, emit_to: EmitTo) -> Result<Vec<T>> {
+ match emit_to {
+ EmitTo::All => self.inner.pop_block().ok_or_else(|| {
Review Comment:
this doesn't seem to match the above description:
> [`EmitTo::All`]: drains every block via repeated `pop_block` and
concatenates the results into a single `Vec<T>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5384052816
> So I took this pr and added support locally for binary and it fixed the issue we had with huge batches coming out of aggregate, however I have concerns because ideally It should be enabled in all cases in order for it to be benefitial - for us it's partial/final with both spill/emit early support but there.
>
> 1. Grouping impl will be complicated: Adding support for more columns types and multi group by is complicated
> 2. In the very hot loop of doing aggregation and jumping in memory you are adding another memory jump
> 3. Complicated aggregate expression code - even though the idea is simple, the actual implementation that needed to be done to support that is really complicated
To quantify the implementation effort for options 1 and 3: we would need to migrate all implementations of `GroupValues` and `GroupsAccumulator` to the new blocked physical layout. There are 20+ implementations in total, it's definitely significant but seems doable.
> Another complexity is supporting `ListGroupValueBuilder`, try to add support for blocking with it(ignore non blocking support and just see how complex even blocked support is)
Maybe it's a good idea to experimenting the hardest one first π€
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5600883299
> The diagram for EmitTo::Last(n) is accurate only if there's a shrink_to_fit() call after every emit, which means reallocating and copying all the elements from the old allocation to the new allocation.
Partly, yes β truncate alone keeps the capacity, so the interim Vec phase relies on shrink_to_fit, and how much RSS that returns is allocator-dependent (Rachelint raised this on Aug 29; it's why the plan doesn't stop at Vec).
But shrink_to_fit is a realloc to a smaller size, not a copy of everything: glibc shrinks mmap'd chunks in place with mremap, jemalloc trims large allocations in place and purges the tail pages, and mimalloc only reallocates when the new size drops below half the old one β so in the worst case it copies the remainder once per halving, O(N) total, the same order as copying each output batch out. The curve in the right column is exact for blocks (a block is dropped whole, no realloc at all) and "as good as the allocator lets it be" for the interim Vec phase. Worth measuring RSS explicitly in #24795 rather than assuming either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3796045297
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/block_store/blocked.rs:
##########
@@ -0,0 +1,436 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Blocked [`BlockStore`] implementation.
+
+use std::{
+ fmt::Debug,
+ mem,
+ ops::{Index, IndexMut},
+};
+
+use crate::aggregate::groups_accumulator::block_store::{Block, BlockStore};
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// Blocks are popped in FIFO order by keeping a cursor into `inner`. Popping a
+/// block swaps it out in O(1) using `mem::take` and advances the cursor,
+/// avoiding the O(n) shift cost of `Vec::remove(0)`.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct BlockedBlockStore<B: Block> {
+ inner: Vec<B>,
+ /// Index of the next active block.
+ cursor: usize,
+ block_size: usize,
+}
+
+impl<B: Block> BlockedBlockStore<B> {
+ /// Create a new blocked store with the given fixed block size.
+ pub fn new(block_size: usize) -> Self {
+ Self {
+ inner: Vec::new(),
+ cursor: 0,
+ block_size,
+ }
+ }
+}
+
+impl<B: Block> BlockStore<B> for BlockedBlockStore<B> {
+ fn push_block(&mut self, block: B) {
+ self.inner.push(block);
+ }
+
+ fn pop_block(&mut self) -> Option<B> {
+ if self.cursor >= self.inner.len() {
+ return None;
+ }
+
+ let block = mem::take(&mut self.inner[self.cursor]);
Review Comment:
You are initializing block even when not needed, which might not reduce memory usage.
why not using `VecDeque` and using `pop_front` instead of `pop` block and having all of this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2888248441
OK... I guess it is due to the computation of `block_id` and `block_offset`.
I found a machine with similar cores and profiling.
Then I found in the old and so good intel core, the `/` and `%` are really expansive.
I am trying to optimize it.
```rust
impl GroupIndexOperations for BlockedGroupIndexOperations {
fn get_block_id(group_index: usize, block_size: usize) -> usize {
group_index / block_size
}
fn get_block_offset(group_index: usize, block_size: usize) -> usize {
group_index % block_size
}
}
```
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890735910
>I wonder what happens if we make it more like at least 1 million or 1MiB so the effect on cache-friendliness is smaller?
We could optimize a growing strategy for the first allocated Vec if memory usage / overhead of first block is a concern.
> I think we should try to minimize the impact of this on low-cardinality cases (e.g. make sure they fit in one array, minimize the overhead of blocks)...
If I don't misunderstand, does it mean strategy like that:
- We make the block size large enough
- For the first block, we still perform `resizing` at firstly
- But after it grow large enough, we switch to `blocked approach`?
> Yeah it is quite efficient, although problematic for large inputs
Agree. It also leads to large memory usage, because we only release memory after all the batches are returned(we hold the really single batch, and only return slice of it now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890524255
> Will do
Thanks!
> No worries -- I am sorry I have not given it as much attention as I would like. Is there anything else I can do to help (like implement one of the grouped memory examples)? I am about to file a ticket to start organizing the follow on work
I think still need some time to evaluate what benefits can this brings
After experiment, I found:
- It can really help to reduce memory usage as @joroKr21 mentioned in #9562
- But it maybe not really help much for performance (performance improve mainly due to removal of usage of expansive `slice`) currently.
But inspired by the `batch_size` based memory allocation, I am thinking can we have some ways to reuse memory? And I am trying it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848548029
> π€: Benchmark completed
(I am surprised this PR didn't yield better results, I am reruning now to see)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848062289
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_1.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 0.57ms β 0.55ms β no change β
β QQuery 1 β 73.94ms β 75.00ms β no change β
β QQuery 2 β 113.27ms β 118.51ms β no change β
β QQuery 3 β 126.34ms β 126.51ms β no change β
β QQuery 4 β 754.71ms β 789.90ms β no change β
β QQuery 5 β 834.40ms β 836.35ms β no change β
β QQuery 6 β 0.70ms β 0.62ms β +1.13x faster β
β QQuery 7 β 87.54ms β 89.84ms β no change β
β QQuery 8 β 921.68ms β 928.83ms β no change β
β QQuery 9 β 1217.57ms β 1197.21ms β no change β
β QQuery 10 β 291.89ms β 308.86ms β 1.06x slower β
β QQuery 11 β 334.85ms β 344.94ms β no change β
β QQuery 12 β 892.72ms β 900.44ms β no change β
β QQuery 13 β 1317.81ms β 1308.76ms β no change β
β QQuery 14 β 840.55ms β 857.04ms β no change β
β QQuery 15 β 1039.30ms β 1101.46ms β 1.06x slower β
β QQuery 16 β 1749.21ms β 1742.09ms β no change β
β QQuery 17 β 1583.83ms β 1614.13ms β no change β
β QQuery 18 β 3067.32ms β 3093.76ms β no change β
β QQuery 19 β 123.92ms β 115.09ms β +1.08x faster β
β QQuery 20 β 1153.71ms β 1169.49ms β no change β
β QQuery 21 β 1396.36ms β 1397.95ms β no change β
β QQuery 22 β 2417.82ms β 2395.93ms β no change β
β QQuery 23 β 8439.01ms β 8627.38ms β no change β
β QQuery 24 β 502.46ms β 491.22ms β no change β
β QQuery 25 β 427.03ms β 433.83ms β no change β
β QQuery 26 β 558.27ms β 575.80ms β no change β
β QQuery 27 β 1822.79ms β 1834.22ms β no change β
β QQuery 28 β 12992.87ms β 12962.28ms β no change β
β QQuery 29 β 567.61ms β 569.25ms β no change β
β QQuery 30 β 834.68ms β 849.39ms β no change β
β QQuery 31 β 874.85ms β 911.69ms β no change β
β QQuery 32 β 2620.91ms β 2712.46ms β no change β
β QQuery 33 β 3374.68ms β 3414.17ms β no change β
β QQuery 34 β 3508.23ms β 3471.25ms β no change β
β QQuery 35 β 1299.74ms β 1283.55ms β no change β
β QQuery 36 β 168.83ms β 178.94ms β 1.06x slower β
β QQuery 37 β 100.58ms β 104.48ms β no change β
β QQuery 38 β 169.06ms β 175.10ms β no change β
β QQuery 39 β 251.47ms β 256.55ms β no change β
β QQuery 40 β 84.92ms β 88.19ms β no change β
β QQuery 41 β 85.41ms β 83.15ms β no change β
β QQuery 42 β 79.02ms β 77.87ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 59102.46ms β
β Total Time (intermeidate-result-blocked-approach) β 59614.05ms β
β Average Time (HEAD) β 1374.48ms β
β Average Time (intermeidate-result-blocked-approach) β 1386.37ms β
β Queries Faster β 2 β
β Queries Slower β 3 β
β Queries with No Change β 38 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
ββββββββββββββββ³βββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 123.82ms β 124.51ms β no change β
β QQuery 2 β 23.48ms β 23.82ms β no change β
β QQuery 3 β 35.83ms β 34.47ms β no change β
β QQuery 4 β 21.19ms β 21.19ms β no change β
β QQuery 5 β 55.49ms β 55.51ms β no change β
β QQuery 6 β 12.07ms β 12.25ms β no change β
β QQuery 7 β 102.21ms β 102.45ms β no change β
β QQuery 8 β 26.52ms β 25.42ms β no change β
β QQuery 9 β 63.17ms β 60.87ms β no change β
β QQuery 10 β 57.73ms β 58.82ms β no change β
β QQuery 11 β 13.17ms β 12.91ms β no change β
β QQuery 12 β 44.63ms β 45.62ms β no change β
β QQuery 13 β 29.14ms β 30.79ms β 1.06x slower β
β QQuery 14 β 9.69ms β 9.76ms β no change β
β QQuery 15 β 24.90ms β 24.88ms β no change β
β QQuery 16 β 24.00ms β 23.69ms β no change β
β QQuery 17 β 96.51ms β 97.27ms β no change β
β QQuery 18 β 241.93ms β 232.20ms β no change β
β QQuery 19 β 27.85ms β 28.84ms β no change β
β QQuery 20 β 39.30ms β 38.43ms β no change β
β QQuery 21 β 173.40ms β 172.61ms β no change β
β QQuery 22 β 17.56ms β 16.65ms β +1.05x faster β
ββββββββββββββββ΄βββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1263.57ms β
β Total Time (intermeidate-result-blocked-approach) β 1252.96ms β
β Average Time (HEAD) β 57.44ms β
β Average Time (intermeidate-result-blocked-approach) β 56.95ms β
β Queries Faster β 1 β
β Queries Slower β 1 β
β Queries with No Change β 20 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermeidate result blocked approach sketch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2840614135
This is very much on my list to review, but I am backed up and likely won't have a chance for another day or two
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075154369
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,20 +94,27 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
Review Comment:
> yes, i mean block size. I would expect something like batchsize (4-8k), or maybe even bigger to have lower overhead? Did you run some experiments?
Yes, I try it.
Now I set `block_size = batch_size`.
I try the smaller `batch_size` like 1024, and `this pr` show improvement compared to `main`.
It is due to `this pr` can also eliminating the call of `Array::slice`, which is non-trivial due to the computation of null count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072859880
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs:
##########
@@ -507,3 +509,157 @@ pub(crate) fn slice_and_maybe_filter(
Ok(sliced_arrays)
}
}
+
+// ===============================================
+// Useful tools for group index
+// ===============================================
+
+/// Operations about group index parsing
+///
+/// There are mainly 2 `group index` needing parsing: `flat` and `blocked`.
+///
+/// # Flat group index
+/// `flat group index` format is like:
+///
+/// ```text
+/// | block_offset(64bit) |
+/// ```
+///
+/// It is used in `flat GroupValues/GroupAccumulator`, only a single block
+/// exists, so its `block_id` is always 0, and use all 64 bits to store the
+/// `block offset`.
+///
+/// # Blocked group index
+/// `blocked group index` format is like:
+///
+/// ```text
+/// | block_id(32bit) | block_offset(32bit)
+/// ```
+///
+/// It is used in `blocked GroupValues/GroupAccumulator`, multiple blocks
+/// exist, and we use high 32 bits to store `block_id`, and low 32 bit to
+/// store `block_offset`.
+///
+/// The `get_block_offset` method requires to return `block_offset` as u64,
+/// that is for compatible for `flat group index`'s parsing.
+///
+pub trait GroupIndexOperations: Debug {
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Impl intermeidate result blocked approach sketch [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2823308363
I add a query in `extened.sql`, the blocked approach can get a obvious improvement as expected:
- sql:
```sql
SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
```
- disable blocked approach:
```
Query 7 iteration 0 took 8532.5 ms and returned 10 rows
Query 7 iteration 1 took 8214.4 ms and returned 10 rows
Query 7 iteration 2 took 8017.6 ms and returned 10 rows
Query 7 iteration 3 took 8283.9 ms and returned 10 rows
Query 7 iteration 4 took 7913.1 ms and returned 10 rows
Query 7 avg time: 8192.27 ms
```
- enable blocked approach:
```
Query 7 iteration 0 took 7613.3 ms and returned 10 rows
Query 7 iteration 1 took 7252.4 ms and returned 10 rows
Query 7 iteration 2 took 7520.5 ms and returned 10 rows
Query 7 iteration 3 took 7193.6 ms and returned 10 rows
Query 7 iteration 4 took 7235.4 ms and returned 10 rows
Query 7 avg time: 7363.00 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Impl intermeidate result blocked approach sketch [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2823308375
I add a query in `extened.sql`, the blocked approach can get a obvious improvement as expected:
- sql:
```sql
SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
```
- disable blocked approach:
```
Query 7 iteration 0 took 8532.5 ms and returned 10 rows
Query 7 iteration 1 took 8214.4 ms and returned 10 rows
Query 7 iteration 2 took 8017.6 ms and returned 10 rows
Query 7 iteration 3 took 8283.9 ms and returned 10 rows
Query 7 iteration 4 took 7913.1 ms and returned 10 rows
Query 7 avg time: 8192.27 ms
```
- enable blocked approach:
```
Query 7 iteration 0 took 7613.3 ms and returned 10 rows
Query 7 iteration 1 took 7252.4 ms and returned 10 rows
Query 7 iteration 2 took 7520.5 ms and returned 10 rows
Query 7 iteration 3 took 7193.6 ms and returned 10 rows
Query 7 iteration 4 took 7235.4 ms and returned 10 rows
Query 7 avg time: 7363.00 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075095403
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,20 +94,27 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
Review Comment:
> this seems rather small? won't larger values have lower overhead?
Does it mean block size?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Impl intermeidate result blocked approach sketch [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2833519008
@Dandandan @alamb this pr may be ready now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3405892443
Maybe someday we'll get back to it π’
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3080567065
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ collections::VecDeque,
+ fmt::Debug,
+ iter,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct Blocks<B: Block> {
+ inner: VecDeque<B>,
Review Comment:
See changes in https://github.com/apache/datafusion/pull/21622/commits/baa7755a200f5c82998cdc3718280226b12ec70c#diff-c8207420967623630914abf198f93c8b6b2ccb0ba30aa6d1b5b641643789b92fR39
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4346040055
Refactoring for zero cost when disabling the blocked approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4343775983
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4343604373)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.20 / 4.69 Β±6.81 / 18.31 ms β 1.24 / 4.81 Β±6.91 / 18.63 ms β no change β
β QQuery 1 β 12.31 / 12.90 Β±0.32 / 13.19 ms β 13.31 / 13.63 Β±0.24 / 13.98 ms β 1.06x slower β
β QQuery 2 β 37.22 / 37.65 Β±0.25 / 37.95 ms β 36.69 / 36.97 Β±0.33 / 37.61 ms β no change β
β QQuery 3 β 32.18 / 33.68 Β±1.39 / 35.77 ms β 32.71 / 33.30 Β±0.64 / 34.13 ms β no change β
β QQuery 4 β 248.07 / 258.84 Β±8.10 / 268.82 ms β 273.71 / 282.89 Β±9.01 / 299.80 ms β 1.09x slower β
β QQuery 5 β 286.89 / 297.68 Β±9.46 / 312.31 ms β 286.98 / 298.35 Β±6.28 / 305.42 ms β no change β
β QQuery 6 β 7.52 / 7.75 Β±0.24 / 8.16 ms β 6.82 / 7.50 Β±0.36 / 7.82 ms β no change β
β QQuery 7 β 14.90 / 15.08 Β±0.12 / 15.20 ms β 14.12 / 14.32 Β±0.12 / 14.46 ms β +1.05x faster β
β QQuery 8 β 337.53 / 345.55 Β±9.01 / 362.21 ms β 333.37 / 351.74 Β±9.95 / 363.07 ms β no change β
β QQuery 9 β 464.60 / 483.70 Β±16.12 / 510.06 ms β 461.18 / 474.85 Β±16.45 / 505.45 ms β no change β
β QQuery 10 β 75.17 / 76.56 Β±0.87 / 77.89 ms β 74.67 / 76.74 Β±3.02 / 82.73 ms β no change β
β QQuery 11 β 86.06 / 87.43 Β±1.05 / 88.94 ms β 87.27 / 88.03 Β±0.48 / 88.76 ms β no change β
β QQuery 12 β 283.93 / 296.27 Β±12.32 / 317.43 ms β 299.81 / 303.52 Β±3.50 / 308.05 ms β no change β
β QQuery 13 β 408.64 / 420.85 Β±9.93 / 437.20 ms β 433.94 / 444.02 Β±9.34 / 459.35 ms β 1.06x slower β
β QQuery 14 β 291.08 / 300.86 Β±7.46 / 307.85 ms β 286.68 / 306.20 Β±10.34 / 316.37 ms β no change β
β QQuery 15 β 292.78 / 302.90 Β±8.71 / 313.91 ms β 316.43 / 342.96 Β±17.00 / 359.50 ms β 1.13x slower β
β QQuery 16 β 650.21 / 660.55 Β±13.01 / 685.91 ms β 620.71 / 645.65 Β±16.13 / 663.58 ms β no change β
β QQuery 17 β 639.90 / 655.32 Β±10.36 / 669.82 ms β 633.90 / 654.99 Β±14.13 / 676.69 ms β no change β
β QQuery 18 β 1327.57 / 1355.57 Β±16.41 / 1376.44 ms β 1317.82 / 1321.98 Β±5.75 / 1333.21 ms β no change β
β QQuery 19 β 30.08 / 32.87 Β±4.67 / 42.16 ms β 28.83 / 36.34 Β±10.06 / 54.79 ms β 1.11x slower β
β QQuery 20 β 520.14 / 533.95 Β±12.12 / 548.47 ms β 523.39 / 529.56 Β±7.76 / 544.75 ms β no change β
β QQuery 21 β 599.06 / 608.53 Β±6.16 / 616.52 ms β 602.97 / 611.65 Β±6.15 / 620.66 ms β no change β
β QQuery 22 β 1080.59 / 1093.18 Β±12.40 / 1114.58 ms β 1085.22 / 1096.94 Β±6.83 / 1104.52 ms β no change β
β QQuery 23 β 3399.00 / 3476.23 Β±51.02 / 3533.61 ms β 3417.06 / 3452.18 Β±38.39 / 3524.14 ms β no change β
β QQuery 24 β 42.58 / 43.01 Β±0.50 / 43.98 ms β 44.93 / 46.95 Β±2.69 / 51.88 ms β 1.09x slower β
β QQuery 25 β 114.67 / 117.89 Β±2.13 / 120.31 ms β 114.49 / 116.49 Β±1.96 / 119.04 ms β no change β
β QQuery 26 β 44.09 / 46.08 Β±2.23 / 50.34 ms β 43.42 / 44.42 Β±1.80 / 48.02 ms β no change β
β QQuery 27 β 671.04 / 684.12 Β±9.27 / 694.03 ms β 691.05 / 701.46 Β±11.09 / 722.79 ms β no change β
β QQuery 28 β 3053.45 / 3071.24 Β±19.18 / 3105.73 ms β 3068.83 / 3108.85 Β±39.04 / 3180.45 ms β no change β
β QQuery 29 β 43.02 / 44.37 Β±1.62 / 47.54 ms β 42.37 / 50.09 Β±14.53 / 79.14 ms β 1.13x slower β
β QQuery 30 β 307.50 / 315.86 Β±10.46 / 335.97 ms β 313.14 / 325.33 Β±8.08 / 335.93 ms β no change β
β QQuery 31 β 306.97 / 321.06 Β±10.68 / 336.95 ms β 310.16 / 324.41 Β±11.33 / 338.17 ms β no change β
β QQuery 32 β 1066.60 / 1077.99 Β±13.14 / 1103.24 ms β 1070.57 / 1099.60 Β±29.96 / 1152.43 ms β no change β
β QQuery 33 β 1495.03 / 1532.43 Β±41.29 / 1605.16 ms β 1484.13 / 1504.08 Β±17.25 / 1534.34 ms β no change β
β QQuery 34 β 1519.36 / 1536.75 Β±22.40 / 1580.33 ms β 1518.67 / 1563.30 Β±32.55 / 1609.63 ms β no change β
β QQuery 35 β 304.44 / 325.27 Β±11.11 / 336.38 ms β 368.67 / 411.91 Β±46.46 / 498.51 ms β 1.27x slower β
β QQuery 36 β 62.39 / 67.07 Β±4.41 / 74.08 ms β 66.32 / 68.84 Β±2.05 / 71.17 ms β no change β
β QQuery 37 β 36.64 / 38.24 Β±1.12 / 39.64 ms β 35.45 / 37.60 Β±2.82 / 43.07 ms β no change β
β QQuery 38 β 41.88 / 47.12 Β±6.11 / 58.64 ms β 40.89 / 45.55 Β±7.32 / 59.97 ms β no change β
β QQuery 39 β 130.92 / 137.60 Β±5.43 / 143.97 ms β 122.17 / 139.96 Β±14.48 / 159.49 ms β no change β
β QQuery 40 β 15.72 / 16.03 Β±0.30 / 16.41 ms β 14.24 / 14.49 Β±0.19 / 14.77 ms β +1.11x faster β
β QQuery 41 β 14.69 / 14.83 Β±0.09 / 14.93 ms β 13.64 / 14.42 Β±1.16 / 16.71 ms β no change β
β QQuery 42 β 13.96 / 16.36 Β±4.32 / 24.99 ms β 14.26 / 20.25 Β±8.14 / 36.17 ms β 1.24x slower β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 20851.93ms β
β Total Time (intermeidate-result-blocked-approach) β 21067.12ms β
β Average Time (HEAD) β 484.93ms β
β Average Time (intermeidate-result-blocked-approach) β 489.93ms β
β Queries Faster β 2 β
β Queries Slower β 9 β
β Queries with No Change β 32 β
β Queries with Failure β 0 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 29.9 GiB |
| Avg memory | 23.0 GiB |
| CPU user | 1101.0s |
| CPU sys | 69.4s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 30.9 GiB |
| Avg memory | 23.4 GiB |
| CPU user | 1106.9s |
| CPU sys | 71.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4334353557
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276086769
Benchmark for [this request](https://github.com/apache/datafusion/pull/15591#issuecomment-4276055952) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
BENCHMARK: tpch
QUERY: All
DATAFUSION_DIR: /workspace/datafusion-base
BRANCH_NAME: HEAD
DATA_DIR: /workspace/datafusion-bench/benchmarks/data
RESULTS_DIR: /workspace/datafusion-bench/benchmarks/results/HEAD
CARGO_COMMAND: cargo run --release
PREFER_HASH_JOIN: true
SIMULATE_LATENCY: false
***************************
RESULTS_FILE: /workspace/datafusion-bench/benchmarks/results/HEAD/tpch_sf1.json
Running tpch benchmark...
+ cargo run --release --bin dfbench -- tpch --iterations 5 --path /workspace/datafusion-bench/benchmarks/data/tpch_sf1 --scale-factor 1 --prefer_hash_join true --format parquet -o /workspace/datafusion-bench/benchmarks/results/HEAD/tpch_sf1.json
Finished `release` profile [optimized] target(s) in 0.22s
Running `/workspace/datafusion-base/target/release/dfbench tpch --iterations 5 --path /workspace/datafusion-bench/benchmarks/data/tpch_sf1 --scale-factor 1 --prefer_hash_join true --format parquet -o /workspace/datafusion-bench/benchmarks/results/HEAD/tpch_sf1.json`
error: unexpected argument '--scale-factor' found
Usage: dfbench tpch --path <PATH> --iterations <ITERATIONS>
For more information, try '--help'.
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3265468487
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
Review Comment:
The problem isn't that we're not transitioning to ProducingBlocks early (this can be done via the `skip partial agg feature`), the problem is that:
* emitting a large RecordBatch releases the memory reservation occupied by this batch, but the memory is not freed, it is used in the ProducingOutput state while producing small output batches
* ProducingBlocks slices the large RecordBatch into smaller ones, but calling `get_array_memory_size` on the small record batch returns the memory of the entire original large RecordBatch, so spilling is broken for the parent operator (e.g. repartitioning)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4512250916
I'm wondering whether this will be merged soon, it solves a lot of issues with the aggregation and I don't want to spend too much time chasing issues in the existing implementation, such as https://github.com/apache/datafusion/pull/22165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alchemist51 (via GitHub)" <gi...@apache.org>.
alchemist51 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4195565002
This branch still has conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3104443249
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,308 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ collections::VecDeque,
+ fmt::Debug,
+ iter,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct Blocks<B: Block> {
+ inner: VecDeque<B>,
Review Comment:
Make sense, I am switching it to Vec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alchemist51 (via GitHub)" <gi...@apache.org>.
alchemist51 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3747616141
@Rachelint any ETA you are targeting for making it ready for review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4739592326
> How about we split this pr into twos or mores? And push forward the part one (GroupValues and GroupAccumulator) in parallel with the aggregation refactoring?
I think it's a good idea, this is important work and it would be easier to review if split into smaller PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4742353436
> > Marking as a draft as I don't think this one is ready to merge quite yet and I am trying to clean up the review / merge queue
>
> Yes, and I think the whole feature will be suitable to push forward after the aggregation refactoring stable.
>
> Howerver, actually to parts are included in this:
>
> * One part is about refactoring `GroupValues` and `GroupAccumulator`
> * The other part is about applying the blocked logic in aggreagating
>
> How about we split this pr into twos or mores? And push forward the part one (`GroupValues` and `GroupAccumulator`) in parallel with the aggregation refactoring? @alamb @2010YOUY01 @ariel-miculas
I think the steps are
1. Complete https://github.com/apache/datafusion/issues/22710
2. Initial PR for blocked states: The major issue is to agree on the API changes for `GroupValues` and `GroupsAccumulator`, and how to organize future works.
3. Update all `GroupValues` and `GroupsAccumulator` (There are around 20 of them IIRC)
The performance seems to be a nearly solved issue, the PoC already showed high cardinality cases are faster (with several micro optimizations left on the table), low cardinality is slightly slower but @alamb's suggestion in https://github.com/apache/datafusion/pull/22712#issuecomment-4672476038 is doable I think, to bring back the performance.
I suggest not trying to parallelize steps 1 and 2, as they will likely conflict with each other. Step 3 should be highly parallelizable.
As for the refactoring progress, I'd estimate it's about 50% complete. I haven't seen any major technical blockers so farβjust need some time to better structure the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4757178261
> I disagree, since the memory management is directly tied to performance via the spilling mechanism when running with memory limits configured.
Good point, no difference to performance is actually just in benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4778636320
> We can run benchmarks with memory limits to force spilling if that helps
@adriangb hello, is it possible to authorize me to trigger benchmark through bot?
I am trying to improve aggregation performance recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4244928294
Busy in work recent days... Fixing the last conflicts now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5331380495
So I took this pr and added support locally for binary and it fixed the issue we had with huge batches coming out of aggregate, however I have concerns because ideally It should be enabled in all cases in order for it to be benefitial - for us it's partial/final with both spill/emit early support but there.
1. Grouping impl will be complicated: Adding support for more columns types and multi group by is complicated
2. In the very hot loop of doing aggregation and jumping in memory you are adding another memory jump
3. Complicated aggregate expression code - even though the idea is simple, the actual implementation that needed to be done to support that is really complicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5400931576
> @rluvaton @alamb @Dandandan
>
> Iβm actually more concerned about the implementation complexity, especially for complex typesβthe work required just to extract the trait was already quite painful.
>
> Iβm somewhat less concerned about the performance impact. Based on the benchmarks that have been run throughout this PR, the regression appears relatively modest, ranging from roughly neutral to around 10%.
However, I think we would likely reject a PR that has a 10% slowdown on some aggregates (it is a core feature of DataFusion)
I agree the complexity is wild -- that is why I am trying to help @2010YOUY01 refactor the aggregate streams so it is actually possible (and maybe more plausible to implement it incrementally)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5426889129
One of the problem with removing `EmitTo::First(n)` is:
lets say batch size is 1000
the input is ordered by group by keys so you know when a group is finished (or other case where we can emit early)
and lets say we get to group 900 and in the middle of it and then we no longer have more memory
we won't be able to say emit first 899 as we can only emit next block, and when calling emit next block it will treat all current items in the block as finished, which is wrong
Did not solve that problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3353185195
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
Review Comment:
Serialize and spill the batch to disk.
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,387 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Aggregation intermediate results blocks in blocked approach
+
+use std::{
+ fmt::Debug,
+ iter, mem,
+ ops::{Index, IndexMut},
+};
+
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// Internally uses a `Vec<B>` with a `start` offset to track the first active
+/// block. When blocks are popped via [`Self::pop_block`], the block is swapped
+/// out in O(1) using `mem::replace` and the `start` cursor advances, avoiding
+/// the O(n) shift cost of `Vec::remove(0)`.
Review Comment:
> This avoids the O(n) shift but it doesn't free the memory, I'm wondering whether a VecDeque would work better here, so inner: VecDeque instead of Vec
The memory usage of `Vec` may not be large (8bytes(pointer) * num_blocks).
But index op of `VecDeque ` is very expansive compared with `Vec`, so I think it may be worth to waste few memory to get better performance?
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/block_store/vec_block_store.rs:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`BlockStore`] wrapper specialized for stores whose blocks are `Vec<T>`.
+
+use std::fmt::Debug;
+use std::iter;
+use std::marker::PhantomData;
+use std::ops::{Index, IndexMut};
+
+use datafusion_common::utils::split_vec_min_alloc;
+use datafusion_common::{Result, internal_datafusion_err, internal_err};
+use datafusion_expr_common::groups_accumulator::EmitTo;
+
+use crate::aggregate::groups_accumulator::block_store::{Block, BlockStore};
+
+/// Thin wrapper around a [`BlockStore<Vec<T>>`] that adds vector-specific
+/// emit semantics on top of the generic block store API.
+///
+/// Emitting `EmitTo::First` requires vector-specific split semantics, so it
+/// lives on this wrapper rather than on [`BlockStore`] itself. The wrapper
+/// re-exposes every [`BlockStore`] method by delegation, plus an [`emit`]
+/// method implemented purely via [`BlockStore::push_block`] /
+/// [`BlockStore::pop_block`] so it works uniformly over flat and blocked
+/// storage.
+///
+/// [`emit`]: VecBlockStore::emit
+#[derive(Debug)]
+pub struct VecBlockStore<T, S>
+where
+ T: Clone + Debug,
+ S: BlockStore<Vec<T>>,
+{
+ inner: S,
+ _phantom: PhantomData<T>,
+}
+
+impl<T, S> VecBlockStore<T, S>
+where
+ T: Clone + Debug,
+ S: BlockStore<Vec<T>>,
+{
+ /// Wrap an existing block store.
+ pub fn new(inner: S) -> Self {
+ Self {
+ inner,
+ _phantom: PhantomData,
+ }
+ }
+
+ // ---- BlockStore method delegation ----------------------------------
+ pub fn reserve_blocks(&mut self) {
+ self.inner.reserve_blocks();
+ }
+
+ pub fn resize(&mut self, total_num_groups: usize, default_value: T) {
+ self.inner.resize(total_num_groups, default_value);
+ }
+
+ pub fn num_blocks(&self) -> usize {
+ self.inner.num_blocks()
+ }
+
+ pub fn block_size(&self) -> Option<usize> {
+ self.inner.block_size()
+ }
+
+ pub fn clear(&mut self) {
+ self.inner.clear();
+ }
+
+ // ---- Emit ----------------------------------------------------------
+
+ /// Emit values according to `emit_to`, expressed only in terms of
+ /// [`BlockStore::push_block`] and [`BlockStore::pop_block`].
+ ///
+ /// - [`EmitTo::All`]: drains every block via repeated `pop_block` and
+ /// concatenates the results into a single `Vec<T>`.
+ /// - [`EmitTo::First`]`(n)`: pops the first block, splits off the first
+ /// `n` values, and pushes the remainder back. Only meaningful when the
+ /// first block holds at least `n` values (true for flat storage); the
+ /// call returns an internal error otherwise.
+ /// - [`EmitTo::NextBlock`]: pops a single block.
+ pub fn emit(&mut self, emit_to: EmitTo) -> Result<Vec<T>> {
+ match emit_to {
+ EmitTo::All => self.inner.pop_block().ok_or_else(|| {
Review Comment:
Maybe a bit confusing now due to:
- `EmitTo::First` and `EmitTo::All` only meaningful in `flat mode`
- `EmitTo::NextBlock` only meaningful in `blocked mode`
Here (`vec_block_store.rs`) maybe more suitable to return error when encountering `EmitTo::First` and `EmitTo::All`?
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/block_store/mod.rs:
##########
@@ -0,0 +1,60 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Storage abstraction for aggregation intermediate result blocks.
+
+use std::fmt::Debug;
+use std::ops::{Index, IndexMut};
+
+use crate::aggregate::groups_accumulator::blocks::Block;
+
+pub mod blocked;
+pub mod flat;
+pub mod vec_values;
+
+pub use blocked::BlockedBlockStore;
+pub use flat::FlatBlockStore;
+pub use vec_values::{VecValues, VecValuesBlockStore};
+
+/// Storage abstraction for aggregation intermediate result blocks.
+///
+/// [`BlockStore`] lets flat and blocked group state share the same accumulation
+/// flow while using different physical layouts. Implementations should keep
+/// block lookup cheap because it is used by per-row accumulator update paths.
+pub trait BlockStore<B: Block>:
Review Comment:
@Dandandan @ariel-miculas @2010YOUY01 @alamb
I think the greatest difficulty currently being faced is that how to abstract the common trait (BlockStore here) to help conveniently impl blocked accumulator and group values
Current one can work and zero cost when we disable blocked mode, but I think it seems too complex...
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/block_store/blocked.rs:
##########
@@ -0,0 +1,436 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Blocked [`BlockStore`] implementation.
+
+use std::{
+ fmt::Debug,
+ mem,
+ ops::{Index, IndexMut},
+};
+
+use crate::aggregate::groups_accumulator::block_store::{Block, BlockStore};
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// Blocks are popped in FIFO order by keeping a cursor into `inner`. Popping a
+/// block swaps it out in O(1) using `mem::take` and advances the cursor,
+/// avoiding the O(n) shift cost of `Vec::remove(0)`.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct BlockedBlockStore<B: Block> {
+ inner: Vec<B>,
+ /// Index of the next active block.
+ cursor: usize,
+ block_size: usize,
+}
+
+impl<B: Block> BlockedBlockStore<B> {
+ /// Create a new blocked store with the given fixed block size.
+ pub fn new(block_size: usize) -> Self {
+ Self {
+ inner: Vec::new(),
+ cursor: 0,
+ block_size,
+ }
+ }
+}
+
+impl<B: Block> BlockStore<B> for BlockedBlockStore<B> {
+ fn push_block(&mut self, block: B) {
+ self.inner.push(block);
+ }
+
+ fn pop_block(&mut self) -> Option<B> {
+ if self.cursor >= self.inner.len() {
+ return None;
+ }
+
+ let block = mem::take(&mut self.inner[self.cursor]);
Review Comment:
`VecDeque` is used at the beginning, but the `index op` in `VecDeque` actually not really trivial as `Vec`, and switch to `Vec` later suggested by reviews.
But I agree with that `VecDeque` may be more suitable, due to:
- its main target is saving memory (although Vec<Empty block> will not occupy much)
- the logic can become simple, and I think as a initial pr, it should keep as simple as possible.
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/block_store/blocked.rs:
##########
@@ -0,0 +1,436 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Blocked [`BlockStore`] implementation.
+
+use std::{
+ fmt::Debug,
+ mem,
+ ops::{Index, IndexMut},
+};
+
+use crate::aggregate::groups_accumulator::block_store::{Block, BlockStore};
+
+/// Structure used to store aggregation intermediate results in `blocked approach`
+///
+/// Aggregation intermediate results will be stored as multiple [`Block`]s
+/// (simply you can think a [`Block`] as a `Vec`). And `Blocks` is the structure
+/// to represent such multiple [`Block`]s.
+///
+/// Blocks are popped in FIFO order by keeping a cursor into `inner`. Popping a
+/// block swaps it out in O(1) using `mem::take` and advances the cursor,
+/// avoiding the O(n) shift cost of `Vec::remove(0)`.
+///
+/// More details about `blocked approach` can see in: [`GroupsAccumulator::supports_blocked_groups`].
+///
+/// [`GroupsAccumulator::supports_blocked_groups`]: datafusion_expr_common::groups_accumulator::GroupsAccumulator::supports_blocked_groups
+///
+#[derive(Debug)]
+pub struct BlockedBlockStore<B: Block> {
Review Comment:
Methods in `Block` trait is mainly abstracted from logic in `prim_op.rs`, and may only useful for `accumulators`.
Maybe we should define some trait different for `groups` case?
For me, how to define the suitable traits is actually a main block point...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5568988959
The diagram for EmitTo::Last(n) is accurate only if there's a shrink_to_fit() call after every emit, which means reallocating and copying all the elements from the old allocation to the new allocation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Impl intermeidate result blocked approach sketch [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2818462767
It is very close, just need to add more tests!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848557998
> π€: Benchmark completed
> Details
>
> ```
> Comparing HEAD and intermeidate-result-blocked-approach
> --------------------
> Benchmark clickbench_extended.json
> --------------------
> ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
> β Query β HEAD β intermeidate-result-blocked-approach β Change β
> β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β QQuery 0 β 1921.22ms β 1898.23ms β no change β
> β QQuery 1 β 657.26ms β 694.92ms β 1.06x slower β
> β QQuery 2 β 1398.94ms β 1419.32ms β no change β
> β QQuery 3 β 713.54ms β 717.51ms β no change β
> β QQuery 4 β 1498.02ms β 1511.57ms β no change β
> β QQuery 5 β 15341.63ms β 15578.65ms β no change β
> β QQuery 6 β 2029.05ms β 2049.81ms β no change β
> ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
> β Benchmark Summary β β
> β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β Total Time (HEAD) β 23559.67ms β
> β Total Time (intermeidate-result-blocked-approach) β 23870.00ms β
> β Average Time (HEAD) β 3365.67ms β
> β Average Time (intermeidate-result-blocked-approach) β 3410.00ms β
> β Queries Faster β 0 β
> β Queries Slower β 1 β
> β Queries with No Change β 6 β
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
> --------------------
> Benchmark clickbench_partitioned.json
> --------------------
> ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
> β Query β HEAD β intermeidate-result-blocked-approach β Change β
> β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β QQuery 0 β 2.32ms β 2.31ms β no change β
> β QQuery 1 β 36.90ms β 38.43ms β no change β
> β QQuery 2 β 91.60ms β 93.54ms β no change β
> β QQuery 3 β 100.24ms β 99.54ms β no change β
> β QQuery 4 β 777.83ms β 801.76ms β no change β
> β QQuery 5 β 860.11ms β 850.06ms β no change β
> β QQuery 6 β 2.10ms β 2.13ms β no change β
> β QQuery 7 β 41.49ms β 43.64ms β 1.05x slower β
> β QQuery 8 β 925.11ms β 877.46ms β +1.05x faster β
> β QQuery 9 β 1201.13ms β 1193.97ms β no change β
> β QQuery 10 β 262.49ms β 270.08ms β no change β
> β QQuery 11 β 302.47ms β 313.89ms β no change β
> β QQuery 12 β 902.20ms β 956.22ms β 1.06x slower β
> β QQuery 13 β 1325.06ms β 1388.51ms β no change β
> β QQuery 14 β 826.08ms β 851.33ms β no change β
> β QQuery 15 β 1026.88ms β 1061.12ms β no change β
> β QQuery 16 β 1720.69ms β 1731.12ms β no change β
> β QQuery 17 β 1582.12ms β 1599.34ms β no change β
> β QQuery 18 β 3066.54ms β 3254.59ms β 1.06x slower β
> β QQuery 19 β 84.76ms β 82.92ms β no change β
> β QQuery 20 β 1116.50ms β 1178.43ms β 1.06x slower β
> β QQuery 21 β 1315.33ms β 1318.33ms β no change β
> β QQuery 22 β 2190.24ms β 2170.50ms β no change β
> β QQuery 23 β 8290.39ms β 8370.27ms β no change β
> β QQuery 24 β 470.87ms β 486.49ms β no change β
> β QQuery 25 β 388.04ms β 399.37ms β no change β
> β QQuery 26 β 527.09ms β 542.36ms β no change β
> β QQuery 27 β 1686.91ms β 1727.53ms β no change β
> β QQuery 28 β 12444.07ms β 12631.97ms β no change β
> β QQuery 29 β 525.46ms β 534.61ms β no change β
> β QQuery 30 β 798.93ms β 822.60ms β no change β
> β QQuery 31 β 829.79ms β 870.36ms β no change β
> β QQuery 32 β 2594.87ms β 2731.68ms β 1.05x slower β
> β QQuery 33 β 3320.81ms β 3380.18ms β no change β
> β QQuery 34 β 3357.44ms β 3387.27ms β no change β
> β QQuery 35 β 1290.00ms β 1282.24ms β no change β
> β QQuery 36 β 130.81ms β 126.75ms β no change β
> β QQuery 37 β 57.49ms β 58.36ms β no change β
> β QQuery 38 β 127.97ms β 128.85ms β no change β
> β QQuery 39 β 204.48ms β 207.11ms β no change β
> β QQuery 40 β 50.31ms β 48.89ms β no change β
> β QQuery 41 β 48.25ms β 44.18ms β +1.09x faster β
> β QQuery 42 β 39.38ms β 38.45ms β no change β
> ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
> β Benchmark Summary β β
> β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β Total Time (HEAD) β 56943.53ms β
> β Total Time (intermeidate-result-blocked-approach) β 57998.74ms β
> β Average Time (HEAD) β 1324.27ms β
> β Average Time (intermeidate-result-blocked-approach) β 1348.81ms β
> β Queries Faster β 2 β
> β Queries Slower β 5 β
> β Queries with No Change β 36 β
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
> --------------------
> Benchmark tpch_mem_sf1.json
> --------------------
> ββββββββββββββββ³βββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
> β Query β HEAD β intermeidate-result-blocked-approach β Change β
> β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β QQuery 1 β 124.78ms β 130.74ms β no change β
> β QQuery 2 β 24.32ms β 23.65ms β no change β
> β QQuery 3 β 35.37ms β 35.46ms β no change β
> β QQuery 4 β 20.87ms β 20.87ms β no change β
> β QQuery 5 β 54.85ms β 57.66ms β 1.05x slower β
> β QQuery 6 β 12.21ms β 12.64ms β no change β
> β QQuery 7 β 99.22ms β 102.82ms β no change β
> β QQuery 8 β 26.11ms β 25.50ms β no change β
> β QQuery 9 β 63.11ms β 64.15ms β no change β
> β QQuery 10 β 57.98ms β 59.17ms β no change β
> β QQuery 11 β 13.04ms β 13.20ms β no change β
> β QQuery 12 β 43.72ms β 44.49ms β no change β
> β QQuery 13 β 28.90ms β 29.45ms β no change β
> β QQuery 14 β 9.78ms β 10.58ms β 1.08x slower β
> β QQuery 15 β 26.05ms β 25.35ms β no change β
> β QQuery 16 β 23.45ms β 23.66ms β no change β
> β QQuery 17 β 95.93ms β 97.78ms β no change β
> β QQuery 18 β 240.22ms β 238.96ms β no change β
> β QQuery 19 β 27.56ms β 28.63ms β no change β
> β QQuery 20 β 38.46ms β 38.79ms β no change β
> β QQuery 21 β 173.00ms β 168.56ms β no change β
> β QQuery 22 β 18.15ms β 17.81ms β no change β
> ββββββββββββββββ΄βββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
> β Benchmark Summary β β
> β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
> β Total Time (HEAD) β 1257.09ms β
> β Total Time (intermeidate-result-blocked-approach) β 1269.90ms β
> β Average Time (HEAD) β 57.14ms β
> β Average Time (intermeidate-result-blocked-approach) β 57.72ms β
> β Queries Faster β 0 β
> β Queries Slower β 2 β
> β Queries with No Change β 20 β
> βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
> ```
Emmm... As expected the new added query is not convered, I think I should submit an new pr for adding the query to `extended.sql` for checking the improvement...
```
SELECT "WatchID", MIN("ResolutionWidth"), MAX("ResolutionWidth"), SUM("IsRefresh") FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848547823
> how is the benchmark triggered and can we run clickbench extended too?
I am using some scripts in https://github.com/alamb/datafusion-benchmarking on a gcp machine (I haven't figured out how we would do this for the community in general
I didn't run the extended tests because they ran into the following issue (which is how I found it initially)
- https://github.com/apache/datafusion/issues/15927
Now that they are fixed I think I can run extended tests and will kick them off
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848565268
> What would be required to improve the performance for one or more of the real clickbench queries? Implementing group management for other data types?
Yes.
Usually high cardinality aggregation can get improvement from, like `q16~q18` and `q32`(group by `UserID` or `WatchID`).
But we need to implement group management for multiple data types for `GroupValues`, or for multiple `GroupAccumulator`.
I think it may be too complex for the initial pr...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848573594
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_extended.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1937.34ms β 1929.54ms β no change β
β QQuery 1 β 664.33ms β 719.97ms β 1.08x slower β
β QQuery 2 β 1395.20ms β 1411.13ms β no change β
β QQuery 3 β 712.50ms β 710.80ms β no change β
β QQuery 4 β 1477.28ms β 1475.06ms β no change β
β QQuery 5 β 15310.28ms β 15186.79ms β no change β
β QQuery 6 β 2049.34ms β 2066.82ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 23546.29ms β
β Total Time (intermeidate-result-blocked-approach) β 23500.10ms β
β Average Time (HEAD) β 3363.76ms β
β Average Time (intermeidate-result-blocked-approach) β 3357.16ms β
β Queries Faster β 0 β
β Queries Slower β 1 β
β Queries with No Change β 6 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark clickbench_partitioned.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 2.53ms β 2.09ms β +1.21x faster β
β QQuery 1 β 35.62ms β 39.76ms β 1.12x slower β
β QQuery 2 β 89.48ms β 92.73ms β no change β
β QQuery 3 β 100.30ms β 100.22ms β no change β
β QQuery 4 β 750.96ms β 754.32ms β no change β
β QQuery 5 β 833.28ms β 857.42ms β no change β
β QQuery 6 β 2.06ms β 2.08ms β no change β
β QQuery 7 β 40.36ms β 42.52ms β 1.05x slower β
β QQuery 8 β 924.79ms β 902.95ms β no change β
β QQuery 9 β 1209.71ms β 1224.03ms β no change β
β QQuery 10 β 264.71ms β 273.51ms β no change β
β QQuery 11 β 302.87ms β 313.03ms β no change β
β QQuery 12 β 917.10ms β 899.06ms β no change β
β QQuery 13 β 1338.05ms β 1255.47ms β +1.07x faster β
β QQuery 14 β 833.92ms β 846.94ms β no change β
β QQuery 15 β 1030.69ms β 1073.94ms β no change β
β QQuery 16 β 1724.70ms β 1732.73ms β no change β
β QQuery 17 β 1605.96ms β 1581.98ms β no change β
β QQuery 18 β 3069.41ms β 3072.93ms β no change β
β QQuery 19 β 83.51ms β 85.65ms β no change β
β QQuery 20 β 1123.00ms β 1140.04ms β no change β
β QQuery 21 β 1333.50ms β 1336.50ms β no change β
β QQuery 22 β 2175.85ms β 2185.26ms β no change β
β QQuery 23 β 8284.06ms β 8470.20ms β no change β
β QQuery 24 β 461.57ms β 485.77ms β 1.05x slower β
β QQuery 25 β 384.10ms β 394.93ms β no change β
β QQuery 26 β 531.30ms β 545.07ms β no change β
β QQuery 27 β 1670.90ms β 1705.20ms β no change β
β QQuery 28 β 12167.48ms β 12606.28ms β no change β
β QQuery 29 β 526.92ms β 533.47ms β no change β
β QQuery 30 β 819.98ms β 848.41ms β no change β
β QQuery 31 β 869.73ms β 868.43ms β no change β
β QQuery 32 β 2619.87ms β 2735.34ms β no change β
β QQuery 33 β 3314.19ms β 3383.18ms β no change β
β QQuery 34 β 3382.94ms β 3381.46ms β no change β
β QQuery 35 β 1271.48ms β 1274.86ms β no change β
β QQuery 36 β 127.20ms β 126.03ms β no change β
β QQuery 37 β 57.38ms β 58.27ms β no change β
β QQuery 38 β 125.72ms β 127.26ms β no change β
β QQuery 39 β 198.86ms β 202.15ms β no change β
β QQuery 40 β 48.58ms β 49.01ms β no change β
β QQuery 41 β 47.74ms β 46.89ms β no change β
β QQuery 42 β 38.64ms β 41.29ms β 1.07x slower β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 56740.96ms β
β Total Time (intermeidate-result-blocked-approach) β 57698.64ms β
β Average Time (HEAD) β 1319.56ms β
β Average Time (intermeidate-result-blocked-approach) β 1341.83ms β
β Queries Faster β 2 β
β Queries Slower β 4 β
β Queries with No Change β 37 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
ββββββββββββββββ³βββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 124.49ms β 125.48ms β no change β
β QQuery 2 β 23.64ms β 23.86ms β no change β
β QQuery 3 β 34.94ms β 35.04ms β no change β
β QQuery 4 β 20.94ms β 21.46ms β no change β
β QQuery 5 β 54.67ms β 56.82ms β no change β
β QQuery 6 β 12.22ms β 12.09ms β no change β
β QQuery 7 β 107.09ms β 100.00ms β +1.07x faster β
β QQuery 8 β 25.48ms β 27.73ms β 1.09x slower β
β QQuery 9 β 61.36ms β 60.88ms β no change β
β QQuery 10 β 57.44ms β 58.40ms β no change β
β QQuery 11 β 13.19ms β 13.10ms β no change β
β QQuery 12 β 44.39ms β 44.66ms β no change β
β QQuery 13 β 29.29ms β 29.01ms β no change β
β QQuery 14 β 9.98ms β 9.77ms β no change β
β QQuery 15 β 25.86ms β 25.80ms β no change β
β QQuery 16 β 23.58ms β 23.34ms β no change β
β QQuery 17 β 93.78ms β 99.44ms β 1.06x slower β
β QQuery 18 β 233.64ms β 243.04ms β no change β
β QQuery 19 β 26.31ms β 28.12ms β 1.07x slower β
β QQuery 20 β 37.88ms β 37.07ms β no change β
β QQuery 21 β 164.99ms β 171.58ms β no change β
β QQuery 22 β 18.39ms β 17.12ms β +1.07x faster β
ββββββββββββββββ΄βββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1243.55ms β
β Total Time (intermeidate-result-blocked-approach) β 1263.81ms β
β Average Time (HEAD) β 56.52ms β
β Average Time (intermeidate-result-blocked-approach) β 57.45ms β
β Queries Faster β 2 β
β Queries Slower β 3 β
β Queries with No Change β 17 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2848629234
@alamb I have submitted an pr about new added query for this pr #15936
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072417977
##########
datafusion/expr-common/src/groups_accumulator.rs:
##########
@@ -250,4 +288,30 @@ pub trait GroupsAccumulator: Send {
/// This function is called once per batch, so it should be `O(n)` to
/// compute, not `O(num_groups)`
fn size(&self) -> usize;
+
+ /// Returns `true` if this accumulator supports blocked groups.
+ fn supports_blocked_groups(&self) -> bool {
+ false
+ }
+
+ /// Alter the block size in the accumulator
+ ///
+ /// If the target block size is `None`, it will use a single big
+ /// block(can think it a `Vec`) to manage the state.
+ ///
+ /// If the target block size` is `Some(blk_size)`, it will try to
+ /// set the block size to `blk_size`, and the try will only success
+ /// when the accumulator has supported blocked mode.
+ ///
+ /// NOTICE: After altering block size, all data in previous will be cleared.
Review Comment:
Yes, it will be usually used in `load back + merge` step in spilling:
- Emit the rest blocks at first
- Clear all stale data, and switch to `flat mode` and perform `sorted aggregation`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2853367261
@jayzhan211 @alamb I guess I nearly understand why
It is related to `target_partitions`, we maintain the `intermediate results` respectively in each `partition`.
So when `target_partitions` is larger, amount of `intermediate results` in one `partition` is smaller.
And when `intermediate results` assigned to `partition` is small enough, the blocked approach will almost make no difference (assume the extreme case, the `Vec` used to store results never resize).
Here is my benchmark result in a production machine (x86_64 + 16core + 3699.178CPU MHz + 64G RAM) with different `target_partitions`
- target_partitions = 4 (amazing improvement!)
```
// blocked approach
Q7: SELECT "WatchID", MIN("ResolutionWidth") as wmin, MAX("ResolutionWidth") as wmax, SUM("IsRefresh") as srefresh FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Query 7 iteration 0 took 4996.7 ms and returned 10 rows
Query 7 iteration 1 took 5177.6 ms and returned 10 rows
Query 7 iteration 2 took 5335.4 ms and returned 10 rows
Query 7 iteration 3 took 5232.7 ms and returned 10 rows
Query 7 iteration 4 took 5350.5 ms and returned 10 rows
Query 7 avg time: 5218.59 ms
// main
Q7: SELECT "WatchID", MIN("ResolutionWidth") as wmin, MAX("ResolutionWidth") as wmax, SUM("IsRefresh") as srefresh FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Query 7 iteration 0 took 8484.9 ms and returned 10 rows
Query 7 iteration 1 took 8194.4 ms and returned 10 rows
Query 7 iteration 2 took 8317.9 ms and returned 10 rows
Query 7 iteration 3 took 8183.2 ms and returned 10 rows
Query 7 iteration 4 took 8225.7 ms and returned 10 rows
Query 7 avg time: 8281.19 ms
```
- target_partitions = 8 (emmm... some improvement)
```
// blocked approach
Q7: SELECT "WatchID", MIN("ResolutionWidth") as wmin, MAX("ResolutionWidth") as wmax, SUM("IsRefresh") as srefresh FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Query 7 iteration 0 took 2599.2 ms and returned 10 rows
Query 7 iteration 1 took 2857.9 ms and returned 10 rows
Query 7 iteration 2 took 3046.3 ms and returned 10 rows
Query 7 iteration 3 took 2766.0 ms and returned 10 rows
Query 7 iteration 4 took 2830.8 ms and returned 10 rows
Query 7 avg time: 2820.04 ms
// main
Q7: SELECT "WatchID", MIN("ResolutionWidth") as wmin, MAX("ResolutionWidth") as wmax, SUM("IsRefresh") as srefresh FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Query 7 iteration 0 took 3820.0 ms and returned 10 rows
Query 7 iteration 1 took 3716.6 ms and returned 10 rows
Query 7 iteration 2 took 3728.3 ms and returned 10 rows
Query 7 iteration 3 took 3628.2 ms and returned 10 rows
Query 7 iteration 4 took 3912.6 ms and returned 10 rows
Query 7 avg time: 3761.15 ms
```
- target_partitions = 32 (sadly, almost no improvement...)
```
// blocked approach
Q7: SELECT "WatchID", MIN("ResolutionWidth") as wmin, MAX("ResolutionWidth") as wmax, SUM("IsRefresh") as srefresh FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Query 7 iteration 0 took 1383.7 ms and returned 10 rows
Query 7 iteration 1 took 1274.3 ms and returned 10 rows
Query 7 iteration 2 took 1321.7 ms and returned 10 rows
Query 7 iteration 3 took 1308.1 ms and returned 10 rows
Query 7 iteration 4 took 1310.6 ms and returned 10 rows
Query 7 avg time: 1319.68 ms
// main
Q7: SELECT "WatchID", MIN("ResolutionWidth") as wmin, MAX("ResolutionWidth") as wmax, SUM("IsRefresh") as srefresh FROM hits GROUP BY "WatchID" ORDER BY "WatchID" DESC LIMIT 10;
Query 7 iteration 0 took 1440.7 ms and returned 10 rows
Query 7 iteration 1 took 1430.3 ms and returned 10 rows
Query 7 iteration 2 took 1357.5 ms and returned 10 rows
Query 7 iteration 3 took 1352.6 ms and returned 10 rows
Query 7 iteration 4 took 1342.2 ms and returned 10 rows
Query 7 avg time: 1384.64 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2075191981
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -93,20 +94,27 @@ where
opt_filter: Option<&BooleanArray>,
total_num_groups: usize,
) -> Result<()> {
Review Comment:
But it make sense, we make `block_size` a dedicated config.
I think it can get improvement in case that `batch_size` is set to a too samll value, and the group by cardinality is actually large.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072859439
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/prim_op.rs:
##########
@@ -198,4 +232,28 @@ where
fn size(&self) -> usize {
self.values.capacity() * size_of::<T::Native>() + self.null_state.size()
}
+
+ fn supports_blocked_groups(&self) -> bool {
+ true
+ }
+
+ fn alter_block_size(&mut self, block_size: Option<usize>) -> Result<()> {
+ self.values.clear();
+ self.null_state = NullStateAdapter::new(block_size);
+ self.block_size = block_size;
+
+ Ok(())
+ }
+}
+
+impl<N: ArrowNativeTypeOp> Block for Vec<N> {
Review Comment:
Addressed.
##########
datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs:
##########
@@ -81,28 +85,49 @@ hash_float!(f16, f32, f64);
pub struct GroupValuesPrimitive<T: ArrowPrimitiveType> {
/// The data type of the output array
data_type: DataType,
+
/// Stores the group index based on the hash of its value
///
/// We don't store the hashes as hashing fixed width primitives
/// is fast enough for this not to benefit performance
- map: HashTable<usize>,
+ map: HashTable<u64>,
+
/// The group index of the null value if any
- null_group: Option<usize>,
+ null_group: Option<u64>,
+
/// The values for each group index
- values: Vec<T::Native>,
+ values: VecDeque<Vec<T::Native>>,
+
/// The random state used to generate hashes
random_state: RandomState,
+
+ /// Block size of current `GroupValues` if exist:
+ /// - If `None`, it means block optimization is disabled,
+ /// all `group values`` will be stored in a single `Vec`
+ ///
+ /// - If `Some(blk_size)`, it means block optimization is enabled,
+ /// `group values` will be stored in multiple `Vec`s, and each
+ /// `Vec` if of `blk_size` len, and we call it a `block`
+ ///
+ block_size: Option<usize>,
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r2072992809
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
@@ -982,6 +1099,9 @@ impl GroupedHashAggregateStream {
&& self.update_memory_reservation().is_err()
{
assert_ne!(self.mode, AggregateMode::Partial);
+ // TODO: support spilling when blocked group optimization is on
Review Comment:
Yes, I will file it after this merged.
Actually there are also some other problems in spilling, and I will fix it together when working on supporting blocked management in spilling path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Impl intermeidate result blocked approach sketch [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2812650817
Has finished development(and test) of all needed common structs!
Rest three things for this one:
- Support blocked related logic in `GroupedHashAggregateStream`(we can copy it from #11943 )
- Logic about deciding when we should enable this optimization
- Example blocked version for `GroupAccumulator` and `GroupValues`
- Fuzzy tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2890634742
> I wonder what happens if we make it more like at least 1 million or 1MiB so the effect on cache-friendliness is smaller?
> We could optimize a growing strategy for the first allocated Vec if memory usage / overhead of first block is a concern.
I have tried to larger the block size(8 * batch, 16 * batch...), but it seems make slight difference to the performance.
So after experiement, I think `single vector + resizing` is efficient enough actually...
- It is more efficient for random access
- Resizing will only happen a few times, so it is acceptable actually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2887720153
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark clickbench_extended.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1952.39ms β 2165.55ms β 1.11x slower β
β QQuery 1 β 725.35ms β 776.75ms β 1.07x slower β
β QQuery 2 β 1474.77ms β 1525.72ms β no change β
β QQuery 3 β 720.28ms β 733.30ms β no change β
β QQuery 4 β 1491.93ms β 1549.46ms β no change β
β QQuery 5 β 15503.37ms β 15748.81ms β no change β
β QQuery 6 β 2118.87ms β 2106.19ms β no change β
β QQuery 7 β 2215.33ms β 2189.80ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 26202.29ms β
β Total Time (intermeidate-result-blocked-approach) β 26795.58ms β
β Average Time (HEAD) β 3275.29ms β
β Average Time (intermeidate-result-blocked-approach) β 3349.45ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 6 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark clickbench_partitioned.json
--------------------
ββββββββββββββββ³βββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 2.42ms β 2.74ms β 1.13x slower β
β QQuery 1 β 36.72ms β 39.03ms β 1.06x slower β
β QQuery 2 β 91.22ms β 95.52ms β no change β
β QQuery 3 β 101.22ms β 101.34ms β no change β
β QQuery 4 β 638.98ms β 639.74ms β no change β
β QQuery 5 β 906.48ms β 875.22ms β no change β
β QQuery 6 β 2.31ms β 2.46ms β 1.06x slower β
β QQuery 7 β 43.69ms β 43.48ms β no change β
β QQuery 8 β 960.12ms β 947.47ms β no change β
β QQuery 9 β 1290.26ms β 1263.99ms β no change β
β QQuery 10 β 273.55ms β 273.89ms β no change β
β QQuery 11 β 306.30ms β 319.30ms β no change β
β QQuery 12 β 969.05ms β 954.35ms β no change β
β QQuery 13 β 1311.18ms β 1265.49ms β no change β
β QQuery 14 β 891.85ms β 867.59ms β no change β
β QQuery 15 β 860.97ms β 883.83ms β no change β
β QQuery 16 β 1764.37ms β 1767.56ms β no change β
β QQuery 17 β 1653.48ms β 1634.83ms β no change β
β QQuery 18 β 3151.14ms β 3177.39ms β no change β
β QQuery 19 β 86.57ms β 83.49ms β no change β
β QQuery 20 β 1164.58ms β 1161.57ms β no change β
β QQuery 21 β 1380.48ms β 1337.39ms β no change β
β QQuery 22 β 2295.06ms β 2227.17ms β no change β
β QQuery 23 β 8437.28ms β 8619.52ms β no change β
β QQuery 24 β 487.41ms β 475.78ms β no change β
β QQuery 25 β 403.06ms β 401.45ms β no change β
β QQuery 26 β 537.58ms β 540.40ms β no change β
β QQuery 27 β 1608.99ms β 1597.83ms β no change β
β QQuery 28 β 12616.94ms β 12274.70ms β no change β
β QQuery 29 β 538.94ms β 547.25ms β no change β
β QQuery 30 β 849.96ms β 846.92ms β no change β
β QQuery 31 β 879.77ms β 888.86ms β no change β
β QQuery 32 β 2719.39ms β 2737.35ms β no change β
β QQuery 33 β 3459.61ms β 3454.83ms β no change β
β QQuery 34 β 3471.82ms β 3530.09ms β no change β
β QQuery 35 β 1327.72ms β 1326.17ms β no change β
β QQuery 36 β 128.38ms β 128.36ms β no change β
β QQuery 37 β 59.04ms β 59.49ms β no change β
β QQuery 38 β 128.01ms β 125.30ms β no change β
β QQuery 39 β 211.26ms β 201.66ms β no change β
β QQuery 40 β 49.25ms β 49.61ms β no change β
β QQuery 41 β 48.96ms β 47.59ms β no change β
β QQuery 42 β 40.31ms β 40.19ms β no change β
ββββββββββββββββ΄βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 58185.65ms β
β Total Time (intermeidate-result-blocked-approach) β 57858.18ms β
β Average Time (HEAD) β 1353.15ms β
β Average Time (intermeidate-result-blocked-approach) β 1345.54ms β
β Queries Faster β 0 β
β Queries Slower β 3 β
β Queries with No Change β 40 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
ββββββββββββββββ³βββββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 124.65ms β 125.09ms β no change β
β QQuery 2 β 23.82ms β 23.97ms β no change β
β QQuery 3 β 36.41ms β 36.11ms β no change β
β QQuery 4 β 21.12ms β 20.25ms β no change β
β QQuery 5 β 57.07ms β 55.34ms β no change β
β QQuery 6 β 12.75ms β 12.51ms β no change β
β QQuery 7 β 109.32ms β 103.95ms β no change β
β QQuery 8 β 26.94ms β 29.50ms β 1.10x slower β
β QQuery 9 β 63.71ms β 65.97ms β no change β
β QQuery 10 β 59.22ms β 58.71ms β no change β
β QQuery 11 β 13.22ms β 12.89ms β no change β
β QQuery 12 β 43.95ms β 45.96ms β no change β
β QQuery 13 β 30.27ms β 31.22ms β no change β
β QQuery 14 β 10.31ms β 10.51ms β no change β
β QQuery 15 β 25.33ms β 26.91ms β 1.06x slower β
β QQuery 16 β 24.44ms β 22.87ms β +1.07x faster β
β QQuery 17 β 102.19ms β 104.36ms β no change β
β QQuery 18 β 250.78ms β 254.73ms β no change β
β QQuery 19 β 26.81ms β 28.56ms β 1.07x slower β
β QQuery 20 β 41.12ms β 42.04ms β no change β
β QQuery 21 β 170.36ms β 173.00ms β no change β
β QQuery 22 β 17.60ms β 17.06ms β no change β
ββββββββββββββββ΄βββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1291.37ms β
β Total Time (intermeidate-result-blocked-approach) β 1301.51ms β
β Average Time (HEAD) β 58.70ms β
β Average Time (intermeidate-result-blocked-approach) β 59.16ms β
β Queries Faster β 1 β
β Queries Slower β 3 β
β Queries with No Change β 18 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2888827007
I make a dedicated benchmark for `accumulate`.
As in the total flamegraph, actually no improvement for `blocked accumulate`, although resizing cost is reduced (the query performance increased is my local, I guess it is due to `removing for expansive array slice`) .
I guess the reasons may be:
- We perform few large memory allocation in `flat`, but will much more small memory allocations in `blocked`
- The memory no continuous anymore, I think it may be not so friendly for cpu(like cache prefetch?)
```
Flat accumulate time: [135.16 ms 135.20 ms 135.25 ms]
change: [-1.9077% -1.8671% -1.8244%] (p = 0.00 < 0.05)
Performance has improved.
Blocked accumulate time: [139.89 ms 139.91 ms 139.92 ms] (even a bit slower)
change: [+0.3130% +0.3246% +0.3359%] (p = 0.00 < 0.05)
Change within noise threshold.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Implement intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-2887592362
Thank you so much for this PR btw @Rachelint -- it is really really nice. I can't wait to see how the performance looks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-3154589330
It is unfortunate we never figured out how to get this over the line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4334382489
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4334353557)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4334353557-1862-xv64r 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (a84d71cf9cc8161aca8398c329749d00137965d3) to bbf67d9 (merge-base) [diff](https://github.com/apache/datafusion/compare/bbf67d999c0dcbaf47956233a5d8e78458c13ffa..a84d71cf9cc8161aca8398c329749d00137965d3) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4334382892
Benchmark for [this request](https://github.com/apache/datafusion/pull/15591#issuecomment-4334353557) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
| ^^^^^^^ pattern `EmitTo::NextBlock` not covered
|
note: `EmitTo` defined here
--> datafusion/expr-common/src/groups_accumulator.rs:25:1
|
25 | pub enum EmitTo {
| ^^^^^^^^^^^^^^^
...
37 | NextBlock,
| --------- not covered
= note: the matched value is of type `EmitTo`
help: ensure that all possible cases are being handled by adding a match arm with a wildcard pattern or an explicit pattern as shown
|
108 ~ EmitTo::First(n) => n,
109 ~ EmitTo::NextBlock => todo!(),
|
For more information about this error, try `rustc --explain E0004`.
error: could not compile `datafusion-functions-aggregate-common` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4723576213
Marking as a draft as I don't think this one is ready to merge quite yet and I am trying to clean up the review / merge queue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4756783946
> > Yes, and actually I think it make few difference to performance after experiment before (some steps are improved like removing slice of record batch, removing Vec resizing, and some steps are regressed like we need to perform 2 index op, and finally near to no difference will be made), and just a better memory management approach.
>
> I disagree, since the memory management is directly tied to performance via the spilling mechanism when running with memory limits configured. See [#22526 (comment)](https://github.com/apache/datafusion/issues/22526#issuecomment-4568611822) The memory overaccounting issues caused by the current design of hash aggregation have a real performance impact in the downstream operators which are either:
>
> * forced to spill prematurely
> * outright fail because they don't have enough memory, see [Accurately reserve memory in the build side of hash joinsΒ #22861](https://github.com/apache/datafusion/issues/22861)
>
> So I believe the new "blocked" approach will have significant performance improvements in production-like workloads.
I agree we could proceed first without worrying too much about the benchmark numbers.
This is like a tradeoff between micro-optimizations and algorithmic improvements to memory efficiency.
I think completely giving up 10%-ish performance for architectural win is already a good idea. But realistically, I also believe it should be possible to avoid the regressions entirely with some low-level optimizations, but we'd better discuss those opportunities later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3347787317
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
Review Comment:
> The problem isn't that we're not transitioning to ProducingBlocks early (this can be done via the `skip partial agg feature`), the problem is that:
>
> * emitting a large RecordBatch releases the memory reservation occupied by this batch, but the memory is not freed, it is used in the ProducingOutput state while producing small output batches
> * ProducingBlocks slices the large RecordBatch into smaller ones, but calling `get_array_memory_size` on the small record batch returns the memory of the entire original large RecordBatch, so spilling is broken for the parent operator (e.g. repartitioning)
Yes... what's worse is when dump the slicing sub-batch, it actually dump all the whole bytes of the parent batch it slicing from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4785071752
@adriangb @alamb Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on code in PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#discussion_r3169039554
##########
datafusion/physical-plan/src/aggregates/row_hash.rs:
##########
Review Comment:
should there also be a transition to `ProducingBlocks` here instead of `ProducingOutput` if enable_blocked_groups is set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4193494034
have fixed all correctness problems, can be ready again today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4195597389
> This branch still have conflicts.
Yes, mainly solved the correctness in `row_hash.rs` and `accumulate.rs`.
Will clean up codes and solve other small problems today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276095798
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4276055952)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and intermeidate-result-blocked-approach
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β intermeidate-result-blocked-approach β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 6.67 / 7.14 Β±0.77 / 8.68 ms β 6.48 / 6.97 Β±0.82 / 8.60 ms β no change β
β QQuery 2 β 146.44 / 147.39 Β±0.58 / 148.19 ms β 151.41 / 152.34 Β±0.68 / 153.33 ms β no change β
β QQuery 3 β 115.14 / 115.64 Β±0.33 / 116.09 ms β 112.82 / 113.52 Β±0.65 / 114.74 ms β no change β
β QQuery 4 β 1301.26 / 1338.21 Β±27.48 / 1380.06 ms β 1305.55 / 1350.93 Β±22.91 / 1368.05 ms β no change β
β QQuery 5 β 171.37 / 172.92 Β±0.86 / 173.76 ms β 171.80 / 173.29 Β±1.63 / 175.75 ms β no change β
β QQuery 6 β 844.90 / 881.64 Β±18.39 / 892.22 ms β 881.15 / 897.61 Β±15.87 / 919.14 ms β no change β
β QQuery 7 β 343.11 / 348.56 Β±3.57 / 354.06 ms β 342.42 / 346.58 Β±3.76 / 353.09 ms β no change β
β QQuery 8 β 113.91 / 115.45 Β±1.17 / 117.35 ms β 113.50 / 115.79 Β±1.53 / 117.46 ms β no change β
β QQuery 9 β 100.62 / 102.68 Β±2.86 / 108.31 ms β 100.83 / 105.93 Β±5.02 / 114.26 ms β no change β
β QQuery 10 β 106.74 / 107.71 Β±0.65 / 108.54 ms β 106.48 / 108.28 Β±1.66 / 111.00 ms β no change β
β QQuery 11 β 936.53 / 949.11 Β±14.84 / 977.12 ms β 937.76 / 957.06 Β±13.41 / 975.39 ms β no change β
β QQuery 12 β 44.75 / 46.81 Β±1.48 / 48.22 ms β 45.62 / 47.73 Β±1.17 / 48.88 ms β no change β
β QQuery 13 β 400.09 / 402.23 Β±1.41 / 404.05 ms β 399.55 / 402.86 Β±1.80 / 404.47 ms β no change β
β QQuery 14 β 1009.96 / 1018.33 Β±5.75 / 1025.12 ms β 1006.86 / 1015.89 Β±6.07 / 1022.71 ms β no change β
β QQuery 15 β 16.06 / 17.00 Β±0.94 / 18.47 ms β 14.87 / 16.13 Β±1.19 / 18.16 ms β +1.05x faster β
β QQuery 16 β 7.23 / 7.64 Β±0.54 / 8.67 ms β 7.31 / 8.17 Β±1.06 / 10.07 ms β 1.07x slower β
β QQuery 17 β 229.90 / 231.55 Β±1.01 / 232.85 ms β 228.68 / 230.47 Β±1.24 / 232.43 ms β no change β
β QQuery 18 β 126.72 / 129.25 Β±1.46 / 131.29 ms β 126.81 / 129.24 Β±1.34 / 130.69 ms β no change β
β QQuery 19 β 153.66 / 156.00 Β±1.18 / 156.76 ms β 154.04 / 155.11 Β±1.16 / 156.94 ms β no change β
β QQuery 20 β 14.04 / 14.86 Β±0.80 / 16.28 ms β 13.52 / 14.69 Β±0.67 / 15.44 ms β no change β
β QQuery 21 β 19.50 / 20.43 Β±0.54 / 21.11 ms β 19.78 / 20.56 Β±0.92 / 22.26 ms β no change β
β QQuery 22 β 479.67 / 484.53 Β±4.52 / 492.40 ms β 485.00 / 486.94 Β±1.74 / 489.06 ms β no change β
β QQuery 23 β 868.49 / 887.15 Β±11.45 / 904.16 ms β 883.66 / 890.52 Β±5.68 / 898.08 ms β no change β
β QQuery 24 β 384.94 / 388.70 Β±4.05 / 394.02 ms β 383.43 / 384.36 Β±1.09 / 386.44 ms β no change β
β QQuery 25 β 341.25 / 342.70 Β±1.08 / 344.14 ms β 342.09 / 344.28 Β±2.16 / 347.73 ms β no change β
β QQuery 26 β 82.14 / 83.90 Β±1.97 / 86.59 ms β 81.52 / 84.07 Β±1.78 / 86.84 ms β no change β
β QQuery 27 β 6.89 / 7.35 Β±0.60 / 8.54 ms β 6.73 / 6.94 Β±0.14 / 7.09 ms β +1.06x faster β
β QQuery 28 β 149.21 / 150.63 Β±0.94 / 151.66 ms β 147.99 / 149.99 Β±1.97 / 152.45 ms β no change β
β QQuery 29 β 281.32 / 284.00 Β±1.50 / 285.51 ms β 282.70 / 284.25 Β±1.05 / 285.47 ms β no change β
β QQuery 30 β 43.57 / 44.71 Β±1.28 / 46.99 ms β 43.08 / 44.45 Β±0.98 / 45.85 ms β no change β
β QQuery 31 β 169.47 / 170.94 Β±1.59 / 173.81 ms β 170.61 / 172.26 Β±1.85 / 175.86 ms β no change β
β QQuery 32 β 58.44 / 59.19 Β±0.54 / 60.01 ms β 57.36 / 58.06 Β±0.69 / 59.33 ms β no change β
β QQuery 33 β 139.38 / 140.77 Β±0.73 / 141.50 ms β 139.73 / 141.69 Β±1.85 / 144.61 ms β no change β
β QQuery 34 β 7.14 / 7.37 Β±0.20 / 7.75 ms β 7.02 / 7.30 Β±0.21 / 7.59 ms β no change β
β QQuery 35 β 107.97 / 110.49 Β±2.50 / 114.08 ms β 108.36 / 108.91 Β±0.42 / 109.31 ms β no change β
β QQuery 36 β 6.74 / 6.94 Β±0.27 / 7.46 ms β 6.45 / 6.63 Β±0.21 / 6.93 ms β no change β
β QQuery 37 β 8.36 / 8.60 Β±0.16 / 8.83 ms β 8.17 / 8.78 Β±0.34 / 9.15 ms β no change β
β QQuery 38 β 83.50 / 87.85 Β±3.47 / 92.86 ms β 86.33 / 88.58 Β±2.84 / 93.98 ms β no change β
β QQuery 39 β 126.59 / 129.18 Β±2.03 / 132.62 ms β 126.36 / 129.87 Β±2.21 / 132.39 ms β no change β
β QQuery 40 β 108.35 / 116.04 Β±6.52 / 127.16 ms β 111.51 / 115.70 Β±7.02 / 129.68 ms β no change β
β QQuery 41 β 14.53 / 15.64 Β±0.94 / 17.06 ms β 14.77 / 15.65 Β±0.49 / 16.18 ms β no change β
β QQuery 42 β 108.46 / 109.85 Β±1.09 / 111.68 ms β 107.66 / 109.39 Β±1.64 / 112.02 ms β no change β
β QQuery 43 β 6.16 / 6.49 Β±0.29 / 6.97 ms β 5.99 / 6.10 Β±0.14 / 6.36 ms β +1.06x faster β
β QQuery 44 β 11.89 / 12.50 Β±0.62 / 13.69 ms β 11.77 / 11.95 Β±0.14 / 12.20 ms β no change β
β QQuery 45 β 51.35 / 52.24 Β±0.91 / 53.76 ms β 49.88 / 51.49 Β±1.06 / 52.85 ms β no change β
β QQuery 46 β 8.48 / 8.88 Β±0.29 / 9.37 ms β 8.28 / 8.92 Β±0.65 / 10.16 ms β no change β
β QQuery 47 β 712.68 / 718.94 Β±3.92 / 724.77 ms β 713.73 / 718.72 Β±4.17 / 725.66 ms β no change β
β QQuery 48 β 290.28 / 297.18 Β±3.57 / 300.42 ms β 288.73 / 291.51 Β±2.31 / 294.67 ms β no change β
β QQuery 49 β 252.77 / 255.07 Β±1.70 / 256.95 ms β 251.01 / 253.45 Β±2.47 / 257.80 ms β no change β
β QQuery 50 β 222.75 / 226.54 Β±2.27 / 229.68 ms β 216.16 / 228.32 Β±6.98 / 235.26 ms β no change β
β QQuery 51 β 181.82 / 184.17 Β±1.98 / 187.66 ms β 181.52 / 184.80 Β±2.77 / 188.23 ms β no change β
β QQuery 52 β 107.38 / 109.50 Β±1.53 / 111.09 ms β 106.89 / 107.85 Β±0.94 / 109.44 ms β no change β
β QQuery 53 β 102.48 / 103.77 Β±1.11 / 105.62 ms β 102.64 / 103.34 Β±0.85 / 104.70 ms β no change β
β QQuery 54 β 145.37 / 148.00 Β±1.80 / 150.86 ms β 144.10 / 146.96 Β±1.47 / 148.02 ms β no change β
β QQuery 55 β 106.77 / 108.44 Β±1.31 / 110.37 ms β 107.19 / 109.02 Β±1.47 / 110.84 ms β no change β
β QQuery 56 β 140.10 / 141.87 Β±1.46 / 144.16 ms β 139.39 / 141.20 Β±1.50 / 143.45 ms β no change β
β QQuery 57 β 173.82 / 177.22 Β±2.90 / 182.56 ms β 176.67 / 177.89 Β±1.21 / 180.12 ms β no change β
β QQuery 58 β 286.05 / 300.64 Β±9.83 / 309.56 ms β 295.92 / 310.56 Β±11.40 / 329.94 ms β no change β
β QQuery 59 β 199.68 / 203.19 Β±3.85 / 210.32 ms β 198.59 / 200.67 Β±1.17 / 202.06 ms β no change β
β QQuery 60 β 142.28 / 144.95 Β±1.39 / 145.98 ms β 142.32 / 143.02 Β±0.66 / 144.07 ms β no change β
β QQuery 61 β 13.18 / 13.49 Β±0.25 / 13.93 ms β 13.18 / 13.39 Β±0.19 / 13.71 ms β no change β
β QQuery 62 β 898.00 / 937.56 Β±35.95 / 999.38 ms β 915.41 / 953.77 Β±29.58 / 987.88 ms β no change β
β QQuery 63 β 103.47 / 104.78 Β±1.78 / 108.24 ms β 103.20 / 106.02 Β±1.96 / 108.63 ms β no change β
β QQuery 64 β 692.14 / 696.68 Β±2.42 / 698.73 ms β 688.84 / 693.53 Β±3.27 / 697.16 ms β no change β
β QQuery 65 β 253.58 / 256.39 Β±2.18 / 260.31 ms β 256.34 / 261.76 Β±3.65 / 266.97 ms β no change β
β QQuery 66 β 238.08 / 257.93 Β±18.10 / 288.67 ms β 236.94 / 262.92 Β±16.92 / 282.62 ms β no change β
β QQuery 67 β 307.13 / 318.31 Β±8.11 / 329.94 ms β 315.94 / 324.34 Β±7.19 / 335.15 ms β no change β
β QQuery 68 β 8.72 / 11.22 Β±1.26 / 12.03 ms β 9.94 / 11.07 Β±0.59 / 11.60 ms β no change β
β QQuery 69 β 103.13 / 104.09 Β±0.94 / 105.51 ms β 103.26 / 106.63 Β±1.98 / 108.49 ms β no change β
β QQuery 70 β 348.91 / 356.46 Β±6.12 / 366.90 ms β 343.69 / 359.61 Β±9.72 / 370.34 ms β no change β
β QQuery 71 β 135.50 / 136.78 Β±1.46 / 138.82 ms β 136.68 / 138.97 Β±1.74 / 141.03 ms β no change β
β QQuery 72 β 618.62 / 632.52 Β±7.75 / 641.72 ms β 630.34 / 641.30 Β±8.03 / 654.12 ms β no change β
β QQuery 73 β 6.92 / 8.72 Β±1.80 / 11.92 ms β 7.67 / 8.98 Β±1.41 / 11.59 ms β no change β
β QQuery 74 β 594.69 / 601.21 Β±5.12 / 609.65 ms β 586.74 / 588.34 Β±1.58 / 590.92 ms β no change β
β QQuery 75 β 276.62 / 278.88 Β±1.43 / 280.12 ms β 274.12 / 276.57 Β±1.37 / 278.11 ms β no change β
β QQuery 76 β 131.76 / 134.84 Β±2.53 / 139.13 ms β 132.35 / 133.92 Β±1.00 / 135.46 ms β no change β
β QQuery 77 β 187.58 / 188.49 Β±0.76 / 189.51 ms β 186.21 / 189.24 Β±2.21 / 192.43 ms β no change β
β QQuery 78 β 341.16 / 346.49 Β±3.49 / 350.96 ms β 342.08 / 344.66 Β±2.78 / 349.73 ms β no change β
β QQuery 79 β 234.06 / 236.03 Β±1.93 / 239.61 ms β 234.01 / 236.62 Β±1.98 / 239.35 ms β no change β
β QQuery 80 β 323.56 / 325.71 Β±2.60 / 330.80 ms β 320.23 / 323.32 Β±1.65 / 324.77 ms β no change β
β QQuery 81 β 26.37 / 27.77 Β±1.46 / 30.18 ms β 27.20 / 28.46 Β±1.00 / 29.56 ms β no change β
β QQuery 82 β 201.17 / 202.55 Β±2.08 / 206.60 ms β 199.66 / 203.08 Β±1.73 / 204.34 ms β no change β
β QQuery 83 β 39.47 / 40.39 Β±0.49 / 40.86 ms β 38.87 / 39.36 Β±0.40 / 39.99 ms β no change β
β QQuery 84 β 49.50 / 50.05 Β±0.71 / 51.33 ms β 48.28 / 48.91 Β±0.55 / 49.70 ms β no change β
β QQuery 85 β 150.08 / 151.67 Β±1.03 / 152.94 ms β 148.78 / 151.14 Β±1.98 / 154.62 ms β no change β
β QQuery 86 β 39.36 / 40.28 Β±0.90 / 41.81 ms β 39.89 / 40.54 Β±0.44 / 41.19 ms β no change β
β QQuery 87 β 83.73 / 89.09 Β±3.60 / 94.17 ms β 83.54 / 87.99 Β±4.10 / 94.10 ms β no change β
β QQuery 88 β 99.97 / 100.74 Β±0.72 / 101.92 ms β 99.43 / 101.29 Β±2.02 / 105.02 ms β no change β
β QQuery 89 β 117.44 / 119.76 Β±1.62 / 122.02 ms β 117.64 / 118.83 Β±1.23 / 120.46 ms β no change β
β QQuery 90 β 23.73 / 23.95 Β±0.24 / 24.34 ms β 23.88 / 24.39 Β±0.30 / 24.71 ms β no change β
β QQuery 91 β 62.91 / 65.61 Β±1.45 / 66.88 ms β 64.26 / 65.52 Β±0.98 / 67.21 ms β no change β
β QQuery 92 β 58.30 / 59.07 Β±0.62 / 59.85 ms β 57.96 / 58.74 Β±0.47 / 59.30 ms β no change β
β QQuery 93 β 187.94 / 189.59 Β±1.29 / 191.48 ms β 185.25 / 187.98 Β±1.58 / 189.39 ms β no change β
β QQuery 94 β 61.76 / 62.46 Β±0.50 / 63.22 ms β 61.47 / 61.82 Β±0.45 / 62.66 ms β no change β
β QQuery 95 β 128.50 / 130.66 Β±1.13 / 131.66 ms β 129.44 / 130.25 Β±1.19 / 132.58 ms β no change β
β QQuery 96 β 73.58 / 74.76 Β±1.14 / 76.44 ms β 73.84 / 75.87 Β±1.92 / 79.13 ms β no change β
β QQuery 97 β 124.46 / 128.28 Β±1.98 / 130.11 ms β 127.48 / 128.92 Β±0.95 / 130.38 ms β no change β
β QQuery 98 β 152.91 / 156.53 Β±2.05 / 159.10 ms β 150.42 / 155.29 Β±2.64 / 157.80 ms β no change β
β QQuery 99 β 10738.36 / 10765.31 Β±16.29 / 10785.18 ms β 10784.87 / 10812.14 Β±30.94 / 10871.85 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 31591.78ms β
β Total Time (intermeidate-result-blocked-approach) β 31699.03ms β
β Average Time (HEAD) β 319.11ms β
β Average Time (intermeidate-result-blocked-approach) β 320.19ms β
β Queries Faster β 3 β
β Queries Slower β 1 β
β Queries with No Change β 95 β
β Queries with Failure β 0 β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 158.3s |
| Peak memory | 5.6 GiB |
| Avg memory | 4.6 GiB |
| CPU user | 261.2s |
| CPU sys | 17.8s |
| Peak spill | 0 B |
**tpcds β branch**
| Metric | Value |
|--------|-------|
| Wall time | 158.8s |
| Peak memory | 5.1 GiB |
| Avg memory | 4.5 GiB |
| CPU user | 261.7s |
| CPU sys | 18.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4276059575
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/15591#issuecomment-4276055952)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4276055952-1549-r78jh 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing intermeidate-result-blocked-approach (a68716b1b99123fef41259899c0c67adda870dbb) to dc973cc (merge-base) [diff](https://github.com/apache/datafusion/compare/dc973cc9e67e3519ec4bc5bd15962e2692debb3e..a68716b1b99123fef41259899c0c67adda870dbb) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-4245046381
I was checking it out as well, and playing around with it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5477734136
Makes sense
> According to the analysis of the coding agent, Emit::Last can be easily implemented for most GroupValues and GroupsAccumulator (compare with the totally blocked approach like this pr)
That might also show us a way towards incremental / blocked management -- we could implement `EmitTo::Last` for all accumulators, and then potentially optimize the implementations one by one to use blocked allocations internally
The trick would be to make sure `EmitTo::Last` lines up with however the internal blocks are laid out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5477626355
> > So perhaps a simpler alternative is to add EmitTo::Last.
>
> This is an interesting idea -- is your idea to keep batches managed by blocks internally, but not change the group_index API? And then EmitTo::Last ? Or were you imagining coping the EmitTo::Last rows to the output from the inner Vec allocation
I want to try the latter, because:
- I found the simple copying is not expansive as we think when I experiment other ideas.
- According to the analysis of the coding agent, `Emit::Last` can be easily implemented for most `GroupValues` and `GroupsAccumulator` (compare with the totally blocked approach like this pr)
- Not like `Emit::First`, `Emit::Last` will be effective for most `GroupValues` and `GroupsAccumulator`(truncate + shrink)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5557732347
> Why do you need `EmitTo::Last` for this? Can't you simply keep producing the large RecordBatch as it's being done now (via EmitTo::All or EmitTo::First(n) in case of partial aggregation early emit, where n is very close to the total number of groups) and simply create separate allocations for the individual Record Batches instead of slicing them?
Here's a diagram β hope it helps!
```
Memory held by the aggregation while it drains N groups (batch_size = B)
today: EmitTo::All + slice copy-out per batch EmitTo::Last(n)
βββββββββββββββββββββββββββ ββββββββββββββββββββββ βββββββββββββββββββββββ
βββββββββββββββββββββββββ βββββββββββββββββββββββββ βββββββββββββββββββββββββ
β one giant batch β β one giant batch β β Vec / blocks β
β [ b0 | b1 | b2 | b3 ] β β [ b0 | b1 | b2 | b3 ] β β [ b0 | b1 | b2 | b3 ] β
βββββββββββββββββββββββββ βββββββββββββββββββββββββ βββββββββββββββββββββββββ
β slices share the buffer β copy b0 β own alloc truncate / pop b3 βββΆ out
βΌ βΌ ββββββββββββββββββββ
b0 out (giant still live) b0 out (giant still live) β [ b0 | b1 | b2 ] β b3 freed
b1 out (giant still live) b1 out (giant still live) ββββββββββββββββββββ
b2 out (giant still live) b2 out (giant still live) [ b0 | b1 ] b2 freed
b3 out β giant freed b3 out β giant freed [ b0 ] b1 freed
[] b0 freed
held β² held β² held β²
βββββββββββββββββ βββ βββββββββββββββββ βββ βββ
βββββββββββββββββ βββ βββββββββββββββββ βββ βββββ
βββββββββββββββββ βββββββββββββββββ βββββββ
βββββββββββββββββ time βΆ βββββββββββββββββ time βΆ βββββββββββββββββ time βΆ
all of it, until the end same, +transient copies shrinks one batch per poll
First(n): shift the rest down + renumber every remaining index βββΆ O(remaining) per call
Last(n): drop the tail βββΆ nothing moves, nothing renumbers
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "ariel-miculas (via GitHub)" <gi...@apache.org>.
ariel-miculas commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5515893865
Why do you need `EmitTo::Last` for this? Can't you simply keep producing the large RecordBatch as it's being done now (via EmitTo::All or EmitTo::First(n) in case of partial aggregation early emit, where n is very close to the total number of groups) and simply create separate allocations for the individual Record Batches instead of slicing them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Intermediate result blocked approach to aggregation memory management [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #15591:
URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5355870195
Another complexity is supporting `ListGroupValueBuilder`, try to add support for blocking with it(ignore non blocking support and just see how complex even blocked support is)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org