You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "2010YOUY01 (via GitHub)" <gi...@apache.org> on 2026/07/03 14:18:49 UTC

[PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

2010YOUY01 opened a new pull request, #23309:
URL: https://github.com/apache/datafusion/pull/23309

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   - Closes #.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   Part of https://github.com/apache/datafusion/issues/22710
   
   Addressing @alamb ’s comment from https://github.com/apache/datafusion/pull/23181#pullrequestreview-4602624169
   
   The motivation behind the EPIC was that the existing aggregation code had too many internal flags and was reused by many different execution paths, making it hard to maintain. The previous refactor made two main changes:
   
   1. Split different aggregation modes, such as partial and final, into separate streams.
   2. Split the hash tables used by different aggregation modes into separate implementations.
   
   Now that the refactor is almost done, with only single aggregation mode and spilling behavior left, I think decision 1 was still solid, while **decision 2 was an overcorrection**.
   
   A module usually becomes hard to maintain when it has many tightly coupled internal flags. In the old `GroupsHashAggregateStream` case, there were ~10 such flags. But a small number of flags, I think <= 3, is still clean and manageable.
   
   This PR unifies the aggregate hash tables used by different modes into a single implementation. The behavioral differences are controlled by one internal `mode` flag. See the comments in `common.rs: struct AggregateHashTable`, for details.
   
   Since this refactor is close to completion, I can confirm the internal complexity of `AggregateHashTable` is not likely to bloat in the long term, so unifying these implementations seems like the better design.
   
   
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   Replace separated aggregate hash table implementation with a unified one
   
   ## Are these changes tested?
   
   Existing tests
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   4. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#issuecomment-4879367731

   πŸ€– Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23309#issuecomment-4879357464)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4879357464-843-5qvcm 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing split-aggr-simplify-ht (7f68c238d8de5a3a8eb461504cf25c8b636c2940) to e4aa41d (merge-base) [diff](https://github.com/apache/datafusion/compare/e4aa41d7c68f2fbbd1f8f9de93a7029ceb3e265c..7f68c238d8de5a3a8eb461504cf25c8b636c2940) using: tpch
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3522628660


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -85,29 +114,63 @@ pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
 
     /// Lifecycle-specific state: building stage / outputting stage.
     pub(super) state: AggregateHashTableState,
-
-    pub(super) _mode: PhantomData<AggrMode>,
 }
 
 /// Methods shared by all aggregate hash table modes.
-impl<AggrMode> AggregateHashTable<AggrMode> {
-    pub(super) fn new_with_filters(
+impl AggregateHashTable {
+    pub(in crate::aggregates) fn new(
         agg: &AggregateExec,
         partition: usize,
         output_schema: SchemaRef,
         batch_size: usize,
-        filters: Vec<Option<Arc<dyn PhysicalExpr>>>,
     ) -> Result<Self> {
         if batch_size == 0 {
             return internal_err!("AggregateHashTable requires config batch_size >= 1");
         }
 
+        // Infer the internal `AggregateTableMode` based on `AggregateExec`'s mode
+        //
+        // TODO(simplification): `AggregateMode` seems bloated for aggregate hash
+        // table semantics. Consider remove `AggregateMode` and only use `AggregateTableMode`
+        // after the refactor has finished.
+        //
+        // Issue: <https://github.com/apache/datafusion/pull/22729>

Review Comment:
   Yes, that's a wrong link. Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 closed pull request #23309: refactor(hash-aggr): Simplify aggregate hash table
URL: https://github.com/apache/datafusion/pull/23309


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3522629240


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -214,6 +321,239 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
         state.batch_group_indices = Vec::new();
         self.state = AggregateHashTableState::Outputting(state);
     }
+
+    /// Aggregates one input batch according to this table's input semantics.
+    ///
+    /// `Partial` and `Single` update accumulator state from raw input rows.
+    /// `Final` and `PartialReduce` merge accumulator state emitted by an
+    /// earlier aggregate stage.
+    pub(in crate::aggregates) fn aggregate_batch(
+        &mut self,
+        batch: &RecordBatch,
+    ) -> Result<()> {
+        let evaluated_batch = self.evaluate_batch(batch)?;
+        let mode = self.mode;
+        let state = self.state.building_mut();
+
+        let timer = self.group_by_metrics.aggregation_time.timer();
+        for group_values in &evaluated_batch.grouping_set_args {
+            state
+                .group_values
+                .intern(group_values, &mut state.batch_group_indices)?;
+            let group_indices = &state.batch_group_indices;
+            let total_num_groups = state.group_values.len();
+
+            for (acc, values) in state
+                .accumulators
+                .iter_mut()
+                .zip(evaluated_batch.accumulator_args.iter())
+            {
+                match mode {

Review Comment:
   I agree, this way seems a little bit easier to read: [6871dc7](https://github.com/apache/datafusion/pull/23309/commits/6871dc74b7b0314d7b8efc9b55582ca597212284)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3522601758


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -60,19 +72,36 @@ pub(in crate::aggregates) struct FinalMarker;
 /// [`GroupsAccumulator`]. Both use columnar storage so aggregation can stay
 /// vectorized.
 ///
-/// # Marker Type
-/// `AggrMode` selects the aggregate semantics.
+/// # Mode
+///
+/// [`AggregateTableMode`] controls how input batches update accumulator state
+/// and how output batches are materialized.
 ///
-/// e.g. `AggregateHashTable::<PartialMarker>::new(...)` creates an aggregate hash table
-/// for the partial hash aggregate stage, the input schema is raw rows and output
-/// schema is intermediate states.
+/// ```text
+/// Example: `AVG(x) GROUP BY k`.
 ///
-/// It is a zero-sized compile-time marker, so each stage keeps its update logic
-/// in a separate impl block, to make the behavior difference explicit.
-pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
+/// In `Partial` mode, the table stores partial state:
+///     k, sum(x), count(x)
+/// The input batch contains raw values:
+///     k, x
+/// The output batch also contains partial state:
+///     k, sum(x), count(x)
+/// ```
+///
+/// So input uses [`GroupsAccumulator::update_batch`], and output uses
+/// [`GroupsAccumulator::state`].
+///
+/// Other modes use different input/output combinations:
+/// - `Final`: merge_batch + evaluate
+/// - `PartialReduce`: merge_batch + state
+/// - `Single`: update_batch + evaluate
+pub(in crate::aggregates) struct AggregateHashTable {

Review Comment:
   ### For Proposed alternative
   
   I have considered this approach: separating aggregate hash tables by mode and extracting shared internal utilities.
   
   My concern is that this would create a [shallow module](https://softengbook.org/articles/deep-modules). To fully understand the implementation, we would still need to reason about the same underlying mode differences, while the external `AggregateHashTable` variants would add complexity with almost no functional benefit.
   
   So I think keeping the mode as an internal flag is simpler overall.
   
   ### Complexity concern
   
   For the concern about growing internal complexity, see the rationale in the issue. I think this should remain manageable in the long term.
   
   ### Partial table
   
   Yes, this is a valid concern. The partial table logic could be moved out of this module entirely to make the structure cleaner. However, I think this can be delayed a bit, and I plan to do it as a follow-up.
   https://github.com/apache/datafusion/issues/23113



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#issuecomment-4880813555

   Thanks for the review @Rachelint and @alamb 
   
   If there are other ideas for making the structure simpler and easier to understand, I’d be happy to discuss and experiment further. I think managing complexity is a hard but important problem, and it’s interesting to figure out methodologies to address it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3522601758


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -60,19 +72,36 @@ pub(in crate::aggregates) struct FinalMarker;
 /// [`GroupsAccumulator`]. Both use columnar storage so aggregation can stay
 /// vectorized.
 ///
-/// # Marker Type
-/// `AggrMode` selects the aggregate semantics.
+/// # Mode
+///
+/// [`AggregateTableMode`] controls how input batches update accumulator state
+/// and how output batches are materialized.
 ///
-/// e.g. `AggregateHashTable::<PartialMarker>::new(...)` creates an aggregate hash table
-/// for the partial hash aggregate stage, the input schema is raw rows and output
-/// schema is intermediate states.
+/// ```text
+/// Example: `AVG(x) GROUP BY k`.
 ///
-/// It is a zero-sized compile-time marker, so each stage keeps its update logic
-/// in a separate impl block, to make the behavior difference explicit.
-pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
+/// In `Partial` mode, the table stores partial state:
+///     k, sum(x), count(x)
+/// The input batch contains raw values:
+///     k, x
+/// The output batch also contains partial state:
+///     k, sum(x), count(x)
+/// ```
+///
+/// So input uses [`GroupsAccumulator::update_batch`], and output uses
+/// [`GroupsAccumulator::state`].
+///
+/// Other modes use different input/output combinations:
+/// - `Final`: merge_batch + evaluate
+/// - `PartialReduce`: merge_batch + state
+/// - `Single`: update_batch + evaluate
+pub(in crate::aggregates) struct AggregateHashTable {

Review Comment:
   ### For Proposed alternative
   
   I have considered this approach: separating aggregate hash tables by mode and extracting shared internal utilities.
   
   My concern is that this would create a [shallow module](https://softengbook.org/articles/deep-modules). To fully understand the implementation, we would still need to reason about the same underlying mode differences, while the external `AggregateHashTable` variants would add complexity with almost no functional contribution.
   
   So I think keeping the mode as an internal flag is simpler overall.
   
   ### Complexity concern
   
   For the concern about growing internal complexity, see the rationale in the issue. I think this should remain manageable in the long term.
   
   ### Partial skip table
   
   Yes, this is a valid concern. The partial skip table logic could be moved out of this module entirely to make the structure cleaner. However, I think this can be delayed a bit, and I plan to do it as a follow-up.
   https://github.com/apache/datafusion/issues/23113



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3522620155


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -31,19 +31,31 @@ use crate::aggregates::group_values::{GroupByMetrics, GroupValues, new_group_val
 use crate::aggregates::order::GroupOrdering;
 use crate::aggregates::row_hash::create_group_accumulator;
 use crate::aggregates::{
-    AggregateExec, PhysicalGroupBy, aggregate_expressions, evaluate_group_by,
+    AggregateExec, AggregateMode, PhysicalGroupBy, aggregate_expressions,
+    evaluate_group_by, group_id_array, max_duplicate_ordinal,
 };
 
-/// Marker for raw rows -> partial state aggregation.
+/// Semantic mode for the aggregate hash table.
+///
+/// See [`AggregateHashTable`] comment's 'Mode' section for details.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub(in crate::aggregates) enum AggregateTableMode {
+    /// Raw rows -> partial aggregate state rows.
+    Partial,
+    /// Partial aggregate state rows -> final aggregate value rows.
+    Final,
+    /// Partial aggregate state rows -> merged partial aggregate state rows.
+    PartialReduce,
+    /// Raw rows -> final aggregate value rows.
+    Single,
+}
+
+/// Marker for ordered raw rows -> partial state aggregation.
 pub(in crate::aggregates) struct PartialMarker;

Review Comment:
   Yes, this can be done similarly.
   
   But I think there is a better fast path to implement for fully ordered case (query `group by a,b`, input order `order by a,b`), this idea can be implemented in a completely separated path without depending on the aggregate hash tables; while the partially ordered case still relies on hash table, and only do early emit optimization.
   
   So I think probably its easier to do after we have further optimized the full order fast path.
   
   Filed https://github.com/apache/datafusion/issues/23318 to track.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3522407373


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -60,19 +72,36 @@ pub(in crate::aggregates) struct FinalMarker;
 /// [`GroupsAccumulator`]. Both use columnar storage so aggregation can stay
 /// vectorized.
 ///
-/// # Marker Type
-/// `AggrMode` selects the aggregate semantics.
+/// # Mode
+///
+/// [`AggregateTableMode`] controls how input batches update accumulator state
+/// and how output batches are materialized.
 ///
-/// e.g. `AggregateHashTable::<PartialMarker>::new(...)` creates an aggregate hash table
-/// for the partial hash aggregate stage, the input schema is raw rows and output
-/// schema is intermediate states.
+/// ```text
+/// Example: `AVG(x) GROUP BY k`.
 ///
-/// It is a zero-sized compile-time marker, so each stage keeps its update logic
-/// in a separate impl block, to make the behavior difference explicit.
-pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
+/// In `Partial` mode, the table stores partial state:
+///     k, sum(x), count(x)
+/// The input batch contains raw values:
+///     k, x
+/// The output batch also contains partial state:
+///     k, sum(x), count(x)
+/// ```
+///
+/// So input uses [`GroupsAccumulator::update_batch`], and output uses
+/// [`GroupsAccumulator::state`].
+///
+/// Other modes use different input/output combinations:
+/// - `Final`: merge_batch + evaluate
+/// - `PartialReduce`: merge_batch + state
+/// - `Single`: update_batch + evaluate
+pub(in crate::aggregates) struct AggregateHashTable {

Review Comment:
   I think maybe we should keep different `AggregateHashTable`s for different usage, and remove duplicated codes based on this?
   Due to :
   - Concerns in `aggregate_batch` and other methods.
   - Some method like `partial_skip_table`, it is logic for partial aggr and seems not common.
   



##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -214,6 +321,239 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
         state.batch_group_indices = Vec::new();
         self.state = AggregateHashTableState::Outputting(state);
     }
+
+    /// Aggregates one input batch according to this table's input semantics.
+    ///
+    /// `Partial` and `Single` update accumulator state from raw input rows.
+    /// `Final` and `PartialReduce` merge accumulator state emitted by an
+    /// earlier aggregate stage.
+    pub(in crate::aggregates) fn aggregate_batch(
+        &mut self,
+        batch: &RecordBatch,
+    ) -> Result<()> {
+        let evaluated_batch = self.evaluate_batch(batch)?;
+        let mode = self.mode;
+        let state = self.state.building_mut();
+
+        let timer = self.group_by_metrics.aggregation_time.timer();
+        for group_values in &evaluated_batch.grouping_set_args {
+            state
+                .group_values
+                .intern(group_values, &mut state.batch_group_indices)?;
+            let group_indices = &state.batch_group_indices;
+            let total_num_groups = state.group_values.len();
+
+            for (acc, values) in state
+                .accumulators
+                .iter_mut()
+                .zip(evaluated_batch.accumulator_args.iter())
+            {
+                match mode {
+                    AggregateTableMode::Partial | AggregateTableMode::Single => {

Review Comment:
   I am concerned codes here seem switching to the way like `row_hash.rs`: all logic in one place, and using `if else` to decide where we go.



##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -85,29 +114,63 @@ pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
 
     /// Lifecycle-specific state: building stage / outputting stage.
     pub(super) state: AggregateHashTableState,
-
-    pub(super) _mode: PhantomData<AggrMode>,
 }
 
 /// Methods shared by all aggregate hash table modes.
-impl<AggrMode> AggregateHashTable<AggrMode> {
-    pub(super) fn new_with_filters(
+impl AggregateHashTable {
+    pub(in crate::aggregates) fn new(
         agg: &AggregateExec,
         partition: usize,
         output_schema: SchemaRef,
         batch_size: usize,
-        filters: Vec<Option<Arc<dyn PhysicalExpr>>>,
     ) -> Result<Self> {
         if batch_size == 0 {
             return internal_err!("AggregateHashTable requires config batch_size >= 1");
         }
 
+        // Infer the internal `AggregateTableMode` based on `AggregateExec`'s mode
+        //
+        // TODO(simplification): `AggregateMode` seems bloated for aggregate hash
+        // table semantics. Consider remove `AggregateMode` and only use `AggregateTableMode`
+        // after the refactor has finished.
+        //
+        // Issue: <https://github.com/apache/datafusion/pull/22729>
+        let mode = match agg.mode {
+            AggregateMode::Partial => AggregateTableMode::Partial,
+            AggregateMode::Final | AggregateMode::FinalPartitioned => {
+                AggregateTableMode::Final
+            }
+            AggregateMode::PartialReduce => AggregateTableMode::PartialReduce,
+            AggregateMode::Single | AggregateMode::SinglePartitioned => {
+                AggregateTableMode::Single
+            }
+        };
+
         let input_schema = agg.input().schema();
+        let aggregate_mode = match mode {
+            AggregateTableMode::Partial => AggregateMode::Partial,
+            AggregateTableMode::Final => AggregateMode::Final,
+            AggregateTableMode::PartialReduce => AggregateMode::PartialReduce,
+            AggregateTableMode::Single => AggregateMode::Single,
+        };
         let aggregate_arguments = aggregate_expressions(
             &agg.aggr_expr,
-            &agg.mode,
+            &aggregate_mode,
             agg.group_by.num_group_exprs(),
         )?;
+
+        // Filters apply only when the table consumes raw input rows. Final and
+        // partial-reduce modes consume partial states, so their filters are not
+        // applicable.
+        let filters = match mode {

Review Comment:
   Same concern like what in `aggregate_batch`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3524618135


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -60,19 +72,36 @@ pub(in crate::aggregates) struct FinalMarker;
 /// [`GroupsAccumulator`]. Both use columnar storage so aggregation can stay
 /// vectorized.
 ///
-/// # Marker Type
-/// `AggrMode` selects the aggregate semantics.
+/// # Mode
+///
+/// [`AggregateTableMode`] controls how input batches update accumulator state
+/// and how output batches are materialized.
 ///
-/// e.g. `AggregateHashTable::<PartialMarker>::new(...)` creates an aggregate hash table
-/// for the partial hash aggregate stage, the input schema is raw rows and output
-/// schema is intermediate states.
+/// ```text
+/// Example: `AVG(x) GROUP BY k`.
 ///
-/// It is a zero-sized compile-time marker, so each stage keeps its update logic
-/// in a separate impl block, to make the behavior difference explicit.
-pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
+/// In `Partial` mode, the table stores partial state:
+///     k, sum(x), count(x)
+/// The input batch contains raw values:
+///     k, x
+/// The output batch also contains partial state:
+///     k, sum(x), count(x)
+/// ```
+///
+/// So input uses [`GroupsAccumulator::update_batch`], and output uses
+/// [`GroupsAccumulator::state`].
+///
+/// Other modes use different input/output combinations:
+/// - `Final`: merge_batch + evaluate
+/// - `PartialReduce`: merge_batch + state
+/// - `Single`: update_batch + evaluate
+pub(in crate::aggregates) struct AggregateHashTable {

Review Comment:
   > One idea could be to leave the structure that is already on main, and refactor the common behavior out as templated functions on the base class
   
   The templated-function also seems good for me.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3522922944


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -214,6 +321,239 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
         state.batch_group_indices = Vec::new();
         self.state = AggregateHashTableState::Outputting(state);
     }
+
+    /// Aggregates one input batch according to this table's input semantics.
+    ///
+    /// `Partial` and `Single` update accumulator state from raw input rows.
+    /// `Final` and `PartialReduce` merge accumulator state emitted by an
+    /// earlier aggregate stage.
+    pub(in crate::aggregates) fn aggregate_batch(
+        &mut self,
+        batch: &RecordBatch,
+    ) -> Result<()> {
+        let evaluated_batch = self.evaluate_batch(batch)?;
+        let mode = self.mode;
+        let state = self.state.building_mut();
+
+        let timer = self.group_by_metrics.aggregation_time.timer();
+        for group_values in &evaluated_batch.grouping_set_args {
+            state
+                .group_values
+                .intern(group_values, &mut state.batch_group_indices)?;
+            let group_indices = &state.batch_group_indices;
+            let total_num_groups = state.group_values.len();
+
+            for (acc, values) in state
+                .accumulators
+                .iter_mut()
+                .zip(evaluated_batch.accumulator_args.iter())
+            {
+                match mode {
+                    AggregateTableMode::Partial | AggregateTableMode::Single => {

Review Comment:
   I wonder if it would help to factor these decisions into other abstractions? Like maybe a method on AggregateTableMode πŸ€” 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#issuecomment-4879368224

   πŸ€– Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23309#issuecomment-4879357464)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4879357464-841-jrdm8 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing split-aggr-simplify-ht (7f68c238d8de5a3a8eb461504cf25c8b636c2940) to e4aa41d (merge-base) [diff](https://github.com/apache/datafusion/compare/e4aa41d7c68f2fbbd1f8f9de93a7029ceb3e265c..7f68c238d8de5a3a8eb461504cf25c8b636c2940) using: clickbench_partitioned
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3521848125


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -85,29 +114,63 @@ pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
 
     /// Lifecycle-specific state: building stage / outputting stage.
     pub(super) state: AggregateHashTableState,
-
-    pub(super) _mode: PhantomData<AggrMode>,
 }
 
 /// Methods shared by all aggregate hash table modes.
-impl<AggrMode> AggregateHashTable<AggrMode> {
-    pub(super) fn new_with_filters(
+impl AggregateHashTable {
+    pub(in crate::aggregates) fn new(
         agg: &AggregateExec,
         partition: usize,
         output_schema: SchemaRef,
         batch_size: usize,
-        filters: Vec<Option<Arc<dyn PhysicalExpr>>>,
     ) -> Result<Self> {
         if batch_size == 0 {
             return internal_err!("AggregateHashTable requires config batch_size >= 1");
         }
 
+        // Infer the internal `AggregateTableMode` based on `AggregateExec`'s mode
+        //
+        // TODO(simplification): `AggregateMode` seems bloated for aggregate hash
+        // table semantics. Consider remove `AggregateMode` and only use `AggregateTableMode`
+        // after the refactor has finished.
+        //
+        // Issue: <https://github.com/apache/datafusion/pull/22729>

Review Comment:
   This seems like a PR (not an issue) -- is it the link you intended?
   
   Note I agree with the fact that AggregateMode seems overly complicated. I vaguely remember trying to remove it once but I can't find the PR now (maybe I never pushed it)
   
   



##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -31,19 +31,31 @@ use crate::aggregates::group_values::{GroupByMetrics, GroupValues, new_group_val
 use crate::aggregates::order::GroupOrdering;
 use crate::aggregates::row_hash::create_group_accumulator;
 use crate::aggregates::{
-    AggregateExec, PhysicalGroupBy, aggregate_expressions, evaluate_group_by,
+    AggregateExec, AggregateMode, PhysicalGroupBy, aggregate_expressions,
+    evaluate_group_by, group_id_array, max_duplicate_ordinal,
 };
 
-/// Marker for raw rows -> partial state aggregation.
+/// Semantic mode for the aggregate hash table.
+///
+/// See [`AggregateHashTable`] comment's 'Mode' section for details.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub(in crate::aggregates) enum AggregateTableMode {
+    /// Raw rows -> partial aggregate state rows.
+    Partial,
+    /// Partial aggregate state rows -> final aggregate value rows.
+    Final,
+    /// Partial aggregate state rows -> merged partial aggregate state rows.
+    PartialReduce,
+    /// Raw rows -> final aggregate value rows.
+    Single,
+}
+
+/// Marker for ordered raw rows -> partial state aggregation.
 pub(in crate::aggregates) struct PartialMarker;

Review Comment:
   do you plan to remove this PartialMarker for the ordered streams too? Maybe we could do the same simplification



##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -214,6 +321,239 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
         state.batch_group_indices = Vec::new();
         self.state = AggregateHashTableState::Outputting(state);
     }
+
+    /// Aggregates one input batch according to this table's input semantics.
+    ///
+    /// `Partial` and `Single` update accumulator state from raw input rows.
+    /// `Final` and `PartialReduce` merge accumulator state emitted by an
+    /// earlier aggregate stage.
+    pub(in crate::aggregates) fn aggregate_batch(
+        &mut self,
+        batch: &RecordBatch,
+    ) -> Result<()> {
+        let evaluated_batch = self.evaluate_batch(batch)?;
+        let mode = self.mode;
+        let state = self.state.building_mut();
+
+        let timer = self.group_by_metrics.aggregation_time.timer();
+        for group_values in &evaluated_batch.grouping_set_args {
+            state
+                .group_values
+                .intern(group_values, &mut state.batch_group_indices)?;
+            let group_indices = &state.batch_group_indices;
+            let total_num_groups = state.group_values.len();
+
+            for (acc, values) in state
+                .accumulators
+                .iter_mut()
+                .zip(evaluated_batch.accumulator_args.iter())
+            {
+                match mode {

Review Comment:
   it probably doesn't matter as this is one match per column, but I wonder if we should do the match outside and the loop inside πŸ€”  



##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -214,6 +321,239 @@ impl<AggrMode> AggregateHashTable<AggrMode> {
         state.batch_group_indices = Vec::new();
         self.state = AggregateHashTableState::Outputting(state);
     }
+
+    /// Aggregates one input batch according to this table's input semantics.
+    ///
+    /// `Partial` and `Single` update accumulator state from raw input rows.
+    /// `Final` and `PartialReduce` merge accumulator state emitted by an
+    /// earlier aggregate stage.
+    pub(in crate::aggregates) fn aggregate_batch(
+        &mut self,
+        batch: &RecordBatch,
+    ) -> Result<()> {
+        let evaluated_batch = self.evaluate_batch(batch)?;
+        let mode = self.mode;
+        let state = self.state.building_mut();
+
+        let timer = self.group_by_metrics.aggregation_time.timer();
+        for group_values in &evaluated_batch.grouping_set_args {
+            state
+                .group_values
+                .intern(group_values, &mut state.batch_group_indices)?;
+            let group_indices = &state.batch_group_indices;
+            let total_num_groups = state.group_values.len();
+
+            for (acc, values) in state
+                .accumulators
+                .iter_mut()
+                .zip(evaluated_batch.accumulator_args.iter())
+            {
+                match mode {
+                    AggregateTableMode::Partial | AggregateTableMode::Single => {
+                        acc.update_batch(values, group_indices, total_num_groups)?
+                    }
+                    AggregateTableMode::Final | AggregateTableMode::PartialReduce => {
+                        acc.merge_batch(values, group_indices, total_num_groups)?
+                    }
+                }
+            }
+        }
+        drop(timer);
+
+        Ok(())
+    }
+
+    /// Emits the next output batch according to this table's output semantics.
+    ///
+    /// `Partial` and `PartialReduce` emit accumulator states for downstream
+    /// aggregate stages. `Final` and `Single` evaluate final aggregate values
+    /// for the query result.
+    pub(in crate::aggregates) fn next_output_batch(
+        &mut self,
+    ) -> Result<Option<RecordBatch>> {
+        let output_schema = Arc::clone(&self.output_schema);
+        let batch_size = self.batch_size;
+        // Take ownership of the output state. `emit_next_materialized_batch`
+        // restores `self.state` to `OutputtingMaterialized` or `Done`.
+        match std::mem::replace(&mut self.state, AggregateHashTableState::Done) {
+            AggregateHashTableState::Outputting(state) => {
+                if state.group_values.is_empty() {
+                    return Ok(None);
+                }
+
+                let output = self.materialize_output(state, output_schema)?;
+                Ok(self.emit_next_materialized_batch(output, batch_size))
+            }
+            AggregateHashTableState::OutputtingMaterialized(output) => {
+                Ok(self.emit_next_materialized_batch(output, batch_size))
+            }
+            AggregateHashTableState::Done => Ok(None),
+            AggregateHashTableState::Building(_) => {
+                internal_err!("next_output_batch must be called in the outputting state")
+            }
+        }
+    }
+
+    fn materialize_output(
+        &self,
+        mut state: AggregateHashTableBuffer,
+        output_schema: SchemaRef,
+    ) -> Result<MaterializedAggregateOutput> {
+        // Final evaluation and partial state emission both consume accumulator
+        // state. Materialize all groups once, then slice on subsequent polls.
+        let emit_to = EmitTo::All;
+        let timer = self.group_by_metrics.emitting_time.timer();
+        let mut output = state.group_values.emit(emit_to)?;
+
+        for acc in state.accumulators.iter_mut() {
+            match self.mode {
+                AggregateTableMode::Partial | AggregateTableMode::PartialReduce => {
+                    output.extend(acc.state(emit_to)?)
+                }
+                AggregateTableMode::Final | AggregateTableMode::Single => {
+                    output.push(acc.evaluate(emit_to)?)
+                }
+            }
+        }
+        drop(timer);
+
+        let batch = RecordBatch::try_new(output_schema, output)?;
+        debug_assert!(batch.num_rows() > 0);
+        Ok(MaterializedAggregateOutput::new(batch))
+    }
+
+    fn emit_next_materialized_batch(
+        &mut self,
+        mut output: MaterializedAggregateOutput,
+        batch_size: usize,
+    ) -> Option<RecordBatch> {
+        let batch = output.next_batch(batch_size);
+        if output.is_exhausted() {
+            self.state = AggregateHashTableState::Done;
+        } else {
+            self.state = AggregateHashTableState::OutputtingMaterialized(output);
+        }
+        batch
+    }
+
+    pub(in crate::aggregates) fn start_output(&mut self) -> Result<()> {
+        if matches!(
+            self.mode,
+            AggregateTableMode::Partial | AggregateTableMode::Single
+        ) {
+            self.init_empty_grouping_sets()?;
+        }
+        self.start_outputting();
+        Ok(())
+    }
+
+    /// Creates the required empty grouping-set rows when the input is empty.
+    ///
+    /// For example, this query must still produce one grand-total group even if
+    /// `t` has no rows:
+    ///
+    /// ```sql
+    /// SELECT COUNT(v)
+    /// FROM t
+    /// GROUP BY GROUPING SETS (());
+    /// ```
+    ///
+    /// The synthetic row is filtered out before accumulator update so aggregates
+    /// see the same state they would see for an empty input, rather than a real
+    /// null-valued row.
+    pub(super) fn init_empty_grouping_sets(&mut self) -> Result<()> {
+        let state = self.state.building_mut();
+        if !state.group_by.has_grouping_set() || !state.group_values.is_empty() {
+            return Ok(());
+        }
+
+        let max_ordinal = max_duplicate_ordinal(state.group_by.groups());
+        let mut ordinals: HashMap<&[bool], usize> = HashMap::new();
+        let group_schema = state.group_by.group_schema(&self.input_schema)?;
+        let n_expr = state.group_by.expr().len();
+        let mut any_interned = false;
+
+        for group in state.group_by.groups() {
+            let ordinal = {
+                let entry = ordinals.entry(group.as_slice()).or_insert(0);
+                let ordinal = *entry;
+                *entry += 1;
+                ordinal
+            };
+
+            if !group.iter().all(|&is_null| is_null) {
+                continue;
+            }
+
+            let mut cols: Vec<ArrayRef> = group_schema
+                .fields()
+                .iter()
+                .take(n_expr)
+                .map(|field| new_null_array(field.data_type(), 1))
+                .collect();
+            cols.push(group_id_array(group, ordinal, max_ordinal, 1)?);
+
+            state
+                .group_values
+                .intern(&cols, &mut state.batch_group_indices)?;
+            any_interned = true;
+        }
+
+        if any_interned {
+            let total_groups = state.group_values.len();
+            let false_filter = BooleanArray::from(vec![false]);
+            for acc in state.accumulators.iter_mut() {
+                let null_args = acc.null_arguments(&self.input_schema)?;
+                let values = EvaluatedAccumulatorArgs {
+                    arguments: null_args,
+                    filter: Some(Arc::new(false_filter.clone())),
+                };
+                acc.update_batch(&values, &[0], total_groups)?;
+            }
+        }
+
+        Ok(())
+    }
+}
+
+/// Hash table used only for converting raw input rows directly into partial
+/// aggregate state rows after partial aggregation has been skipped.
+pub(in crate::aggregates) struct PartialSkipHashTable {

Review Comment:
   i like that this has a named newtype wrapper



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#issuecomment-4879367758

   πŸ€– Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23309#issuecomment-4879357464)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4879357464-842-kqn89 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing split-aggr-simplify-ht (7f68c238d8de5a3a8eb461504cf25c8b636c2940) to e4aa41d (merge-base) [diff](https://github.com/apache/datafusion/compare/e4aa41d7c68f2fbbd1f8f9de93a7029ceb3e265c..7f68c238d8de5a3a8eb461504cf25c8b636c2940) using: tpcds
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#issuecomment-4879424791

   πŸ€– Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23309#issuecomment-4879357464)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and split-aggr-simplify-ht
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃                split-aggr-simplify-ht ┃        Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚           5.52 / 6.00 Β±0.86 / 7.72 ms β”‚           5.58 / 6.08 Β±0.86 / 7.79 ms β”‚     no change β”‚
   β”‚ QQuery 2  β”‚        82.18 / 82.38 Β±0.15 / 82.59 ms β”‚        80.74 / 81.00 Β±0.21 / 81.25 ms β”‚     no change β”‚
   β”‚ QQuery 3  β”‚        29.51 / 29.76 Β±0.24 / 30.22 ms β”‚        29.55 / 29.83 Β±0.18 / 30.06 ms β”‚     no change β”‚
   β”‚ QQuery 4  β”‚     485.95 / 492.28 Β±3.32 / 495.00 ms β”‚    545.86 / 566.81 Β±10.66 / 575.63 ms β”‚  1.15x slower β”‚
   β”‚ QQuery 5  β”‚        51.91 / 52.16 Β±0.31 / 52.75 ms β”‚        51.60 / 52.96 Β±1.44 / 55.56 ms β”‚     no change β”‚
   β”‚ QQuery 6  β”‚        36.54 / 37.18 Β±0.37 / 37.57 ms β”‚        36.19 / 36.67 Β±0.25 / 36.92 ms β”‚     no change β”‚
   β”‚ QQuery 7  β”‚        94.56 / 95.36 Β±0.67 / 96.37 ms β”‚       93.47 / 96.31 Β±2.59 / 100.82 ms β”‚     no change β”‚
   β”‚ QQuery 8  β”‚        37.01 / 38.27 Β±1.76 / 41.73 ms β”‚        36.56 / 36.99 Β±0.52 / 38.01 ms β”‚     no change β”‚
   β”‚ QQuery 9  β”‚        53.01 / 55.13 Β±1.76 / 58.34 ms β”‚        52.87 / 53.92 Β±0.82 / 55.17 ms β”‚     no change β”‚
   β”‚ QQuery 10 β”‚        62.91 / 63.38 Β±0.29 / 63.72 ms β”‚        63.82 / 64.55 Β±0.67 / 65.77 ms β”‚     no change β”‚
   β”‚ QQuery 11 β”‚     305.88 / 315.55 Β±6.94 / 326.16 ms β”‚     297.00 / 301.90 Β±2.66 / 305.12 ms β”‚     no change β”‚
   β”‚ QQuery 12 β”‚        29.31 / 29.61 Β±0.19 / 29.78 ms β”‚        28.55 / 28.81 Β±0.14 / 28.96 ms β”‚     no change β”‚
   β”‚ QQuery 13 β”‚     119.63 / 120.33 Β±0.89 / 122.06 ms β”‚     118.62 / 120.18 Β±1.21 / 122.34 ms β”‚     no change β”‚
   β”‚ QQuery 14 β”‚     413.99 / 419.88 Β±3.84 / 425.70 ms β”‚     413.96 / 418.42 Β±3.74 / 425.29 ms β”‚     no change β”‚
   β”‚ QQuery 15 β”‚        57.85 / 58.62 Β±0.64 / 59.56 ms β”‚        57.10 / 58.67 Β±1.10 / 60.16 ms β”‚     no change β”‚
   β”‚ QQuery 16 β”‚           6.84 / 6.96 Β±0.16 / 7.28 ms β”‚           6.75 / 6.93 Β±0.21 / 7.35 ms β”‚     no change β”‚
   β”‚ QQuery 17 β”‚        80.93 / 82.30 Β±1.95 / 86.15 ms β”‚        80.99 / 82.38 Β±1.28 / 84.08 ms β”‚     no change β”‚
   β”‚ QQuery 18 β”‚     124.44 / 125.71 Β±1.02 / 127.04 ms β”‚     123.43 / 125.26 Β±1.42 / 127.13 ms β”‚     no change β”‚
   β”‚ QQuery 19 β”‚        41.54 / 41.66 Β±0.10 / 41.82 ms β”‚        41.67 / 42.59 Β±0.94 / 44.40 ms β”‚     no change β”‚
   β”‚ QQuery 20 β”‚        35.80 / 36.97 Β±1.83 / 40.62 ms β”‚        35.84 / 37.01 Β±0.99 / 38.43 ms β”‚     no change β”‚
   β”‚ QQuery 21 β”‚        17.59 / 18.04 Β±0.31 / 18.48 ms β”‚        18.02 / 18.18 Β±0.15 / 18.44 ms β”‚     no change β”‚
   β”‚ QQuery 22 β”‚        62.39 / 62.90 Β±0.42 / 63.66 ms β”‚        62.30 / 64.02 Β±1.21 / 65.42 ms β”‚     no change β”‚
   β”‚ QQuery 23 β”‚     348.97 / 352.54 Β±2.80 / 356.84 ms β”‚     341.57 / 347.49 Β±3.94 / 351.81 ms β”‚     no change β”‚
   β”‚ QQuery 24 β”‚     226.10 / 231.53 Β±6.39 / 241.59 ms β”‚     225.63 / 227.34 Β±2.82 / 232.94 ms β”‚     no change β”‚
   β”‚ QQuery 25 β”‚     118.74 / 119.43 Β±0.45 / 120.15 ms β”‚     110.17 / 111.75 Β±1.12 / 113.56 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 26 β”‚        62.07 / 63.03 Β±0.60 / 63.94 ms β”‚        58.68 / 59.59 Β±0.70 / 60.83 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 27 β”‚           7.11 / 7.21 Β±0.17 / 7.55 ms β”‚           6.38 / 6.51 Β±0.17 / 6.84 ms β”‚ +1.11x faster β”‚
   β”‚ QQuery 28 β”‚        63.64 / 64.12 Β±0.57 / 65.18 ms β”‚        61.00 / 61.70 Β±0.68 / 62.90 ms β”‚     no change β”‚
   β”‚ QQuery 29 β”‚     102.26 / 107.95 Β±6.86 / 120.47 ms β”‚        97.00 / 97.57 Β±0.68 / 98.89 ms β”‚ +1.11x faster β”‚
   β”‚ QQuery 30 β”‚        36.18 / 36.86 Β±0.55 / 37.84 ms β”‚        33.04 / 34.74 Β±1.95 / 38.56 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 31 β”‚     117.80 / 120.84 Β±1.74 / 122.86 ms β”‚     111.98 / 112.23 Β±0.20 / 112.54 ms β”‚ +1.08x faster β”‚
   β”‚ QQuery 32 β”‚        21.78 / 22.68 Β±0.68 / 23.83 ms β”‚        20.53 / 21.00 Β±0.53 / 21.85 ms β”‚ +1.08x faster β”‚
   β”‚ QQuery 33 β”‚        38.92 / 39.55 Β±0.65 / 40.46 ms β”‚        37.87 / 38.33 Β±0.33 / 38.64 ms β”‚     no change β”‚
   β”‚ QQuery 34 β”‚        10.56 / 10.97 Β±0.43 / 11.75 ms β”‚         9.93 / 11.72 Β±2.70 / 17.04 ms β”‚  1.07x slower β”‚
   β”‚ QQuery 35 β”‚        77.97 / 78.29 Β±0.19 / 78.51 ms β”‚        72.44 / 73.35 Β±0.70 / 74.48 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 36 β”‚           6.29 / 6.43 Β±0.15 / 6.72 ms β”‚           6.03 / 6.15 Β±0.16 / 6.47 ms β”‚     no change β”‚
   β”‚ QQuery 37 β”‚           7.53 / 7.58 Β±0.05 / 7.68 ms β”‚           7.04 / 7.17 Β±0.11 / 7.35 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 38 β”‚        62.93 / 65.41 Β±2.01 / 67.84 ms β”‚        61.85 / 62.52 Β±0.57 / 63.38 ms β”‚     no change β”‚
   β”‚ QQuery 39 β”‚        90.78 / 91.98 Β±0.84 / 93.17 ms β”‚        89.98 / 91.18 Β±0.91 / 92.76 ms β”‚     no change β”‚
   β”‚ QQuery 40 β”‚        23.69 / 24.10 Β±0.30 / 24.39 ms β”‚        23.73 / 24.24 Β±0.37 / 24.83 ms β”‚     no change β”‚
   β”‚ QQuery 41 β”‚        11.55 / 11.66 Β±0.12 / 11.89 ms β”‚        11.57 / 11.79 Β±0.24 / 12.25 ms β”‚     no change β”‚
   β”‚ QQuery 42 β”‚        23.75 / 24.13 Β±0.24 / 24.41 ms β”‚        23.77 / 24.20 Β±0.35 / 24.77 ms β”‚     no change β”‚
   β”‚ QQuery 43 β”‚           4.91 / 5.02 Β±0.16 / 5.34 ms β”‚           5.12 / 5.22 Β±0.15 / 5.52 ms β”‚     no change β”‚
   β”‚ QQuery 44 β”‚           9.34 / 9.42 Β±0.07 / 9.51 ms β”‚           9.50 / 9.67 Β±0.11 / 9.85 ms β”‚     no change β”‚
   β”‚ QQuery 45 β”‚        39.77 / 42.79 Β±3.12 / 48.83 ms β”‚        37.80 / 38.44 Β±0.88 / 40.17 ms β”‚ +1.11x faster β”‚
   β”‚ QQuery 46 β”‚        11.89 / 12.49 Β±0.50 / 13.17 ms β”‚        11.88 / 12.13 Β±0.17 / 12.38 ms β”‚     no change β”‚
   β”‚ QQuery 47 β”‚     231.00 / 233.21 Β±2.95 / 238.89 ms β”‚    228.03 / 261.78 Β±26.13 / 293.45 ms β”‚  1.12x slower β”‚
   β”‚ QQuery 48 β”‚        96.83 / 97.63 Β±0.57 / 98.58 ms β”‚      98.69 / 101.32 Β±2.00 / 104.57 ms β”‚     no change β”‚
   β”‚ QQuery 49 β”‚        75.98 / 77.31 Β±0.93 / 78.91 ms β”‚        79.49 / 81.60 Β±2.52 / 86.50 ms β”‚  1.06x slower β”‚
   β”‚ QQuery 50 β”‚        58.86 / 59.35 Β±0.35 / 59.88 ms β”‚        61.97 / 62.93 Β±0.61 / 63.81 ms β”‚  1.06x slower β”‚
   β”‚ QQuery 51 β”‚        91.42 / 94.01 Β±2.92 / 99.21 ms β”‚       97.24 / 99.03 Β±1.40 / 100.82 ms β”‚  1.05x slower β”‚
   β”‚ QQuery 52 β”‚        23.87 / 24.75 Β±0.85 / 25.92 ms β”‚        25.42 / 26.15 Β±0.89 / 27.86 ms β”‚  1.06x slower β”‚
   β”‚ QQuery 53 β”‚        29.65 / 29.96 Β±0.29 / 30.49 ms β”‚        30.38 / 30.84 Β±0.41 / 31.52 ms β”‚     no change β”‚
   β”‚ QQuery 54 β”‚        55.55 / 55.88 Β±0.33 / 56.46 ms β”‚        57.37 / 57.84 Β±0.38 / 58.26 ms β”‚     no change β”‚
   β”‚ QQuery 55 β”‚        23.24 / 23.56 Β±0.38 / 24.29 ms β”‚        24.18 / 24.44 Β±0.24 / 24.82 ms β”‚     no change β”‚
   β”‚ QQuery 56 β”‚        38.66 / 39.02 Β±0.21 / 39.21 ms β”‚        40.47 / 40.58 Β±0.10 / 40.73 ms β”‚     no change β”‚
   β”‚ QQuery 57 β”‚     177.64 / 179.94 Β±2.34 / 184.17 ms β”‚     176.51 / 183.06 Β±6.13 / 191.30 ms β”‚     no change β”‚
   β”‚ QQuery 58 β”‚     118.49 / 119.65 Β±1.09 / 121.24 ms β”‚     114.88 / 118.06 Β±2.48 / 122.34 ms β”‚     no change β”‚
   β”‚ QQuery 59 β”‚     118.48 / 118.90 Β±0.37 / 119.38 ms β”‚     117.34 / 117.83 Β±0.37 / 118.32 ms β”‚     no change β”‚
   β”‚ QQuery 60 β”‚        39.87 / 41.31 Β±1.41 / 43.66 ms β”‚        39.59 / 40.27 Β±0.69 / 41.58 ms β”‚     no change β”‚
   β”‚ QQuery 61 β”‚        12.35 / 12.52 Β±0.21 / 12.93 ms β”‚        12.60 / 12.73 Β±0.15 / 13.02 ms β”‚     no change β”‚
   β”‚ QQuery 62 β”‚        46.45 / 47.03 Β±0.51 / 47.79 ms β”‚        46.22 / 46.43 Β±0.22 / 46.82 ms β”‚     no change β”‚
   β”‚ QQuery 63 β”‚        29.79 / 30.03 Β±0.27 / 30.50 ms β”‚        29.90 / 30.14 Β±0.13 / 30.27 ms β”‚     no change β”‚
   β”‚ QQuery 64 β”‚    410.82 / 423.58 Β±12.27 / 443.75 ms β”‚     408.39 / 415.09 Β±6.76 / 425.01 ms β”‚     no change β”‚
   β”‚ QQuery 65 β”‚     154.74 / 159.81 Β±4.59 / 167.76 ms β”‚     141.85 / 144.49 Β±2.32 / 148.57 ms β”‚ +1.11x faster β”‚
   β”‚ QQuery 66 β”‚        85.40 / 86.47 Β±0.67 / 87.32 ms β”‚        79.06 / 79.50 Β±0.37 / 79.93 ms β”‚ +1.09x faster β”‚
   β”‚ QQuery 67 β”‚    265.23 / 273.79 Β±10.47 / 294.15 ms β”‚     237.81 / 242.42 Β±4.56 / 250.95 ms β”‚ +1.13x faster β”‚
   β”‚ QQuery 68 β”‚        11.97 / 12.17 Β±0.25 / 12.66 ms β”‚        11.91 / 12.08 Β±0.19 / 12.46 ms β”‚     no change β”‚
   β”‚ QQuery 69 β”‚        57.43 / 57.96 Β±0.49 / 58.83 ms β”‚        58.78 / 60.17 Β±2.14 / 64.42 ms β”‚     no change β”‚
   β”‚ QQuery 70 β”‚     105.29 / 110.36 Β±7.11 / 124.16 ms β”‚     104.78 / 107.04 Β±2.30 / 111.39 ms β”‚     no change β”‚
   β”‚ QQuery 71 β”‚        35.49 / 36.11 Β±0.52 / 37.08 ms β”‚        35.20 / 35.82 Β±0.54 / 36.64 ms β”‚     no change β”‚
   β”‚ QQuery 72 β”‚ 2098.62 / 2148.50 Β±50.13 / 2236.60 ms β”‚ 2144.19 / 2247.86 Β±82.17 / 2394.52 ms β”‚     no change β”‚
   β”‚ QQuery 73 β”‚          9.65 / 9.92 Β±0.22 / 10.24 ms β”‚         9.74 / 10.01 Β±0.29 / 10.45 ms β”‚     no change β”‚
   β”‚ QQuery 74 β”‚     170.45 / 174.40 Β±3.77 / 179.43 ms β”‚     166.95 / 171.09 Β±4.80 / 180.15 ms β”‚     no change β”‚
   β”‚ QQuery 75 β”‚     149.41 / 151.02 Β±1.61 / 154.05 ms β”‚     148.47 / 149.74 Β±1.02 / 150.88 ms β”‚     no change β”‚
   β”‚ QQuery 76 β”‚        35.17 / 35.65 Β±0.31 / 36.10 ms β”‚        35.03 / 35.54 Β±0.57 / 36.39 ms β”‚     no change β”‚
   β”‚ QQuery 77 β”‚        61.42 / 65.04 Β±3.90 / 71.61 ms β”‚        61.13 / 61.49 Β±0.34 / 62.11 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 78 β”‚     218.67 / 224.47 Β±5.22 / 233.96 ms β”‚     194.52 / 198.27 Β±4.18 / 205.00 ms β”‚ +1.13x faster β”‚
   β”‚ QQuery 79 β”‚        72.20 / 72.85 Β±0.35 / 73.21 ms β”‚        67.06 / 68.06 Β±1.46 / 70.96 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 80 β”‚     106.93 / 113.35 Β±4.85 / 121.48 ms β”‚      98.68 / 100.35 Β±1.25 / 102.43 ms β”‚ +1.13x faster β”‚
   β”‚ QQuery 81 β”‚        28.75 / 29.08 Β±0.46 / 29.95 ms β”‚        25.94 / 26.11 Β±0.18 / 26.45 ms β”‚ +1.11x faster β”‚
   β”‚ QQuery 82 β”‚        18.10 / 18.29 Β±0.15 / 18.49 ms β”‚        16.62 / 16.67 Β±0.05 / 16.76 ms β”‚ +1.10x faster β”‚
   β”‚ QQuery 83 β”‚        42.77 / 43.29 Β±0.71 / 44.65 ms β”‚        40.40 / 41.06 Β±0.90 / 42.85 ms β”‚ +1.05x faster β”‚
   β”‚ QQuery 84 β”‚        31.79 / 32.01 Β±0.18 / 32.33 ms β”‚        30.47 / 30.74 Β±0.19 / 30.97 ms β”‚     no change β”‚
   β”‚ QQuery 85 β”‚     112.76 / 115.84 Β±3.78 / 122.97 ms β”‚     107.16 / 110.48 Β±3.46 / 117.11 ms β”‚     no change β”‚
   β”‚ QQuery 86 β”‚        26.61 / 26.84 Β±0.27 / 27.36 ms β”‚        24.84 / 25.07 Β±0.12 / 25.18 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 87 β”‚        68.02 / 68.95 Β±1.02 / 70.89 ms β”‚        62.92 / 63.41 Β±0.36 / 63.79 ms β”‚ +1.09x faster β”‚
   β”‚ QQuery 88 β”‚        65.42 / 66.28 Β±0.73 / 67.43 ms β”‚        62.99 / 64.18 Β±0.70 / 65.19 ms β”‚     no change β”‚
   β”‚ QQuery 89 β”‚        37.84 / 38.36 Β±0.62 / 39.46 ms β”‚        36.33 / 36.87 Β±0.52 / 37.82 ms β”‚     no change β”‚
   β”‚ QQuery 90 β”‚        18.30 / 18.46 Β±0.13 / 18.67 ms β”‚        17.35 / 17.59 Β±0.18 / 17.85 ms β”‚     no change β”‚
   β”‚ QQuery 91 β”‚        46.87 / 48.23 Β±0.75 / 49.17 ms β”‚        46.82 / 47.19 Β±0.21 / 47.45 ms β”‚     no change β”‚
   β”‚ QQuery 92 β”‚        29.78 / 31.76 Β±3.02 / 37.77 ms β”‚        29.76 / 30.39 Β±0.52 / 31.14 ms β”‚     no change β”‚
   β”‚ QQuery 93 β”‚        50.74 / 51.67 Β±0.85 / 53.16 ms β”‚        50.55 / 51.03 Β±0.32 / 51.45 ms β”‚     no change β”‚
   β”‚ QQuery 94 β”‚        38.98 / 39.29 Β±0.24 / 39.69 ms β”‚        41.21 / 43.52 Β±2.76 / 48.81 ms β”‚  1.11x slower β”‚
   β”‚ QQuery 95 β”‚        81.49 / 82.27 Β±0.41 / 82.62 ms β”‚        86.47 / 87.49 Β±0.57 / 88.01 ms β”‚  1.06x slower β”‚
   β”‚ QQuery 96 β”‚        24.43 / 26.14 Β±2.70 / 31.51 ms β”‚        25.46 / 25.57 Β±0.07 / 25.65 ms β”‚     no change β”‚
   β”‚ QQuery 97 β”‚        47.12 / 47.81 Β±0.63 / 48.94 ms β”‚        49.07 / 49.89 Β±0.87 / 51.55 ms β”‚     no change β”‚
   β”‚ QQuery 98 β”‚        42.35 / 43.27 Β±1.04 / 45.20 ms β”‚        46.85 / 49.09 Β±3.30 / 55.64 ms β”‚  1.13x slower β”‚
   β”‚ QQuery 99 β”‚        70.76 / 71.14 Β±0.37 / 71.77 ms β”‚        73.10 / 73.81 Β±0.46 / 74.27 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                     ┃            ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                     β”‚ 10164.89ms β”‚
   β”‚ Total Time (split-aggr-simplify-ht)   β”‚ 10189.70ms β”‚
   β”‚ Average Time (HEAD)                   β”‚   102.68ms β”‚
   β”‚ Average Time (split-aggr-simplify-ht) β”‚   102.93ms β”‚
   β”‚ Queries Faster                        β”‚         22 β”‚
   β”‚ Queries Slower                        β”‚         10 β”‚
   β”‚ Queries with No Change                β”‚         67 β”‚
   β”‚ Queries with Failure                  β”‚          0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds β€” base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 2.0 GiB |
   | Avg memory | 1.4 GiB |
   | CPU user | 228.8s |
   | CPU sys | 5.8s |
   | Peak spill | 0 B |
   
   **tpcds β€” branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 55.0s |
   | Peak memory | 2.2 GiB |
   | Avg memory | 1.5 GiB |
   | CPU user | 233.8s |
   | CPU sys | 5.7s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3524618135


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -60,19 +72,36 @@ pub(in crate::aggregates) struct FinalMarker;
 /// [`GroupsAccumulator`]. Both use columnar storage so aggregation can stay
 /// vectorized.
 ///
-/// # Marker Type
-/// `AggrMode` selects the aggregate semantics.
+/// # Mode
+///
+/// [`AggregateTableMode`] controls how input batches update accumulator state
+/// and how output batches are materialized.
 ///
-/// e.g. `AggregateHashTable::<PartialMarker>::new(...)` creates an aggregate hash table
-/// for the partial hash aggregate stage, the input schema is raw rows and output
-/// schema is intermediate states.
+/// ```text
+/// Example: `AVG(x) GROUP BY k`.
 ///
-/// It is a zero-sized compile-time marker, so each stage keeps its update logic
-/// in a separate impl block, to make the behavior difference explicit.
-pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
+/// In `Partial` mode, the table stores partial state:
+///     k, sum(x), count(x)
+/// The input batch contains raw values:
+///     k, x
+/// The output batch also contains partial state:
+///     k, sum(x), count(x)
+/// ```
+///
+/// So input uses [`GroupsAccumulator::update_batch`], and output uses
+/// [`GroupsAccumulator::state`].
+///
+/// Other modes use different input/output combinations:
+/// - `Final`: merge_batch + evaluate
+/// - `PartialReduce`: merge_batch + state
+/// - `Single`: update_batch + evaluate
+pub(in crate::aggregates) struct AggregateHashTable {

Review Comment:
   The templated-function also seems good for me.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3522932671


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -60,19 +72,36 @@ pub(in crate::aggregates) struct FinalMarker;
 /// [`GroupsAccumulator`]. Both use columnar storage so aggregation can stay
 /// vectorized.
 ///
-/// # Marker Type
-/// `AggrMode` selects the aggregate semantics.
+/// # Mode
+///
+/// [`AggregateTableMode`] controls how input batches update accumulator state
+/// and how output batches are materialized.
 ///
-/// e.g. `AggregateHashTable::<PartialMarker>::new(...)` creates an aggregate hash table
-/// for the partial hash aggregate stage, the input schema is raw rows and output
-/// schema is intermediate states.
+/// ```text
+/// Example: `AVG(x) GROUP BY k`.
 ///
-/// It is a zero-sized compile-time marker, so each stage keeps its update logic
-/// in a separate impl block, to make the behavior difference explicit.
-pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
+/// In `Partial` mode, the table stores partial state:
+///     k, sum(x), count(x)
+/// The input batch contains raw values:
+///     k, x
+/// The output batch also contains partial state:
+///     k, sum(x), count(x)
+/// ```
+///
+/// So input uses [`GroupsAccumulator::update_batch`], and output uses
+/// [`GroupsAccumulator::state`].
+///
+/// Other modes use different input/output combinations:
+/// - `Final`: merge_batch + evaluate
+/// - `PartialReduce`: merge_batch + state
+/// - `Single`: update_batch + evaluate
+pub(in crate::aggregates) struct AggregateHashTable {

Review Comment:
   > My concern is that this would create a [shallow module](https://softengbook.org/articles/deep-modules). To fully understand the implementation, we would still need to reason about the same underlying mode differences, while the external AggregateHashTable variants would add complexity with almost no functional contribution.
   
   One idea could be to leave the structure that is already on main, and refactor the common behavior out as templated functions on the base class
   
   So for example, leave  `AggregateHashTable<AggrMode>`
   
   But then in the parts that are very similar like `aggregate_batch` instead of
   
   ```rust
   impl AggregateHashTable<PartialReduceMarker> {
   ...
      pub(in crate::aggregates) fn aggregate_batch(
           &mut self,
           batch: &RecordBatch,
       ) -> Result<()> {
           let evaluated_batch = self.evaluate_batch(batch)?;
           let state = self.state.building_mut();
   
           let timer = self.group_by_metrics.aggregation_time.timer();
           for group_values in &evaluated_batch.grouping_set_args {
               state
                   .group_values
                   .intern(group_values, &mut state.batch_group_indices)?;
               let group_indices = &state.batch_group_indices;
               let total_num_groups = state.group_values.len();
   
               for (acc, values) in state
                   .accumulators
                   .iter_mut()
                   .zip(evaluated_batch.accumulator_args.iter())
               {
                   acc.merge_batch(values, group_indices, total_num_groups)?;
               }
           }
           drop(timer);
   
           Ok(())
       }
   ..
   }
   ```
   
   It could be structured something so that the specialization was called
   
   ```rust
   impl AggregateHashTable<PartialReduceMarker> {
   ...
      pub(in crate::aggregates) fn aggregate_batch(
           &mut self,
           batch: &RecordBatch,
       ) -> Result<()> {
        // call a templated function with a closure with the hash table specific functionality
         self.aggregate_batch_inner(batch, |acc, values, group_indices, total_num_groups| {
                   acc.merge_batch(values, group_indices, total_num_groups)
          })?;
       }
   ...
   }
   ```
   
   And then move the common logic into the generic method
   
   ```rust
   /// Implement in "base" class (not the specialization)
   impl AggregateHashTable<AggrMode> {
   ...
       fn <F: Fn(...) -> Result<()>)aggregate_batch_inner(
           batch: &RecordBatch,
           agg_function: F,
       ) -> Result<()> {
         let evaluated_batch = self.evaluate_batch(batch)?;
           let state = self.state.building_mut();
   
           let timer = self.group_by_metrics.aggregation_time.timer();
           for group_values in &evaluated_batch.grouping_set_args {
               state
                   .group_values
                   .intern(group_values, &mut state.batch_group_indices)?;
               let group_indices = &state.batch_group_indices;
               let total_num_groups = state.group_values.len();
   
               for (acc, values) in state
                   .accumulators
                   .iter_mut()
                   .zip(evaluated_batch.accumulator_args.iter())
               {
                 // *** Note here call the generic function agg_function: F
                  agg_function(values, group_indices, total_num_groups)?; <---------
               }
           }
           drop(timer);
   
           Ok(())
       }
     }
   }
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#issuecomment-4879357464

   run benchmarks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#issuecomment-4879435675

   πŸ€– Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23309#issuecomment-4879357464)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and split-aggr-simplify-ht
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃                split-aggr-simplify-ht ┃        Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 0  β”‚          1.23 / 4.05 Β±5.49 / 15.03 ms β”‚          1.24 / 3.97 Β±5.40 / 14.77 ms β”‚     no change β”‚
   β”‚ QQuery 1  β”‚        12.66 / 13.02 Β±0.19 / 13.16 ms β”‚        12.34 / 12.63 Β±0.15 / 12.78 ms β”‚     no change β”‚
   β”‚ QQuery 2  β”‚        35.82 / 36.11 Β±0.24 / 36.38 ms β”‚        35.83 / 36.19 Β±0.28 / 36.56 ms β”‚     no change β”‚
   β”‚ QQuery 3  β”‚        30.77 / 31.62 Β±1.12 / 33.83 ms β”‚        30.51 / 30.92 Β±0.46 / 31.72 ms β”‚     no change β”‚
   β”‚ QQuery 4  β”‚     223.84 / 227.07 Β±2.41 / 230.03 ms β”‚     223.49 / 226.15 Β±2.47 / 230.33 ms β”‚     no change β”‚
   β”‚ QQuery 5  β”‚     273.31 / 273.96 Β±0.54 / 274.91 ms β”‚     271.76 / 272.72 Β±1.12 / 274.60 ms β”‚     no change β”‚
   β”‚ QQuery 6  β”‚           1.31 / 1.47 Β±0.23 / 1.91 ms β”‚           1.28 / 1.43 Β±0.22 / 1.86 ms β”‚     no change β”‚
   β”‚ QQuery 7  β”‚        13.86 / 14.04 Β±0.12 / 14.23 ms β”‚        13.76 / 13.98 Β±0.13 / 14.15 ms β”‚     no change β”‚
   β”‚ QQuery 8  β”‚     323.12 / 326.73 Β±2.02 / 329.19 ms β”‚     316.74 / 322.16 Β±2.75 / 324.43 ms β”‚     no change β”‚
   β”‚ QQuery 9  β”‚    438.72 / 465.95 Β±14.14 / 478.29 ms β”‚     453.60 / 461.13 Β±6.62 / 471.50 ms β”‚     no change β”‚
   β”‚ QQuery 10 β”‚        69.97 / 70.81 Β±0.83 / 71.85 ms β”‚        71.15 / 72.41 Β±1.30 / 74.76 ms β”‚     no change β”‚
   β”‚ QQuery 11 β”‚        80.88 / 82.73 Β±2.48 / 87.64 ms β”‚        82.22 / 83.65 Β±1.26 / 85.93 ms β”‚     no change β”‚
   β”‚ QQuery 12 β”‚     267.15 / 271.64 Β±4.61 / 277.44 ms β”‚     266.16 / 274.36 Β±8.00 / 284.98 ms β”‚     no change β”‚
   β”‚ QQuery 13 β”‚     367.13 / 377.31 Β±9.65 / 394.53 ms β”‚    364.32 / 380.35 Β±14.23 / 402.66 ms β”‚     no change β”‚
   β”‚ QQuery 14 β”‚     281.15 / 283.77 Β±2.21 / 287.64 ms β”‚     278.90 / 283.77 Β±5.89 / 294.76 ms β”‚     no change β”‚
   β”‚ QQuery 15 β”‚     274.29 / 281.90 Β±7.93 / 295.76 ms β”‚     268.77 / 274.53 Β±3.27 / 277.48 ms β”‚     no change β”‚
   β”‚ QQuery 16 β”‚     612.00 / 619.45 Β±6.46 / 629.63 ms β”‚     614.30 / 618.34 Β±3.44 / 624.36 ms β”‚     no change β”‚
   β”‚ QQuery 17 β”‚     628.63 / 634.48 Β±4.92 / 643.40 ms β”‚     614.06 / 626.81 Β±9.07 / 641.75 ms β”‚     no change β”‚
   β”‚ QQuery 18 β”‚ 1238.98 / 1265.03 Β±19.70 / 1294.62 ms β”‚ 1240.74 / 1260.59 Β±15.76 / 1283.65 ms β”‚     no change β”‚
   β”‚ QQuery 19 β”‚        27.70 / 29.25 Β±2.75 / 34.74 ms β”‚        27.46 / 30.29 Β±4.64 / 39.46 ms β”‚     no change β”‚
   β”‚ QQuery 20 β”‚     517.07 / 528.49 Β±6.79 / 535.75 ms β”‚     515.30 / 519.37 Β±2.74 / 522.02 ms β”‚     no change β”‚
   β”‚ QQuery 21 β”‚     512.11 / 518.18 Β±5.81 / 529.24 ms β”‚     513.94 / 519.96 Β±5.72 / 529.95 ms β”‚     no change β”‚
   β”‚ QQuery 22 β”‚    985.77 / 996.83 Β±8.63 / 1008.67 ms β”‚    979.96 / 988.81 Β±9.32 / 1005.15 ms β”‚     no change β”‚
   β”‚ QQuery 23 β”‚ 3044.81 / 3070.47 Β±20.65 / 3099.53 ms β”‚ 3083.92 / 3107.22 Β±24.64 / 3151.60 ms β”‚     no change β”‚
   β”‚ QQuery 24 β”‚        41.03 / 47.91 Β±8.30 / 60.86 ms β”‚       42.10 / 50.69 Β±10.43 / 66.91 ms β”‚  1.06x slower β”‚
   β”‚ QQuery 25 β”‚     111.43 / 112.34 Β±1.01 / 114.24 ms β”‚     111.74 / 113.16 Β±1.57 / 116.17 ms β”‚     no change β”‚
   β”‚ QQuery 26 β”‚        41.73 / 42.32 Β±0.41 / 43.00 ms β”‚        41.47 / 41.95 Β±0.57 / 43.00 ms β”‚     no change β”‚
   β”‚ QQuery 27 β”‚     665.83 / 673.12 Β±5.60 / 682.99 ms β”‚     673.06 / 676.68 Β±2.71 / 679.75 ms β”‚     no change β”‚
   β”‚ QQuery 28 β”‚  3037.33 / 3045.50 Β±6.08 / 3055.04 ms β”‚ 3021.32 / 3051.62 Β±17.60 / 3076.16 ms β”‚     no change β”‚
   β”‚ QQuery 29 β”‚        40.47 / 44.60 Β±7.69 / 59.97 ms β”‚        40.65 / 40.84 Β±0.13 / 41.03 ms β”‚ +1.09x faster β”‚
   β”‚ QQuery 30 β”‚    305.50 / 315.80 Β±10.20 / 334.07 ms β”‚    302.12 / 314.55 Β±16.25 / 346.64 ms β”‚     no change β”‚
   β”‚ QQuery 31 β”‚    287.86 / 296.56 Β±10.55 / 312.63 ms β”‚     281.08 / 295.23 Β±8.23 / 306.84 ms β”‚     no change β”‚
   β”‚ QQuery 32 β”‚    918.23 / 949.74 Β±20.64 / 969.99 ms β”‚    911.77 / 926.81 Β±15.75 / 954.52 ms β”‚     no change β”‚
   β”‚ QQuery 33 β”‚ 1439.46 / 1464.72 Β±23.17 / 1507.76 ms β”‚ 1428.11 / 1461.49 Β±24.67 / 1491.91 ms β”‚     no change β”‚
   β”‚ QQuery 34 β”‚  1475.01 / 1481.97 Β±4.32 / 1486.36 ms β”‚ 1478.44 / 1513.12 Β±29.95 / 1558.43 ms β”‚     no change β”‚
   β”‚ QQuery 35 β”‚    278.56 / 295.84 Β±23.63 / 342.62 ms β”‚    276.20 / 289.44 Β±14.85 / 316.56 ms β”‚     no change β”‚
   β”‚ QQuery 36 β”‚        65.78 / 71.55 Β±4.17 / 78.28 ms β”‚        66.51 / 70.28 Β±4.34 / 77.41 ms β”‚     no change β”‚
   β”‚ QQuery 37 β”‚        35.91 / 40.60 Β±4.08 / 46.64 ms β”‚        35.27 / 38.76 Β±3.21 / 44.53 ms β”‚     no change β”‚
   β”‚ QQuery 38 β”‚        40.98 / 43.78 Β±2.75 / 48.74 ms β”‚        41.27 / 43.90 Β±2.78 / 48.66 ms β”‚     no change β”‚
   β”‚ QQuery 39 β”‚     136.07 / 147.42 Β±7.73 / 157.86 ms β”‚     142.77 / 153.15 Β±8.31 / 165.88 ms β”‚     no change β”‚
   β”‚ QQuery 40 β”‚        14.09 / 16.36 Β±3.13 / 22.45 ms β”‚        14.02 / 16.88 Β±5.24 / 27.36 ms β”‚     no change β”‚
   β”‚ QQuery 41 β”‚        13.65 / 13.93 Β±0.22 / 14.19 ms β”‚        13.58 / 14.14 Β±0.55 / 14.98 ms β”‚     no change β”‚
   β”‚ QQuery 42 β”‚        13.44 / 15.57 Β±4.08 / 23.72 ms β”‚        13.25 / 15.26 Β±3.76 / 22.78 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                     ┃            ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                     β”‚ 19543.95ms β”‚
   β”‚ Total Time (split-aggr-simplify-ht)   β”‚ 19549.70ms β”‚
   β”‚ Average Time (HEAD)                   β”‚   454.51ms β”‚
   β”‚ Average Time (split-aggr-simplify-ht) β”‚   454.64ms β”‚
   β”‚ Queries Faster                        β”‚          1 β”‚
   β”‚ Queries Slower                        β”‚          1 β”‚
   β”‚ Queries with No Change                β”‚         41 β”‚
   β”‚ Queries with Failure                  β”‚          0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **clickbench_partitioned β€” base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 12.2 GiB |
   | Avg memory | 4.6 GiB |
   | CPU user | 1006.4s |
   | CPU sys | 65.8s |
   | Peak spill | 0 B |
   
   **clickbench_partitioned β€” branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 100.0s |
   | Peak memory | 12.0 GiB |
   | Avg memory | 4.4 GiB |
   | CPU user | 1007.0s |
   | CPU sys | 66.1s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#issuecomment-4879419004

   πŸ€– Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23309#issuecomment-4879357464)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and split-aggr-simplify-ht
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃         split-aggr-simplify-ht ┃        Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚ 38.84 / 41.07 Β±1.41 / 42.40 ms β”‚ 38.64 / 39.49 Β±1.06 / 41.53 ms β”‚     no change β”‚
   β”‚ QQuery 2  β”‚ 19.52 / 20.54 Β±0.75 / 21.51 ms β”‚ 19.25 / 19.69 Β±0.32 / 20.08 ms β”‚     no change β”‚
   β”‚ QQuery 3  β”‚ 31.49 / 33.63 Β±1.32 / 34.79 ms β”‚ 31.58 / 34.32 Β±1.53 / 36.22 ms β”‚     no change β”‚
   β”‚ QQuery 4  β”‚ 17.63 / 18.12 Β±0.28 / 18.47 ms β”‚ 17.97 / 18.36 Β±0.51 / 19.36 ms β”‚     no change β”‚
   β”‚ QQuery 5  β”‚ 40.19 / 42.56 Β±1.41 / 44.13 ms β”‚ 38.63 / 41.15 Β±1.31 / 42.33 ms β”‚     no change β”‚
   β”‚ QQuery 6  β”‚ 16.53 / 16.79 Β±0.20 / 17.11 ms β”‚ 16.45 / 16.54 Β±0.06 / 16.62 ms β”‚     no change β”‚
   β”‚ QQuery 7  β”‚ 45.55 / 48.36 Β±2.17 / 51.38 ms β”‚ 43.79 / 45.84 Β±1.48 / 48.29 ms β”‚ +1.05x faster β”‚
   β”‚ QQuery 8  β”‚ 44.16 / 44.31 Β±0.16 / 44.53 ms β”‚ 43.60 / 43.76 Β±0.13 / 43.97 ms β”‚     no change β”‚
   β”‚ QQuery 9  β”‚ 50.66 / 51.30 Β±0.58 / 52.24 ms β”‚ 50.66 / 51.42 Β±0.52 / 51.94 ms β”‚     no change β”‚
   β”‚ QQuery 10 β”‚ 43.09 / 43.82 Β±1.04 / 45.84 ms β”‚ 43.12 / 43.44 Β±0.31 / 43.85 ms β”‚     no change β”‚
   β”‚ QQuery 11 β”‚ 14.04 / 14.70 Β±0.80 / 16.24 ms β”‚ 13.91 / 13.97 Β±0.06 / 14.05 ms β”‚     no change β”‚
   β”‚ QQuery 12 β”‚ 24.41 / 26.10 Β±2.41 / 30.84 ms β”‚ 24.58 / 24.76 Β±0.19 / 25.10 ms β”‚ +1.05x faster β”‚
   β”‚ QQuery 13 β”‚ 34.31 / 36.23 Β±1.66 / 39.33 ms β”‚ 32.50 / 34.47 Β±1.16 / 35.80 ms β”‚     no change β”‚
   β”‚ QQuery 14 β”‚ 24.58 / 25.34 Β±1.11 / 27.51 ms β”‚ 24.59 / 25.15 Β±0.54 / 26.09 ms β”‚     no change β”‚
   β”‚ QQuery 15 β”‚ 31.86 / 32.68 Β±0.77 / 33.99 ms β”‚ 32.03 / 32.70 Β±0.50 / 33.56 ms β”‚     no change β”‚
   β”‚ QQuery 16 β”‚ 14.04 / 14.40 Β±0.26 / 14.77 ms β”‚ 14.61 / 14.76 Β±0.12 / 14.96 ms β”‚     no change β”‚
   β”‚ QQuery 17 β”‚ 74.24 / 75.68 Β±1.10 / 76.95 ms β”‚ 76.21 / 77.13 Β±1.30 / 79.71 ms β”‚     no change β”‚
   β”‚ QQuery 18 β”‚ 59.40 / 60.83 Β±1.27 / 62.74 ms β”‚ 60.90 / 62.32 Β±1.46 / 64.28 ms β”‚     no change β”‚
   β”‚ QQuery 19 β”‚ 33.73 / 33.97 Β±0.21 / 34.25 ms β”‚ 33.90 / 34.10 Β±0.21 / 34.49 ms β”‚     no change β”‚
   β”‚ QQuery 20 β”‚ 33.20 / 33.31 Β±0.12 / 33.50 ms β”‚ 33.00 / 33.25 Β±0.42 / 34.08 ms β”‚     no change β”‚
   β”‚ QQuery 21 β”‚ 57.04 / 58.10 Β±1.01 / 59.81 ms β”‚ 57.58 / 58.61 Β±0.93 / 60.15 ms β”‚     no change β”‚
   β”‚ QQuery 22 β”‚ 14.42 / 14.55 Β±0.09 / 14.63 ms β”‚ 14.08 / 14.75 Β±0.48 / 15.49 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                     ┃          ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                     β”‚ 786.41ms β”‚
   β”‚ Total Time (split-aggr-simplify-ht)   β”‚ 779.96ms β”‚
   β”‚ Average Time (HEAD)                   β”‚  35.75ms β”‚
   β”‚ Average Time (split-aggr-simplify-ht) β”‚  35.45ms β”‚
   β”‚ Queries Faster                        β”‚        2 β”‚
   β”‚ Queries Slower                        β”‚        0 β”‚
   β”‚ Queries with No Change                β”‚       20 β”‚
   β”‚ Queries with Failure                  β”‚        0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch β€” base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.1 GiB |
   | Avg memory | 507.1 MiB |
   | CPU user | 22.8s |
   | CPU sys | 1.8s |
   | Peak spill | 0 B |
   
   **tpch β€” branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.1 GiB |
   | Avg memory | 501.2 MiB |
   | CPU user | 23.0s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor(hash-aggr): Simplify aggregate hash table [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #23309:
URL: https://github.com/apache/datafusion/pull/23309#discussion_r3523163131


##########
datafusion/physical-plan/src/aggregates/aggregate_hash_table/common.rs:
##########
@@ -60,19 +72,36 @@ pub(in crate::aggregates) struct FinalMarker;
 /// [`GroupsAccumulator`]. Both use columnar storage so aggregation can stay
 /// vectorized.
 ///
-/// # Marker Type
-/// `AggrMode` selects the aggregate semantics.
+/// # Mode
+///
+/// [`AggregateTableMode`] controls how input batches update accumulator state
+/// and how output batches are materialized.
 ///
-/// e.g. `AggregateHashTable::<PartialMarker>::new(...)` creates an aggregate hash table
-/// for the partial hash aggregate stage, the input schema is raw rows and output
-/// schema is intermediate states.
+/// ```text
+/// Example: `AVG(x) GROUP BY k`.
 ///
-/// It is a zero-sized compile-time marker, so each stage keeps its update logic
-/// in a separate impl block, to make the behavior difference explicit.
-pub(in crate::aggregates) struct AggregateHashTable<AggrMode> {
+/// In `Partial` mode, the table stores partial state:
+///     k, sum(x), count(x)
+/// The input batch contains raw values:
+///     k, x
+/// The output batch also contains partial state:
+///     k, sum(x), count(x)
+/// ```
+///
+/// So input uses [`GroupsAccumulator::update_batch`], and output uses
+/// [`GroupsAccumulator::state`].
+///
+/// Other modes use different input/output combinations:
+/// - `Final`: merge_batch + evaluate
+/// - `PartialReduce`: merge_batch + state
+/// - `Single`: update_batch + evaluate
+pub(in crate::aggregates) struct AggregateHashTable {

Review Comment:
   The templated-function approach seems like a good idea. I’ll give it a try and see how it looks.
   
   If the templated function is not expressive enough in some places/at some point, we can still fully separate the implementation by aggregation variant.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org