You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/08 00:54:35 UTC

(datafusion) branch gh-readonly-queue/main/pr-26069-b372961e722170a5a1166cb344bf60f40425cd96 created (now 97c7593f60)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-26069-b372961e722170a5a1166cb344bf60f40425cd96
in repository https://gitbox.apache.org/repos/asf/datafusion.git


      at 97c7593f60 refactor: Simplify `LimitedDistinctAggregation` optimizer rule, push limit in more cases (#26069)

This branch includes the following new commits:

     new 97c7593f60 refactor: Simplify `LimitedDistinctAggregation` optimizer rule, push limit in more cases (#26069)

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org


(datafusion) 01/01: refactor: Simplify `LimitedDistinctAggregation` optimizer rule, push limit in more cases (#26069)

Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-26069-b372961e722170a5a1166cb344bf60f40425cd96
in repository https://gitbox.apache.org/repos/asf/datafusion.git

commit 97c7593f609b277f65c086a7868a92c24ab5f8f2
Author: Yongting You <20...@gmail.com>
AuthorDate: Thu Oct 8 00:54:13 2026 +0000

    refactor: Simplify `LimitedDistinctAggregation` optimizer rule, push limit in more cases (#26069)
    
    ## Which issue does this PR close?
    
    <!--
    We generally require a GitHub issue to be filed for all bug fixes and
    enhancements and this helps us generate change logs for our releases.
    You can link an issue to this PR using the GitHub syntax. For example
    `Closes #123` indicates that this PR will close issue #123.
    -->
    
    This is primarily a simplifying refactor, with an optimizer regression
    fix piggy-backed on top (an inefficient plan shape, not a correctness
    issue):
    
    -
    https://github.com/apache/datafusion/pull/25983#issuecomment-5976014730
    
    The optimizer fix itself is only a one-line change. If this
    simplification is not desirable, I'll open a replacement PR containing
    just the direct fix.
    
    ## Rationale for this change
    
    <!--
    Why are you proposing this change? If this is already explained clearly
    in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand
    your changes and offer better suggestions for fixes.
    
    Please explain the problem you are trying to solve in terms of the
    user-visible
    behavior, rather than the implementation.
    
    For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of
    the
    implementation. "COUNT(DISTINCT) returns wrong results when the column
    contains
    nulls" is the user-visible problem.
    -->
    ## Part 1: Optimizer fix
    See sqllogictest diff for the reproducer
    I'll mark the fix in comments
    
    ## Part 2: Optimizer simplification
    
    During different stages in physical optimization, a logical aggregation
    can be either
    ```
    AggregateExec(mode=final)
    --AggregateExec(mode=partial)
    
    AggregateExec(mode=single)
    
    AggregateExec(mode=final)
    --RepartitionExec
    ----AggregateExec(mode=partial)
    ```
    
    So the existing implementation is using a nested dfs (the closure has
    another dfs inside, to match non-consecutive aggregates) try to match
    all of the 3 cases.
    
    However, after looking at related optimizer rules:
    - initial physical planning
    - CombinePartialFinalAggregate
    - EnsureDistributions
    - and this rule
    
    We can find only case 1 is possible, so the implementation can be
    simplified into a naive pattern matching
    ```
    Find the exact below shape, and try push limits
    
    LimitExec
    --AggregateExec(mode=final)
    ----AggregateExec(mode=partial)
    ```
    
    ## What changes are included in this PR?
    
    <!--
    There is no need to duplicate the description in the issue here, but it
    is sometimes worth providing a summary of the individual changes in this
    PR.
    -->
    - simplify an optimizer rule according to the above rationale
    - fix a small bug
    
    ## What is the testing strategy for this PR?
    
    <!--
    We typically require tests for all PRs in order to:
    1. Prevent the code from being accidentally broken by subsequent changes
    2. Serve as another way to document the expected behavior of the code
    
    Briefly describe how this PR is tested, and point to the specific tests
    you added. For example: 'This new feature is covered by the
    `sqllogictest` cases added in `foo.slt`'.
    
    If this PR does not add tests, explain why. For example, if the change
    is already covered by existing tests, please mention it.
    
    You should also check the `codecov` bot reply on this PR to confirm the
    changed code is exercised.
    -->
    - for bug fix, changes in `slt`
    - To ensure refactor correctness: no e2e test need to be changed
    
    ## Are there any user-facing changes?
    
    <!--
    If there are user-facing changes then we may require documentation to be
    updated before approving the PR.
    
    If there are any breaking changes to public APIs, please add the `api
    change` label.
    -->
---
 .../limited_distinct_aggregation.rs                | 109 +++++++----
 .../src/limited_distinct_aggregation.rs            | 213 +++++++++++----------
 datafusion/physical-plan/src/aggregates/mod.rs     |  17 ++
 datafusion/sqllogictest/test_files/aggregate.slt   |  12 +-
 4 files changed, 201 insertions(+), 150 deletions(-)

diff --git a/datafusion/core/tests/physical_optimizer/limited_distinct_aggregation.rs b/datafusion/core/tests/physical_optimizer/limited_distinct_aggregation.rs
index 1d5737d243..fae73af425 100644
--- a/datafusion/core/tests/physical_optimizer/limited_distinct_aggregation.rs
+++ b/datafusion/core/tests/physical_optimizer/limited_distinct_aggregation.rs
@@ -29,10 +29,13 @@ use arrow::datatypes::DataType;
 use arrow::{compute::SortOptions, util::pretty::pretty_format_batches};
 use datafusion::prelude::SessionContext;
 use datafusion_common::Result;
+use datafusion_common::config::ConfigOptions;
 use datafusion_execution::config::SessionConfig;
 use datafusion_expr::Operator;
 use datafusion_physical_expr::expressions::{self, cast, col};
 use datafusion_physical_expr_common::sort_expr::PhysicalSortExpr;
+use datafusion_physical_optimizer::PhysicalOptimizerRule;
+use datafusion_physical_optimizer::limited_distinct_aggregation::LimitedDistinctAggregation;
 use datafusion_physical_plan::{
     ExecutionPlan,
     aggregates::{AggregateExec, AggregateMode},
@@ -48,6 +51,37 @@ async fn run_plan_and_format(plan: Arc<dyn ExecutionPlan>) -> Result<String> {
     Ok(actual)
 }
 
+/// Construct test plan:
+///
+/// AggregateExec(mode=Final)
+/// -- AggregateExec(mode=Partial)
+fn distinct_aggregate_pair(
+    input: Arc<dyn ExecutionPlan>,
+    columns: &[&str],
+) -> Result<Arc<dyn ExecutionPlan>> {
+    let schema = input.schema();
+    let group_by = build_group_by(
+        &schema,
+        columns.iter().map(|column| (*column).to_string()).collect(),
+    );
+    let partial = AggregateExec::try_new(
+        AggregateMode::Partial,
+        group_by.clone(),
+        vec![],
+        vec![],
+        input,
+        Arc::clone(&schema),
+    )?;
+    Ok(Arc::new(AggregateExec::try_new(
+        AggregateMode::Final,
+        group_by.as_final(),
+        vec![],
+        vec![],
+        Arc::new(partial),
+        schema,
+    )?))
+}
+
 #[tokio::test]
 async fn test_partial_final() -> Result<()> {
     let source = mock_data()?;
@@ -338,12 +372,19 @@ async fn single_distinct_aggregate_stream_respects_soft_limit() -> Result<()> {
     Ok(())
 }
 
+/// This optimizer rule is guaranteed to have input shape
+///
+/// AggregateExec(mode=Final)
+///   AggregateExec(mode=Partial)
+///
+/// Combining compatible final/partial aggregate pair into one single aggregate
+/// happen in later optimizer phase, so no-op here.
 #[tokio::test]
-async fn test_single_local() -> Result<()> {
+async fn test_single_local_is_unchanged() -> Result<()> {
     let source = mock_data()?;
     let schema = source.schema();
 
-    // `SELECT a FROM DataSourceExec GROUP BY a LIMIT 4;`, Single AggregateExec
+    // A prebuilt Single aggregate is outside this rule's final/partial contract.
     let single_agg = AggregateExec::try_new(
         AggregateMode::Single,
         build_group_by(&schema.clone(), vec!["a".to_string()]),
@@ -356,15 +397,14 @@ async fn test_single_local() -> Result<()> {
         Arc::new(single_agg),
         4, // fetch
     );
-    // expected to push the limit to the AggregateExec
-    let plan: Arc<dyn ExecutionPlan> = Arc::new(limit_exec);
-    let formatted = get_optimized_plan(&plan)?;
-    let actual = formatted.trim();
+    let plan = LimitedDistinctAggregation::new()
+        .optimize(Arc::new(limit_exec), &ConfigOptions::new())?;
+    let actual = displayable(plan.as_ref()).indent(true).to_string();
     assert_snapshot!(
         actual,
         @r"
     LocalLimitExec: fetch=4
-      AggregateExec: mode=Single, gby=[a@0 as a], aggr=[], lim=[4]
+      AggregateExec: mode=Single, gby=[a@0 as a], aggr=[]
         DataSourceExec: partitions=1, partition_sizes=[1]
     "
     );
@@ -385,12 +425,13 @@ async fn test_single_local() -> Result<()> {
     Ok(())
 }
 
+/// See comments for `test_single_local_is_unchanged`
 #[tokio::test]
-async fn test_single_global() -> Result<()> {
+async fn test_single_global_is_unchanged() -> Result<()> {
     let source = mock_data()?;
     let schema = source.schema();
 
-    // `SELECT a FROM DataSourceExec GROUP BY a LIMIT 4;`, Single AggregateExec
+    // A global limit also leaves a prebuilt Single aggregate unchanged.
     let single_agg = AggregateExec::try_new(
         AggregateMode::Single,
         build_group_by(&schema.clone(), vec!["a".to_string()]),
@@ -404,15 +445,14 @@ async fn test_single_global() -> Result<()> {
         1,       // skip
         Some(3), // fetch
     );
-    // expected to push the skip+fetch limit to the AggregateExec
-    let plan: Arc<dyn ExecutionPlan> = Arc::new(limit_exec);
-    let formatted = get_optimized_plan(&plan)?;
-    let actual = formatted.trim();
+    let plan = LimitedDistinctAggregation::new()
+        .optimize(Arc::new(limit_exec), &ConfigOptions::new())?;
+    let actual = displayable(plan.as_ref()).indent(true).to_string();
     assert_snapshot!(
         actual,
         @r"
     GlobalLimitExec: skip=1, fetch=3
-      AggregateExec: mode=Single, gby=[a@0 as a], aggr=[], lim=[4]
+      AggregateExec: mode=Single, gby=[a@0 as a], aggr=[]
         DataSourceExec: partitions=1, partition_sizes=[1]
     "
     );
@@ -432,43 +472,32 @@ async fn test_single_global() -> Result<()> {
     Ok(())
 }
 
+/// For nested aggregation with limit, the limit only get pushed into parent aggregation
 #[tokio::test]
 async fn test_distinct_cols_different_than_group_by_cols() -> Result<()> {
     let source = mock_data()?;
-    let schema = source.schema();
 
-    // `SELECT distinct a FROM DataSourceExec GROUP BY a, b LIMIT 4;`, Single/Single AggregateExec
-    let group_by_agg = AggregateExec::try_new(
-        AggregateMode::Single,
-        build_group_by(&schema.clone(), vec!["a".to_string(), "b".to_string()]),
-        vec![],         /* aggr_expr */
-        vec![],         /* filter_expr */
-        source,         /* input */
-        schema.clone(), /* input_schema */
-    )?;
-    let distinct_agg = AggregateExec::try_new(
-        AggregateMode::Single,
-        build_group_by(&schema.clone(), vec!["a".to_string()]),
-        vec![],                 /* aggr_expr */
-        vec![],                 /* filter_expr */
-        Arc::new(group_by_agg), /* input */
-        schema.clone(),         /* input_schema */
-    )?;
+    // Each DISTINCT operation has its own final/partial pair.
+    let group_by_agg = distinct_aggregate_pair(source, &["a", "b"])?;
+    let distinct_agg = distinct_aggregate_pair(group_by_agg, &["a"])?;
     let limit_exec = LocalLimitExec::new(
-        Arc::new(distinct_agg),
+        distinct_agg,
         4, // fetch
     );
-    // expected to push the limit to the outer AggregateExec only
-    let plan: Arc<dyn ExecutionPlan> = Arc::new(limit_exec);
-    let formatted = get_optimized_plan(&plan)?;
-    let actual = formatted.trim();
+    // Only the outer pair receives the hint: four distinct (a, b) pairs might
+    // contain fewer than four distinct a values.
+    let plan = LimitedDistinctAggregation::new()
+        .optimize(Arc::new(limit_exec), &ConfigOptions::new())?;
+    let actual = displayable(plan.as_ref()).indent(true).to_string();
     assert_snapshot!(
         actual,
         @r"
     LocalLimitExec: fetch=4
-      AggregateExec: mode=Single, gby=[a@0 as a], aggr=[], lim=[4]
-        AggregateExec: mode=Single, gby=[a@0 as a, b@1 as b], aggr=[]
-          DataSourceExec: partitions=1, partition_sizes=[1]
+      AggregateExec: mode=Final, gby=[a@0 as a], aggr=[], lim=[4]
+        AggregateExec: mode=Partial, gby=[a@0 as a], aggr=[], lim=[4]
+          AggregateExec: mode=Final, gby=[a@0 as a, b@1 as b], aggr=[]
+            AggregateExec: mode=Partial, gby=[a@0 as a, b@1 as b], aggr=[]
+              DataSourceExec: partitions=1, partition_sizes=[1]
     "
     );
     let expected = run_plan_and_format(plan).await?;
diff --git a/datafusion/physical-optimizer/src/limited_distinct_aggregation.rs b/datafusion/physical-optimizer/src/limited_distinct_aggregation.rs
index 60833e0401..3636952050 100644
--- a/datafusion/physical-optimizer/src/limited_distinct_aggregation.rs
+++ b/datafusion/physical-optimizer/src/limited_distinct_aggregation.rs
@@ -15,26 +15,73 @@
 // specific language governing permissions and limitations
 // under the License.
 
-//! A special-case optimizer rule that pushes limit into a grouped aggregation
-//! which has no aggregate expressions or sorting requirements
+//! Stop unordered DISTINCT aggregation once enough groups have been found.
 
 use std::sync::Arc;
 
 use datafusion_physical_plan::aggregates::AggregateExec;
 use datafusion_physical_plan::limit::{GlobalLimitExec, LocalLimitExec};
-use datafusion_physical_plan::{ExecutionPlan, ExecutionPlanProperties};
+use datafusion_physical_plan::{
+    ChildrenPropertiesMode, ExecutionPlan, ExecutionPlanProperties,
+    ReplaceChildrenOptions,
+};
 
 use datafusion_common::Result;
 use datafusion_common::config::ConfigOptions;
 use datafusion_common::tree_node::{Transformed, TransformedResult, TreeNode};
 
 use crate::PhysicalOptimizerRule;
-use itertools::Itertools;
 
-/// An optimizer rule that passes a `limit` hint into grouped aggregations which don't require all
-/// rows in the group to be processed for correctness. Example queries fitting this description are:
-/// - `SELECT distinct l_orderkey FROM lineitem LIMIT 10;`
-/// - `SELECT l_orderkey FROM lineitem GROUP BY l_orderkey LIMIT 10;`
+/// Pushes a soft limit into unordered DISTINCT aggregations. For
+/// `SELECT DISTINCT a FROM t LIMIT 10`, any ten distinct values are sufficient:
+/// later rows cannot change the groups already found. The same reasoning applies
+/// to `GROUP BY a` without aggregate expressions. Aggregates such as `SUM` need
+/// every row in each group.
+///
+/// Each partial stage can also stop after finding ten distinct keys: those keys
+/// remain distinct in the final stage, even if other partitions contribute
+/// duplicates.
+///
+/// Before:
+///
+/// ```txt
+/// Limit(10)
+///   Aggregate(Final, a)
+///     Aggregate(Partial, a)
+///       Scan
+/// ```
+///
+/// After:
+///
+/// ```txt
+/// Limit(10)
+///   Aggregate(Final, a, soft_limit=10)
+///     Aggregate(Partial, a, soft_limit=10)
+///       Scan
+/// ```
+///
+/// # What this rule assumes
+///
+/// This rule assumes the logical aggregate only have one shape showed below, this
+/// is what the current physical planning produces.
+///
+/// ```txt
+/// Limit
+///   AggregateExec(mode=Final)
+///     AggregateExec(mode=Partial)
+/// ```
+///
+/// If future changes or extensions produce a different shape, this rule skips the
+/// rewrite rather than reporting an error, potentially missing an optimization
+/// opportunity.
+///
+/// # What this rule promises
+///
+/// Immediately after an eligible rewrite, both stages have a soft-limit hint.
+/// Otherwise, this rule leaves both stages unchanged.
+///
+/// If a later rule removes the limit, it won't affect correctness, but it may
+/// miss an optimization opportunity.
 #[derive(Debug)]
 pub struct LimitedDistinctAggregation {}
 
@@ -44,95 +91,66 @@ impl LimitedDistinctAggregation {
         Self {}
     }
 
-    fn transform_agg(
-        aggr: &AggregateExec,
-        limit: usize,
-    ) -> Option<Transformed<Arc<dyn ExecutionPlan>>> {
-        let new_aggr = aggr.clone().try_optimize_distinct_soft_limit(limit)?;
-        // An already limited aggregate still permits optimizing its partial child.
-        Some(new_aggr.update_data(|aggr| Arc::new(aggr) as Arc<dyn ExecutionPlan>))
-    }
+    /// Rewrite a limit and its immediately adjacent final/partial aggregate pair.
+    fn transform_limit(
+        plan: Arc<dyn ExecutionPlan>,
+    ) -> Result<Transformed<Arc<dyn ExecutionPlan>>> {
+        // Step 1: Identify the plan shape,
+        //
+        // Limit
+        //   Aggregate(final)
+        //     Aggregate(partial)
+
+        // Check the current plan is limit, and extract limit value, input plan.
+        let (limit, input) = match (
+            plan.downcast_ref::<LocalLimitExec>(),
+            plan.downcast_ref::<GlobalLimitExec>(),
+        ) {
+            (Some(local), _) => (local.fetch(), local.input()),
+            (_, Some(global)) => match global.fetch() {
+                Some(fetch) => (global.skip() + fetch, global.input()),
+                None => return Ok(Transformed::no(plan)),
+            },
+            _ => return Ok(Transformed::no(plan)),
+        };
 
-    /// transform_limit matches an `AggregateExec` as the child of a `LocalLimitExec`
-    /// or `GlobalLimitExec` and pushes the limit into the aggregation as a soft limit when
-    /// there is a group by, but no sorting, no aggregate expressions, and no filters in the
-    /// aggregation
-    fn transform_limit(plan: Arc<dyn ExecutionPlan>) -> Option<Arc<dyn ExecutionPlan>> {
-        let limit: usize;
-        let mut global_fetch: Option<usize> = None;
-        let mut global_skip: usize = 0;
-        let children: Vec<Arc<dyn ExecutionPlan>>;
-        let mut is_global_limit = false;
-        if let Some(local_limit) = plan.downcast_ref::<LocalLimitExec>() {
-            limit = local_limit.fetch();
-            children = local_limit.children().into_iter().cloned().collect();
-        } else {
-            let global_limit = plan.downcast_ref::<GlobalLimitExec>()?;
-            global_fetch = global_limit.fetch();
-            global_fetch?;
-            global_skip = global_limit.skip();
-            // the aggregate must read at least fetch+skip number of rows
-            limit = global_fetch.unwrap() + global_skip;
-            children = global_limit.children().into_iter().cloned().collect();
-            is_global_limit = true
-        }
-        let child = children.iter().exactly_one().ok()?;
-        // ensure there is no output ordering; can this rule be relaxed?
-        if plan.output_ordering().is_some() {
-            return None;
-        }
-        // ensure no ordering is required on the input
-        if plan.required_input_ordering()[0].is_some() {
-            return None;
+        if plan.output_ordering().is_some() || plan.required_input_ordering()[0].is_some()
+        {
+            return Ok(Transformed::no(plan));
         }
 
-        // if found_match_aggr is true, match_aggr holds a parent aggregation whose group_by
-        // must match that of a child aggregation in order to rewrite the child aggregation
-        let mut match_aggr: Arc<dyn ExecutionPlan> = plan;
-        let mut found_match_aggr = false;
-
-        let mut rewrite_applicable = true;
-        let closure = |plan: Arc<dyn ExecutionPlan>| {
-            if !rewrite_applicable {
-                return Ok(Transformed::no(plan));
-            }
-            if let Some(aggr) = plan.downcast_ref::<AggregateExec>() {
-                if found_match_aggr
-                    && let Some(parent_aggr) = match_aggr.downcast_ref::<AggregateExec>()
-                    && !parent_aggr.group_expr().eq(aggr.group_expr())
-                {
-                    // a partial and final aggregation with different groupings disqualifies
-                    // rewriting the child aggregation
-                    rewrite_applicable = false;
-                    return Ok(Transformed::no(plan));
-                }
-                // either we run into an Aggregate and transform it, or disable the rewrite
-                // for subsequent children
-                match Self::transform_agg(aggr, limit) {
-                    None => {}
-                    Some(new_aggr) => {
-                        match_aggr = plan;
-                        found_match_aggr = true;
-                        return Ok(new_aggr);
-                    }
-                }
-            }
-            rewrite_applicable = false;
-            Ok(Transformed::no(plan))
+        let Some(final_agg) = input.downcast_ref::<AggregateExec>() else {
+            return Ok(Transformed::no(plan));
         };
-        let child = child.to_owned().transform_down(closure).ok()?;
-        if !child.transformed {
-            return None;
+        let Some(partial_agg) = final_agg.input().downcast_ref::<AggregateExec>() else {
+            return Ok(Transformed::no(plan));
+        };
+
+        // Step 2: Verify partial/final aggregate is compatible, then apply optimization
+        if !final_agg.matches_partial(partial_agg) {
+            return Ok(Transformed::no(plan));
         }
-        let child = child.data;
-        if is_global_limit {
-            return Some(Arc::new(GlobalLimitExec::new(
-                child,
-                global_skip,
-                global_fetch,
-            )));
+
+        // Validate both stages before replacing either one.
+        let (Some(final_agg), Some(partial_agg)) = (
+            final_agg.clone().try_optimize_distinct_soft_limit(limit),
+            partial_agg.clone().try_optimize_distinct_soft_limit(limit),
+        ) else {
+            return Ok(Transformed::no(plan));
+        };
+        if !final_agg.transformed && !partial_agg.transformed {
+            return Ok(Transformed::no(plan));
         }
-        Some(Arc::new(LocalLimitExec::new(child, limit)))
+
+        let input = Arc::new(final_agg.data).replace_children(
+            vec![Arc::new(partial_agg.data)],
+            ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
+        )?;
+        plan.replace_children(
+            vec![input],
+            ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
+        )
+        .map(Transformed::yes)
     }
 }
 
@@ -149,18 +167,7 @@ impl PhysicalOptimizerRule for LimitedDistinctAggregation {
         config: &ConfigOptions,
     ) -> Result<Arc<dyn ExecutionPlan>> {
         if config.optimizer.enable_distinct_aggregation_soft_limit {
-            plan.transform_down(|plan| {
-                Ok(
-                    if let Some(plan) =
-                        LimitedDistinctAggregation::transform_limit(plan.to_owned())
-                    {
-                        Transformed::yes(plan)
-                    } else {
-                        Transformed::no(plan)
-                    },
-                )
-            })
-            .data()
+            plan.transform_down(Self::transform_limit).data()
         } else {
             Ok(plan)
         }
diff --git a/datafusion/physical-plan/src/aggregates/mod.rs b/datafusion/physical-plan/src/aggregates/mod.rs
index f454c0a830..6b56bd308b 100644
--- a/datafusion/physical-plan/src/aggregates/mod.rs
+++ b/datafusion/physical-plan/src/aggregates/mod.rs
@@ -1393,6 +1393,23 @@ impl AggregateExec {
         &self.mode
     }
 
+    /// Returns whether this is a compatible partial/final aggregation pair.
+    ///
+    /// # Public Only for Internal Use:
+    ///
+    /// This is made public for the physical optimizer to use, this is not part of
+    /// the public API.
+    #[doc(hidden)]
+    pub fn matches_partial(&self, partial: &Self) -> bool {
+        matches!(
+            (self.mode, partial.mode),
+            (
+                AggregateMode::Final | AggregateMode::FinalPartitioned,
+                AggregateMode::Partial,
+            )
+        ) && self.group_expr() == &partial.group_expr().as_final()
+    }
+
     /// Set a legacy limit hint. Unsupported requests leave ordinary aggregation.
     #[deprecated(
         since = "56.0.0",
diff --git a/datafusion/sqllogictest/test_files/aggregate.slt b/datafusion/sqllogictest/test_files/aggregate.slt
index a9ede1b1b4..61e4bd2543 100644
--- a/datafusion/sqllogictest/test_files/aggregate.slt
+++ b/datafusion/sqllogictest/test_files/aggregate.slt
@@ -7553,7 +7553,7 @@ physical_plan
 02)--GlobalLimitExec: skip=0, fetch=3
 03)----AggregateExec: mode=Final, gby=[c2@0 as c2, c3@1 as c3, __grouping_id@2 as __grouping_id], aggr=[], lim=[3]
 04)------CoalescePartitionsExec
-05)--------AggregateExec: mode=Partial, gby=[(NULL as c2, NULL as c3), (c2@0 as c2, NULL as c3), (c2@0 as c2, c3@1 as c3)], aggr=[]
+05)--------AggregateExec: mode=Partial, gby=[(NULL as c2, NULL as c3), (c2@0 as c2, NULL as c3), (c2@0 as c2, c3@1 as c3)], aggr=[], lim=[3]
 06)----------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
 07)------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/testing/data/csv/aggregate_test_100_with_dates.csv]]}, projection=[c2, c3], file_type=csv, has_header=true
 
@@ -7639,9 +7639,8 @@ logical_plan
 03)----TableScan: distinct_limit_combine projection=[a, b]
 physical_plan
 01)GlobalLimitExec: skip=0, fetch=2
-02)--AggregateExec: mode=Final, gby=[b@0 as b, a@1 as a], aggr=[], lim=[2]
-03)----AggregateExec: mode=Partial, gby=[b@1 as b, a@0 as a], aggr=[]
-04)------DataSourceExec: partitions=1, partition_sizes=[1]
+02)--AggregateExec: mode=Single, gby=[b@1 as b, a@0 as a], aggr=[], lim=[2]
+03)----DataSourceExec: partitions=1, partition_sizes=[1]
 
 # LIMIT without ORDER BY may return any two groups
 query II
@@ -7659,9 +7658,8 @@ logical_plan
 03)----TableScan: distinct_limit_combine projection=[a]
 physical_plan
 01)GlobalLimitExec: skip=0, fetch=2
-02)--AggregateExec: mode=Final, gby=[distinct_limit_combine.a % Int64(3)@0 as distinct_limit_combine.a % Int64(3)], aggr=[], lim=[2]
-03)----AggregateExec: mode=Partial, gby=[a@0 % 3 as distinct_limit_combine.a % Int64(3)], aggr=[]
-04)------DataSourceExec: partitions=1, partition_sizes=[1]
+02)--AggregateExec: mode=Single, gby=[a@0 % 3 as distinct_limit_combine.a % Int64(3)], aggr=[], lim=[2]
+03)----DataSourceExec: partitions=1, partition_sizes=[1]
 
 query II
 SELECT count(*), count(DISTINCT m)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org