You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "NGA-TRAN (via GitHub)" <gi...@apache.org> on 2026/08/17 21:58:10 UTC
[PR] feat: stream aggregates when group keys are partition-disjoint [datafusion]
NGA-TRAN opened a new pull request, #24440:
URL: https://github.com/apache/datafusion/pull/24440
## Which issue does this PR close?
- Closes #24438.
## Rationale for this change
When data is range-partitioned on group-by keys, merging many data partitions into fewer DataFusion partitions (to keep `target_partitions` near the CPU count) can lose the global `(key, time)` sort order. `AggregateExec` then falls back to blocking hash aggregation even though group-by keys still do not overlap across the merged partitions.
This PR lets callers declare that guarantee so aggregation can stream without requiring a global sort.
## What changes are included in this PR?
- Add a `partition_disjoint_exprs` property on `EquivalenceProperties`, with setter/clear/satisfies helpers.
- Let `MemoryExec` / `TestMemoryExec` attach the property via `try_with_partition_disjoint_keys`.
- When `InputOrderMode` is `Linear` but group-by keys are partition-disjoint, `AggregateExec` uses incremental emission and a `GroupOrdering::PartitionDisjoint` path that emits completed groups when each partition finishes.
- Clear the property on `CoalescePartitionsExec` and `RepartitionExec`, where the disjointness guarantee is lost.
- Treat the property like ordering when deciding whether preserving distribution enables streaming.
## Are these changes tested?
- Unit test `partition_disjoint_partial_aggregate_planning` checks that the property enables streaming planning, and that the same unsorted input without the property stays blocking.
- Fuzz test `partition_disjoint_streaming_aggregate_test` compares partition-disjoint incremental aggregation against a concatenated baseline.
## Are there any user-facing changes?
Yes: public APIs on `EquivalenceProperties` and `MemoryExec` / `TestMemoryExec` to declare partition-disjoint group keys. No SQL syntax changes. Downstream engines that merge range-partitioned files can set this property so `AggregateExec` streams.
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: stream aggregates when group keys are partition-disjoint [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24440:
URL: https://github.com/apache/datafusion/pull/24440#issuecomment-5320803138
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v54.1.0 (current)
Built [ 48.171s] (current)
Parsing datafusion v54.1.0 (current)
Parsed [ 0.030s] (current)
Building datafusion v54.1.0 (baseline)
Built [ 46.593s] (baseline)
Parsing datafusion v54.1.0 (baseline)
Parsed [ 0.030s] (baseline)
Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.738s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 97.405s] datafusion
Building datafusion-datasource v54.1.0 (current)
Built [ 35.606s] (current)
Parsing datafusion-datasource v54.1.0 (current)
Parsed [ 0.027s] (current)
Building datafusion-datasource v54.1.0 (baseline)
Built [ 35.943s] (baseline)
Parsing datafusion-datasource v54.1.0 (baseline)
Parsed [ 0.028s] (baseline)
Checking datafusion-datasource v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.286s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 72.803s] datafusion-datasource
Building datafusion-physical-expr v54.1.0 (current)
Built [ 23.915s] (current)
Parsing datafusion-physical-expr v54.1.0 (current)
Parsed [ 0.041s] (current)
Building datafusion-physical-expr v54.1.0 (baseline)
Built [ 23.959s] (baseline)
Parsing datafusion-physical-expr v54.1.0 (baseline)
Parsed [ 0.042s] (baseline)
Checking datafusion-physical-expr v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.404s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 49.092s] datafusion-physical-expr
Building datafusion-physical-optimizer v54.1.0 (current)
Built [ 33.689s] (current)
Parsing datafusion-physical-optimizer v54.1.0 (current)
Parsed [ 0.019s] (current)
Building datafusion-physical-optimizer v54.1.0 (baseline)
Built [ 32.154s] (baseline)
Parsing datafusion-physical-optimizer v54.1.0 (baseline)
Parsed [ 0.019s] (baseline)
Checking datafusion-physical-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.122s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 66.937s] datafusion-physical-optimizer
Building datafusion-physical-plan v54.1.0 (current)
Built [ 30.538s] (current)
Parsing datafusion-physical-plan v54.1.0 (current)
Parsed [ 0.120s] (current)
Building datafusion-physical-plan v54.1.0 (baseline)
Built [ 29.952s] (baseline)
Parsing datafusion-physical-plan v54.1.0 (baseline)
Parsed [ 0.121s] (baseline)
Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.762s] 223 checks: 221 pass, 2 fail, 0 warn, 31 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron
Failed in:
variant GroupOrdering:PartitionDisjoint in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/order/mod.rs:44
--- failure method_parameter_count_changed: pub method parameter count changed ---
Description:
A publicly-visible method now takes a different number of parameters, not counting the receiver (self) parameter.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/method_parameter_count_changed.ron
Failed in:
datafusion_physical_plan::aggregates::AggregateExec::compute_properties takes 7 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/9be51f2368a0cdeb800473095ecd452bc1ae30c1/datafusion/physical-plan/src/aggregates/mod.rs:1369, but now takes 8 parameters in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/mod.rs:1383
datafusion_physical_plan::aggregates::order::GroupOrdering::try_new takes 1 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/9be51f2368a0cdeb800473095ecd452bc1ae30c1/datafusion/physical-plan/src/aggregates/order/mod.rs:44, but now takes 2 parameters in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/aggregates/order/mod.rs:49
Summary semver requires new major version: 2 major and 0 minor checks failed
Finished [ 62.517s] datafusion-physical-plan
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: stream aggregates when group keys are partition-disjoint [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #24440: feat: stream aggregates when group keys are partition-disjoint
URL: https://github.com/apache/datafusion/pull/24440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: stream aggregates when group keys are partition-disjoint [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24440:
URL: https://github.com/apache/datafusion/pull/24440#issuecomment-6228052259
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org