You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/10 14:01:45 UTC
[I] AQE: dynamically coalesce shuffle partitions based on runtime statistics [datafusion-ballista]
andygrove opened a new issue, #1987:
URL: https://github.com/apache/datafusion-ballista/issues/1987
**Is your feature request related to a problem or challenge?**
The adaptive execution path currently does not dynamically coalesce shuffle partitions. When a shuffle stage produces very little data spread across many partitions, downstream stages still schedule one task per partition, adding scheduler overhead and small/empty tasks. This is the AQE analog of Spark's coalesce-shuffle-partitions (elastic shuffle parallelism).
**Describe the solution you'd like**
After a shuffle stage completes, use its per-partition output statistics to coalesce partitions in the downstream `ShuffleReaderExec` partitioning, reducing the number of tasks for the next stage. The EPIC design describes this under "Coalesce Shuffle Partitions (Elastic Shuffle Parallelism)".
**Additional context**
A `coalesce` module stub exists at `ballista/scheduler/src/state/aqe/coalesce.rs` but runtime coalescing is not yet wired in. Older closed issue #813 covered making the static shuffle partition *count* configurable, which is a different concern.
## Context
This is one of the documented limitations of the experimental Adaptive Query Execution (AQE) path, listed in [`docs/source/user-guide/tuning-guide.md`](https://github.com/apache/datafusion-ballista/blob/main/docs/source/user-guide/tuning-guide.md) under "Current limitations" and in the doc comment on `AdaptiveExecutionGraph` (`ballista/scheduler/src/state/aqe/mod.rs`). Filing a discrete tracking issue so the gap is visible outside the doc.
Parent epic: #1359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] AQE: dynamically coalesce shuffle partitions based on runtime statistics [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1987:
URL: https://github.com/apache/datafusion-ballista/issues/1987#issuecomment-4946330597
@andygrove is this same or different from #1684 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] AQE: dynamically coalesce shuffle partitions based on runtime statistics [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1987: AQE: dynamically coalesce shuffle partitions based on runtime statistics
URL: https://github.com/apache/datafusion-ballista/issues/1987
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] AQE: dynamically coalesce shuffle partitions based on runtime statistics [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1987:
URL: https://github.com/apache/datafusion-ballista/issues/1987#issuecomment-4946532755
> [@andygrove](https://github.com/andygrove) is this same or different from [#1684](https://github.com/apache/datafusion-ballista/pull/1684) ?
Let me check. I created this issue because it was listed as a limitation in the documentation. Perhaps the docs are out of date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] AQE: dynamically coalesce shuffle partitions based on runtime statistics [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1987:
URL: https://github.com/apache/datafusion-ballista/issues/1987#issuecomment-4946600199
> > [@andygrove](https://github.com/andygrove) is this same or different from [#1684](https://github.com/apache/datafusion-ballista/pull/1684) ?
>
> Let me check. I created this issue because it was listed as a limitation in the documentation. Perhaps the docs are out of date.
Yes, it's the same. I'll close this issue and update the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org