You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2026/07/08 04:44:00 UTC
[jira] [Resolved] (SPARK-58004) Add a hard reducer-partition fan-in bound to AQE shuffle coalescing
[ https://issues.apache.org/jira/browse/SPARK-58004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-58004.
-----------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57087
[https://github.com/apache/spark/pull/57087]
> Add a hard reducer-partition fan-in bound to AQE shuffle coalescing
> -------------------------------------------------------------------
>
> Key: SPARK-58004
> URL: https://issues.apache.org/jira/browse/SPARK-58004
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Chao Sun
> Assignee: Chao Sun
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> AQE's CoalesceShufflePartitions rule currently packs contiguous reducer partitions using post-shuffle byte size as its primary bound. For a shuffle with a very large number of tiny or empty reducer partitions, one CoalescedPartitionSpec can therefore span many thousands of original reducer partitions while remaining below the advisory target size. This can create excessive per-task shuffle fan-in and may force users to disable AQE coalescing entirely.
> For example, an AQE-enabled aggregation with 100,000 initial shuffle partitions and sparse output can produce coalesced tasks whose byte sizes are reasonable but whose reducer-partition ranges are extremely wide.
> Proposed behavior:
> * Add spark.sql.adaptive.coalescePartitions.maxReducerPartitionsPerTask as a positive integer configuration with an effectively unbounded default.
> * Enforce both the advisory byte target and the maximum reducer-partition span while coalescing.
> * Count empty reducer partitions toward the span limit.
> * Prevent the small-tail backward merge from violating the limit.
> * Preserve skew-split PartialReducerPartitionSpec semantics and apply the same ranges to all shuffle inputs in a coalesce group.
> * Preserve existing behavior when the configuration is not set.
> A limit based on remote shuffle blocks is intentionally out of scope because reliable block-count information is not available in this AQE planning path. That can be considered separately if such information becomes available.
> Related: SPARK-50257 discusses another case of overly aggressive AQE partition coalescing, but does not provide a general reducer-partition fan-in bound.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org