You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "XiDuo You (Jira)" <ji...@apache.org> on 2026/07/02 10:12:00 UTC

[jira] [Commented] (SPARK-57878) Add task assignment strategy to support binpack/balance task scheduling

    [ https://issues.apache.org/jira/browse/SPARK-57878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093198#comment-18093198 ] 

XiDuo You commented on SPARK-57878:
-----------------------------------

Relevant history work: SPARK-17637, SPARK-43086

> Add task assignment strategy to support binpack/balance task scheduling
> -----------------------------------------------------------------------
>
>                 Key: SPARK-57878
>                 URL: https://issues.apache.org/jira/browse/SPARK-57878
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 4.3.0
>            Reporter: XiDuo You
>            Priority: Major
>
> The scheduler hard-coded a single placement policy (random shuffle), only overridable through a protected test hook. Different workloads want different placement:
> - binpack concentrates tasks so idle executors can be reclaimed by dynamic allocation / decommissioning — useful for cost efficiency on elastic clusters.
> - balance spreads tasks to maximize parallelism and avoid hot-spotting a single executor.
> Making the policy pluggable and config-driven lets operators pick per workload without code changes, while keeping the existing shuffle behavior as the default.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org