You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "abhishek kumar tiwari (Jira)" <ji...@apache.org> on 2021/06/03 10:19:00 UTC
[jira] [Resolved] (SPARK-35626) Decommission executors in batches
to avoid overloading network by block migrations.
[ https://issues.apache.org/jira/browse/SPARK-35626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
abhishek kumar tiwari resolved SPARK-35626.
-------------------------------------------
Resolution: Duplicate
> Decommission executors in batches to avoid overloading network by block migrations.
> -----------------------------------------------------------------------------------
>
> Key: SPARK-35626
> URL: https://issues.apache.org/jira/browse/SPARK-35626
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.2
> Reporter: abhishek kumar tiwari
> Priority: Major
>
> Currenlty, each executor is asked to starts offloading rdd and shuffle blocks as soon it is decommissioned. This can overload the network bandwidth of the application.
>
> We should limit the number of executors migrating rdd and shuffle blocks at one time. This would be configurable to enable users to decommission executors conservatively or aggressively based on the use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org