You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/09/09 02:46:00 UTC
[jira] [Assigned] (SPARK-36699) Reuse compatible executors for
stage-level scheduling
[ https://issues.apache.org/jira/browse/SPARK-36699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-36699:
------------------------------------
Assignee: Apache Spark
> Reuse compatible executors for stage-level scheduling
> -----------------------------------------------------
>
> Key: SPARK-36699
> URL: https://issues.apache.org/jira/browse/SPARK-36699
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.2
> Reporter: Xiaochang Wu
> Assignee: Apache Spark
> Priority: Major
> Fix For: 3.3.0
>
>
> Current stage-level scheduling allocated separated set of executors for different profiles. This approach simplified implementation, however is a waste of executor resources when the existing executors have enough resources to run the following tasks.
> We proposed to reuse executors by defining a "compatible" executor concept: two executors binding to different resource profiles are compatible only when the executorResources (cores in particular if not defining custom resources) are the same, but taskResources can be different. When the executors are compatible, the tasks can be allocated to any of them even when in the different profiles. Users defining profiles should make sure the different taskResources are properly specified against the same executorResources.
> The typical user scenario is for different stages, user wants to use different core number for the task with same executor resources. For instance in CPU machine learning scenario, to achieve the best performance, given the same executor resources, when in ETL stage, user will allocate 1 core per task and many tasks, and in the following CPU training stage, user will use more cores per task and less tasks. Reusing executors allows better CPU resource utilization and better performance.
> The first PR will focus on reusing executors with same cores without custom resources. A SparkConf option is defined to change the default behavior which is not reusing executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org