You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2020/12/04 08:33:01 UTC
[jira] [Updated] (SPARK-30978) Remove multiple workers on the same
host support from Standalone backend
[ https://issues.apache.org/jira/browse/SPARK-30978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-30978:
---------------------------------
Target Version/s: 3.2.0 (was: 3.1.0)
> Remove multiple workers on the same host support from Standalone backend
> ------------------------------------------------------------------------
>
> Key: SPARK-30978
> URL: https://issues.apache.org/jira/browse/SPARK-30978
> Project: Spark
> Issue Type: Task
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Xingbo Jiang
> Assignee: Xingbo Jiang
> Priority: Major
>
> Based on our experience, there is no scenario that necessarily requires deploying multiple Workers on the same node with Standalone backend. A worker should book all the resources reserved to Spark on the host it is launched, then it can allocate those resources to one or more executors launched by this worker. Since each executor runs in a separated JVM, we can limit the memory of each executor to avoid long GC pause.
> The remaining concern is the local-cluster mode is implemented by launching multiple workers on the local host, we might need to re-implement LocalSparkCluster to launch only one Worker and multiple executors. It should be fine because local-cluster mode is only used in running Spark unit test cases, thus end users should not be affected by this change.
> Removing multiple workers on the same host support could simplify the deploy model of Standalone backend, and also reduce the burden to support legacy deploy pattern in the future feature developments.
> The proposal is to update the document to deprecate the support of system environment `SPARK_WORKER_INSTANCES` in 3.0, and remove the support in the next major version (3.1.0).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org