You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "weixiuli (JIRA)" <ji...@apache.org> on 2018/12/06 14:09:00 UTC
[jira] [Updated] (SPARK-26288) add initRegisteredExecutorsDB in
ExternalShuffleService
[ https://issues.apache.org/jira/browse/SPARK-26288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
weixiuli updated SPARK-26288:
-----------------------------
Description:
As we all know that spark on Yarn uses DB to record RegisteredExecutors information which can be reloaded and used again when the ExternalShuffleService is restarted .
The RegisteredExecutors information can't be recorded both in the mode of spark's standalone and spark on k8s , which will cause the RegisteredExecutors information to be lost ,when the ExternalShuffleService is restarted.
To solve the problem above, a method is proposed and is committed .
was:
As we all know that spark on Yarn uses DB to record RegisteredExecutors information, when the ExternalShuffleService restart and it can be reloaded, which will be used as well .
While neither spark's standalone nor spark on k8s can record it's RegisteredExecutors information by db or others ,so when ExternalShuffleService restart ,which RegisteredExecutors information will be lost,it is't what we looking forward to .
This commit add initRegisteredExecutorsDB which can be used either spark standalone or spark on k8s to record RegisteredExecutors information , when the ExternalShuffleService restart and it can be reloaded, which will be used as well .
> add initRegisteredExecutorsDB in ExternalShuffleService
> -------------------------------------------------------
>
> Key: SPARK-26288
> URL: https://issues.apache.org/jira/browse/SPARK-26288
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes, Shuffle
> Affects Versions: 2.4.0
> Reporter: weixiuli
> Priority: Major
> Fix For: 2.4.0
>
>
> As we all know that spark on Yarn uses DB to record RegisteredExecutors information which can be reloaded and used again when the ExternalShuffleService is restarted .
> The RegisteredExecutors information can't be recorded both in the mode of spark's standalone and spark on k8s , which will cause the RegisteredExecutors information to be lost ,when the ExternalShuffleService is restarted.
> To solve the problem above, a method is proposed and is committed .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org