[jira] [Commented] (SPARK-26288) add initRegisteredExecutorsDB in ExternalShuffleService

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-26288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711068#comment-16711068 ] 

Apache Spark commented on SPARK-26288:
--------------------------------------

User 'weixiuli' has created a pull request for this issue:
https://github.com/apache/spark/pull/23243

> add initRegisteredExecutorsDB in ExternalShuffleService
> -------------------------------------------------------
>
>                 Key: SPARK-26288
>                 URL: https://issues.apache.org/jira/browse/SPARK-26288
>             Project: Spark
>          Issue Type: New Feature
>          Components: Kubernetes, Shuffle
>    Affects Versions: 2.4.0
>            Reporter: weixiuli
>            Priority: Major
>             Fix For: 2.4.0
>
>
> As we all know that spark on Yarn uses DB to record RegisteredExecutors information, when the ExternalShuffleService restart and it can be reloaded, which will be used as well .
> While neither spark's standalone nor spark on k8s can record it's RegisteredExecutors information by db or others ,so when ExternalShuffleService restart ,which RegisteredExecutors information will be lost,it is't what we looking forward to .
> This commit add initRegisteredExecutorsDB which can be used either spark standalone or spark on k8s to record RegisteredExecutors information , when the ExternalShuffleService restart and it can be reloaded, which will be used as well .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org