You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Ashwin Agate (JIRA)" <ji...@apache.org> on 2018/12/11 00:23:00 UTC

[jira] [Commented] (SPARK-23530) It's not appropriate to let the original master exit while the leader of zookeeper shutdown

    [ https://issues.apache.org/jira/browse/SPARK-23530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715839#comment-16715839 ] 

Ashwin Agate commented on SPARK-23530:
--------------------------------------

Consequence is that during our software stack upgrade, Spark Master on some other box (not necessarily the box that is being upgraded) shuts down. The box where the shutdown might happen is not predictable, since we try to do software stack upgrade one box at a time to keep things running, we no other way except  but to resort to brute force restart of spark master on all the boxes to ensure that we catch any spark master after each box upgrad. 

This is really bad behavior and defeats the purpose of Spark Masters being highly available, they really are not HA in this scenario. Spark Master process really should not shutdown if zookeeper on some other box in the cluster shuts down.

> It's not appropriate to let the original master exit while the leader of zookeeper shutdown
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-23530
>                 URL: https://issues.apache.org/jira/browse/SPARK-23530
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.2.1, 2.3.0
>            Reporter: liuxianjiao
>            Priority: Major
>
> When the leader of zookeeper shutdown,the current method of spark is letting the master exit to revoke the leadership.However,this sacrifice a master node.According the treatment of hadoop and storm ,we should let the origin active master to be standby ,or Re-election for spark master,or any other ways to revoke leadership gracefully.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org