You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Alexandre CLEMENT (Jira)" <ji...@apache.org> on 2021/05/06 09:05:00 UTC
[jira] [Commented] (SPARK-32007) Spark Driver Supervise does not
work reliably
[ https://issues.apache.org/jira/browse/SPARK-32007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340103#comment-17340103 ]
Alexandre CLEMENT commented on SPARK-32007:
-------------------------------------------
{{[~EveLiao] Here are the logs after I manually kill an EC2 instance of the cluster.}}
{{ 1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:23,420 INFO master.Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:23,420 INFO master.Master: Launching driver driver-20210506085223-0195 on worker worker-20210506084657-10.0.16.127-42291}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:26,891 INFO master.Master: Registering app ml4ra-sopra2019 }}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:26,891 INFO master.Master: Registered app ml4ra-sopra2019 with ID app-20210506085226-0192}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:27,813 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 0.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:36,716 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 1.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:36,716 INFO master.Master: Launching executor app-20210506085226-0192/0 on worker worker-20210506084657-10.0.16.125-45495}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:43,541 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 0.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:44,648 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 2.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:44,648 INFO master.Master: Launching executor app-20210506085226-0192/1 on worker worker-20210506084657-10.0.16.127-42291}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:45,654 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 3.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:45,654 INFO master.Master: Launching executor app-20210506085226-0192/2 on worker worker-20210506084657-10.0.16.125-45495}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:46,660 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 4.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:51,975 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 3.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:53,783 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 2.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:57,796 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 1.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:52:59,503 INFO master.Master: Application app-20210506085226-0192 requested to set total executors to 0.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:53:24,922 INFO master.Master: Received unregister request from application app-20210506085226-0192}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:53:24,922 INFO master.Master: Removing app app-20210506085226-0192}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:53:25,026 INFO master.Master: 10.0.16.127:56846 got disassociated, removing it.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:53:25,026 INFO master.Master: 0b1cb263dc91:37659 got disassociated, removing it.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:53:25,479 INFO master.Master: Removing driver: driver-20210506085223-0195}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:53:25,737 INFO master.Master: 10.0.16.130:43694 got disassociated, removing it.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:53:25,737 INFO master.Master: 10.0.16.125:45495 got disassociated, removing it.}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:53:25,737 INFO master.Master: Removing worker worker-20210506084657-10.0.16.125-45495 on 10.0.16.125:45495}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:53:25,737 INFO master.Master: Telling app of lost worker: worker-20210506084657-10.0.16.125-45495}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:54:23,305 WARN master.Master: Removing worker-20210506084657-10.0.16.127-42291 because we got no heartbeat in 60 seconds}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:54:23,305 INFO master.Master: Removing worker worker-20210506084657-10.0.16.127-42291 on 10.0.16.127:42291}}
{{spark_master.1.uf5h6wl5y4b0@spark-dev-master | 2021-05-06 08:54:23,305 INFO master.Master: Telling app of lost worker: worker-20210506084657-10.0.16.127-42291}}
> Spark Driver Supervise does not work reliably
> ---------------------------------------------
>
> Key: SPARK-32007
> URL: https://issues.apache.org/jira/browse/SPARK-32007
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.4
> Environment: |Java Version|1.8.0_121 (Oracle Corporation)|
> |Java Home|/usr/java/jdk1.8.0_121/jre|
> |Scala Version|version 2.11.12|
> |OS|Amazon Linux|
> h4.
> Reporter: Suraj Sharma
> Priority: Major
>
> I have a standalone cluster setup. I DO NOT have a streaming use case. I use AWS EC2 machines to have spark master and worker processes.
> *Problem*: If a spark worker machine running some drivers and executor dies, then the driver is not spawned again on other healthy machines.
> *Below are my findings:*
> ||Action/Behaviour||Executor||Driver||
> |Worker Machine Stop|Relaunches on an active machine|NO Relaunch|
> |kill -9 to process|Relaunches on other machines|Relaunches on other machines|
> |kill to process|Relaunches on other machines|Relaunches on other machines|
> *Cluster Setup:*
> # I have a spark standalone cluster
> # {{spark.driver.supervise=true}}
> # Spark Master HA is enabled and is backed by zookeeper
> # Spark version = 2.4.4
> # I am using a systemd script for the spark worker process
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org