You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:13:39 UTC
[jira] [Resolved] (SPARK-18994) worker clean up app directory block
the heartbeat sending
[ https://issues.apache.org/jira/browse/SPARK-18994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-18994.
----------------------------------
Resolution: Incomplete
> worker clean up app directory block the heartbeat sending
> ----------------------------------------------------------
>
> Key: SPARK-18994
> URL: https://issues.apache.org/jira/browse/SPARK-18994
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 1.6.1
> Reporter: liujianhui
> Priority: Major
> Labels: bulk-closed
>
> h2. scene
> # when an application stop, it will trigger master send ApplicationFinish event to worker
> # worker will clean up the directories for the application, but it takes a long time to delete them, as a result of the worker endpoint is Thread safe, and then it block the heartbeat sending to master
> # master find the heartbeat expired for that worker, and then remove the worker, it will re-launching the driver belong to that worker, actually that driver doesn't exit, finally this will exist two app with same driver-id
> the worker log
> {code}
> 2016-12-23 11:39:33,033 |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker | Executor app-20161208222600-0020/200 finished with state KILLED exitStatus 137 | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:39:33,033 |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker | Cleaning up local directories for application app-20161208222600-0020 | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 | org.apache.spark.network.shuffle.ExternalShuffleBlockResolver | Application app-20161208222600-0020 removed, cleanupLocalDirs = true | org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.applicationRemoved(ExternalShuffleBlockResolver.java:206)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker | Master with url spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-2 | org.apache.spark.deploy.worker.Worker | Connecting to master bigdata-sirius-ser02.gz01:8090... | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-3 | org.apache.spark.deploy.worker.Worker | Connecting to master bigdata-sirius-ser03.gz01:8090... | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker | Master with url spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | shuffle-client-1 | org.apache.spark.deploy.worker.Worker | Successfully registered with master spark://bigdata-sirius-ser02.gz01:8090 | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | shuffle-client-1 | org.apache.spark.deploy.worker.Worker | Worker cleanup enabled; old application directories will be deleted in: /data1/spark/work | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker | Not spawning another attempt to register with the master, since there is an attempt scheduled already. | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker | Master with url spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker | Master with url spark://bigdata-sirius-ser02.gz01:8090 requested this worker to reconnect. | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-3 | org.apache.spark.deploy.worker.Worker | Connecting to master bigdata-sirius-ser03.gz01:8090... | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | dispatcher-event-loop-26 | org.apache.spark.deploy.worker.Worker | Not spawning another attempt to register with the master, since there is an attempt scheduled already. | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> 2016-12-23 11:45:21,021 |INFO | worker-register-master-threadpool-2 | org.apache.spark.deploy.worker.Worker | Connecting to master bigdata-sirius-ser02.gz01:8090... | org.apache.spark.Logging$class.logInfo(Logging.scala:58)
> {code}
> *master* log
> {code}
> 2016-12-23 11:40:46,046 |WARN | dispatcher-event-loop-20 | org.apache.spark.deploy.master.Master | Removing worker-20161208233029-100.69.146.57-26874 because we got no heartbeat in 60 seconds | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> 2016-12-23 11:45:21,021 |WARN | dispatcher-event-loop-0 | org.apache.spark.deploy.master.Master | Got heartbeat from unregistered worker worker-20161208233029-100.69.146.57-26874. Asking it to re-register. | org.apache.spark.Logging$class.logWarning(Logging.scala:70)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org