You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "L. C. Hsieh (Jira)" <ji...@apache.org> on 2020/08/09 05:46:00 UTC
[jira] [Commented] (SPARK-32571) yarnClient.killApplication(appId)
is never called
[ https://issues.apache.org/jira/browse/SPARK-32571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173750#comment-17173750 ]
L. C. Hsieh commented on SPARK-32571:
-------------------------------------
I think by design in cluster mode the spark application is supposed to continue to run, even spark-submit process is killed. This is also the most common use cases for cluster mode. If you want to be able to stop the spark application, the client mode provides the control.
> yarnClient.killApplication(appId) is never called
> -------------------------------------------------
>
> Key: SPARK-32571
> URL: https://issues.apache.org/jira/browse/SPARK-32571
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit, YARN
> Affects Versions: 2.4.0, 3.0.0
> Reporter: A Tester
> Priority: Major
>
> *Problem Statement:*
> When an application is submitted using spark-submit in cluster mode using yarn, the spark application continues to run on the cluster, even if spark-submit itself has been requested to shutdown (Ctrl-C/SIGTERM/etc.)
> While there is code inside org.apache.spark.deploy.yarn.Client.scala that would lead you to believe the spark application on the cluster will shut down, this code is not currently reachable.
> Example of behavior:
> spark-submit ...
> <Ctrl-C> or kill -15 <pid>
> spark-submit itself dies
> job can still be found running on the cluster
>
> *Expectation:*
> When spark-submit is in monitoring a yarn app and spark-submit itself is requested to shutdown (SIGTERM, HUP,etc.), it should call yarnClient.killApplication(appId) so that the actual spark application running on the cluster is killed.
>
>
> *Proposal*
> There is already a shutdown hook registered which cleans up temp files. Could this be extended to call yarnClient.killApplication?
> I believe the default behavior should be to request yarn to kill the application, however I can imagine use cases where you may still want it to run. So facilitate these use cases, an option should be provided to skip this hook.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org