You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2016/11/03 07:23:58 UTC
[jira] [Commented] (SPARK-18225) job will miss when driver removed
by master in spark streaming
[ https://issues.apache.org/jira/browse/SPARK-18225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631915#comment-15631915 ]
Saisai Shao commented on SPARK-18225:
-------------------------------------
I simply doubt that kill button provided on the web UI does not give a clean stop for spark streaming jobs. If you want a graceful stop, it would be better to call StreamingContext#stop instead.
> job will miss when driver removed by master in spark streaming
> ---------------------------------------------------------------
>
> Key: SPARK-18225
> URL: https://issues.apache.org/jira/browse/SPARK-18225
> Project: Spark
> Issue Type: Bug
> Components: DStreams, Scheduler
> Affects Versions: 1.6.1, 1.6.2
> Reporter: liujianhui
>
> kill the application on spark ui, the master will send an ApplicationRemoved to driver, driver will abort the all pending job,and then the job finish with exception "Master removed our application:Killed",and then Jobscheduler will remove the job from jobsets, but the jobgenerator still docheckpoint without the job which removed before, and then driver stop；when recover from the check point file，it miss all jobs which aborted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org