You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:23:28 UTC
[jira] [Updated] (SPARK-18225) job will miss when driver removed by
master in spark streaming
[ https://issues.apache.org/jira/browse/SPARK-18225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-18225:
---------------------------------
Labels: bulk-closed (was: )
> job will miss when driver removed by master in spark streaming
> ---------------------------------------------------------------
>
> Key: SPARK-18225
> URL: https://issues.apache.org/jira/browse/SPARK-18225
> Project: Spark
> Issue Type: Bug
> Components: DStreams, Scheduler
> Affects Versions: 1.6.1, 1.6.2
> Reporter: liujianhui
> Priority: Major
> Labels: bulk-closed
>
> kill the application on spark ui, the master will send an ApplicationRemoved to driver, driver will abort the all pending job,and then the job finish with exception "Master removed our application:Killed",and then Jobscheduler will remove the job from jobsets, but the jobgenerator still docheckpoint without the job which removed before, and then driver stop；when recover from the check point file，it miss all jobs which aborted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org