You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/11/13 23:49:11 UTC

[jira] [Assigned] (SPARK-11740) Fix DStream checkpointing logic to prevent failures during checkpoint recovery

     [ https://issues.apache.org/jira/browse/SPARK-11740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-11740:
------------------------------------

    Assignee:     (was: Apache Spark)

> Fix DStream checkpointing logic to prevent failures during checkpoint recovery
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-11740
>                 URL: https://issues.apache.org/jira/browse/SPARK-11740
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>            Reporter: Shixiong Zhu
>
> We will do checkpoint when generating a batch and completing a batch. When the processing time of a batch is greater than the batch interval, checkpointing for completing an old batch may run after checkpointing of a new batch. If this happens, checkpoint of an old batch actually has the latest information, but we won't recovery from it. Then we may see some RDD checkpoint file missing exception during checkpoint recovery. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org