You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/11/08 04:45:58 UTC

[jira] [Commented] (SPARK-18342) HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail

    [ https://issues.apache.org/jira/browse/SPARK-18342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15646484#comment-15646484 ] 

Apache Spark commented on SPARK-18342:
--------------------------------------

User 'brkyvz' has created a pull request for this issue:
https://github.com/apache/spark/pull/15804

> HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-18342
>                 URL: https://issues.apache.org/jira/browse/SPARK-18342
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.0.1
>            Reporter: Burak Yavuz
>            Priority: Critical
>
> The HDFSBackedStateStore renames temporary files to delta files as it commits new versions. It however doesn't check whether the rename succeeded. If the rename fails, then recovery will not be possible. It should fail during the rename stage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org