You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Burak Yavuz (JIRA)" <ji...@apache.org> on 2016/11/08 04:04:58 UTC

[jira] [Created] (SPARK-18342) HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail

Burak Yavuz created SPARK-18342:
-----------------------------------

             Summary: HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail
                 Key: SPARK-18342
                 URL: https://issues.apache.org/jira/browse/SPARK-18342
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.0.1
            Reporter: Burak Yavuz
            Priority: Critical


The HDFSBackedStateStore renames temporary files to delta files as it commits new versions. It however doesn't check whether the rename succeeded. If the rename fails, then recovery will not be possible. It should fail during the rename stage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org