You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Tathagata Das (JIRA)" <ji...@apache.org> on 2015/10/28 00:02:27 UTC

[jira] [Resolved] (SPARK-11324) Flag to close Write Ahead Log after writing

     [ https://issues.apache.org/jira/browse/SPARK-11324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tathagata Das resolved SPARK-11324.
-----------------------------------
       Resolution: Fixed
         Assignee: Burak Yavuz
    Fix Version/s: 1.6.0

> Flag to close Write Ahead Log after writing
> -------------------------------------------
>
>                 Key: SPARK-11324
>                 URL: https://issues.apache.org/jira/browse/SPARK-11324
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Burak Yavuz
>            Assignee: Burak Yavuz
>             Fix For: 1.6.0
>
>
> Currently the Write Ahead Log in Spark Streaming flushes data as writes need to be made. S3 does not support flushing of data, data is written once the stream is actually closed. 
> In case of failure, the data for the last minute (default rolling interval) will not be properly written. Therefore we need a flag to close the stream after the write, so that we achieve read after write consistency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org