[jira] [Updated] (SPARK-1647) Prevent data loss when Streaming driver goes down

["Tathagata Das (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tathagata Das updated SPARK-1647:
---------------------------------

    Fix Version/s: 1.1.0

> Prevent data loss when Streaming driver goes down
> -------------------------------------------------
>
>                 Key: SPARK-1647
>                 URL: https://issues.apache.org/jira/browse/SPARK-1647
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>            Reporter: Hari Shreedharan
>             Fix For: 1.1.0
>
>
> Currently when the driver goes down, any uncheckpointed data is lost from within spark. If the system from which messages are pulled can  replay messages, the data may be available - but for some systems, like Flume this is not the case. 
> Also, all windowing information is lost for windowing functions. 
> We must persist raw data somehow, and be able to replay this data if required. We also must persist windowing information with the data itself.
> This will likely require quite a bit of work to complete and probably will have to be split into several sub-jiras.



--
This message was sent by Atlassian JIRA
(v6.2#6252)