You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2014/09/04 11:16:51 UTC

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

    [ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121160#comment-14121160 ] 

Saisai Shao commented on SPARK-3129:
------------------------------------

Hi [~hshreedharan], one more question:

Is your design goal trying to fix the receiver node failure caused data loss issue? Seems potentially data will be lost when data is only stored in BlockGenerator not yet in BM when node is failed. Your design doc mainly focused on driver failure, so what's your thought?

> Prevent data loss in Spark Streaming
> ------------------------------------
>
>                 Key: SPARK-3129
>                 URL: https://issues.apache.org/jira/browse/SPARK-3129
>             Project: Spark
>          Issue Type: New Feature
>            Reporter: Hari Shreedharan
>            Assignee: Hari Shreedharan
>         Attachments: StreamingPreventDataLoss.pdf
>
>
> Spark Streaming can small amounts of data when the driver goes down - and the sending system cannot re-send the data (or the data has already expired on the sender side). The document attached has more details. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org