You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hari Shreedharan (JIRA)" <ji...@apache.org> on 2014/09/08 20:30:30 UTC

[jira] [Updated] (SPARK-3129) Prevent data loss in Spark Streaming

     [ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hari Shreedharan updated SPARK-3129:
------------------------------------
    Attachment: SecurityFix.diff

Looks like this should be enough, correct?

Since the SecurityManager constructor seems to generate and set the secret to the credentials which then gets written out to the ContainerLaunchContext. This should make sure that a restarting AM has the same credentials.

> Prevent data loss in Spark Streaming
> ------------------------------------
>
>                 Key: SPARK-3129
>                 URL: https://issues.apache.org/jira/browse/SPARK-3129
>             Project: Spark
>          Issue Type: New Feature
>            Reporter: Hari Shreedharan
>            Assignee: Hari Shreedharan
>         Attachments: SecurityFix.diff, StreamingPreventDataLoss.pdf
>
>
> Spark Streaming can small amounts of data when the driver goes down - and the sending system cannot re-send the data (or the data has already expired on the sender side). The document attached has more details. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org