You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2016/03/01 20:19:18 UTC

[jira] [Resolved] (SPARK-10694) Prevent Data Loss in Spark Streaming when used with OFF_HEAP ExternalBlockStore (Tachyon)

     [ https://issues.apache.org/jira/browse/SPARK-10694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Rosen resolved SPARK-10694.
--------------------------------
    Resolution: Won't Fix

Due to the removal of the ExternalBlockStore API in SPARK-12667, I think that this is now "Won't Fix".

> Prevent Data Loss in Spark Streaming when used with OFF_HEAP ExternalBlockStore (Tachyon)
> -----------------------------------------------------------------------------------------
>
>                 Key: SPARK-10694
>                 URL: https://issues.apache.org/jira/browse/SPARK-10694
>             Project: Spark
>          Issue Type: Bug
>          Components: Block Manager, Streaming
>    Affects Versions: 1.5.0
>            Reporter: Dibyendu Bhattacharya
>
> If Streaming application stores the blocks OFF_HEAP, it may not need any WAL like feature to recover from Driver failure. As long as the writing of blocks to Tachyon from Streaming receiver is durable, it should be recoverable from Tachyon directly on Driver failure. 
> This can solve the issue of expensive WAL write and duplicating the blocks both in MEMORY and also WAL and also guarantee end to end No-Data-Loss channel using OFF_HEAP store.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org