[jira] [Created] (SPARK-10694) Prevent Data Loss in Spark Streaming when used with OFF_HEAP ExternalBlockStore (Tachyon)

["Dibyendu Bhattacharya (JIRA)" <ji...@apache.org>]

Dibyendu Bhattacharya created SPARK-10694:
---------------------------------------------

             Summary: Prevent Data Loss in Spark Streaming when used with OFF_HEAP ExternalBlockStore (Tachyon)
                 Key: SPARK-10694
                 URL: https://issues.apache.org/jira/browse/SPARK-10694
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.5.0
            Reporter: Dibyendu Bhattacharya


If Streaming application stores the blocks OFF_HEAP, it may not need any WAL like feature to recover from Driver failure. As long as the writing of blocks to Tachyon from Streaming receiver is durable, it should be recoverable from Tachyon directly on Driver failure. 
This can solve the issue of expensive WAL write and duplicating the blocks both in MEMORY and also WAL and also guarantee end to end No-Data-Loss channel using OFF_HEAP store.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org