[jira] [Commented] (SPARK-2985) Buffered data in BlockGenerator gets lost when receiver crashes

["Tathagata Das (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224555#comment-14224555 ] 

Tathagata Das commented on SPARK-2985:
--------------------------------------

This is by design. If you are using the receiver.store(<single record>) API to store one record at a time, the block generator is used to divide the stream of records into blocks. While this interface is simple (user does not have to worry about blocking), it does not provide any control over when the data has been persisted reliably. Instead if you use receiver.store(<multiple records>), that function returns only when the data has been replicated (if replication is enabled, it is enabled by default). So your receiver needs to be implemented accordingly - either use simple API with no feedback, or harder API with full control.

> Buffered data in BlockGenerator gets lost when receiver crashes
> ---------------------------------------------------------------
>
>                 Key: SPARK-2985
>                 URL: https://issues.apache.org/jira/browse/SPARK-2985
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.0.0
>            Reporter: dai zhiyuan
>            Priority: Critical
>
> If recevierTracker crashes,the buffer data of BlockGenerator will be lost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org