[jira] [Updated] (SPARK-16417) spark 1.5.2 receiver store(single-record) with ahead log enabled makes executor crash if there is an exception when BlockGenerator storing block

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon updated SPARK-16417:
---------------------------------
    Labels: bulk-closed  (was: )

> spark 1.5.2 receiver store(single-record) with ahead log enabled makes executor crash if there is an exception when BlockGenerator storing block
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-16417
>                 URL: https://issues.apache.org/jira/browse/SPARK-16417
>             Project: Spark
>          Issue Type: Bug
>          Components: DStreams
>    Affects Versions: 1.5.2
>         Environment: spark streaming version 1.5.2.
>            Reporter: ren xing
>            Priority: Major
>              Labels: bulk-closed
>
> receiver has the store(single-record) function which actually puts the record to a buffer. One backend thread will periodically search this buffer and generate a block and store this block to spark. If enabled the ahead log, sometimes there be an exception when writing the ahead log. This exception will be caught by the backend thread. However the backend thread just print some message AND EXIT! This means there will be no consumer for the receiver inner buffered records. As time goes on, the executor will be OOM



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org