You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kan Zhang (JIRA)" <ji...@apache.org> on 2014/04/17 04:10:14 UTC

[jira] [Comment Edited] (SPARK-1475) Draining event logging queue before stopping event logger

    [ https://issues.apache.org/jira/browse/SPARK-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967228#comment-13967228 ] 

Kan Zhang edited comment on SPARK-1475 at 4/17/14 2:08 AM:
-----------------------------------------------------------

When event queue is not drained, users may observe similar issues as those reported in SPARK-1407 (when sc.stop() is not called). 

https://github.com/apache/spark/pull/366

The above PR fixes this issue. It does require applications to call sc.stop() to properly stop SparkListenerBus and event logger.


was (Author: kzhang):
When event queue is not drained, users may observe similar issues as those reported in SPARK-1407 (when sc.stop() is not called). 

https://github.com/apache/spark/pull/366

The above PR fixes this issue. It does require applications to call SparkContext.stop() to properly stop SparkListenerBus and event logger.

> Draining event logging queue before stopping event logger
> ---------------------------------------------------------
>
>                 Key: SPARK-1475
>                 URL: https://issues.apache.org/jira/browse/SPARK-1475
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>            Priority: Blocker
>             Fix For: 1.0.0
>
>
> When stopping SparkListenerBus, its event queue needs to be drained. And this needs to happen before event logger is stopped. Otherwise, any event still waiting to be processed in the queue may be lost and consequently event log file may be incomplete. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)