You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by mrm <ma...@skimlinks.com> on 2014/07/21 19:23:41 UTC

Re: LiveListenerBus throws exception and weird web UI bug

I have the same error! Did you manage to fix it?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/LiveListenerBus-throws-exception-and-weird-web-UI-bug-tp8330p10324.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

答复: LiveListenerBus throws exception and weird web UI bug

Posted by "余根茂(木艮)" <ge...@alibaba-inc.com>.

Hi all, 

         Here is my fix https://github.com/apache/spark/pull/1356, although not handsome, but work well.  Any Suggestions?

 

--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/LiveListenerBus-throws-exception-and-weird-web-UI-bug-tp8330p10324.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: LiveListenerBus throws exception and weird web UI bug

Posted by Andrew Or <an...@databricks.com>.

Hi all,

This error happens because we receive a "completed" event for a particular
stage that we don't know about, i.e. a stage we haven't received a
"submitted" event for. The root cause of this, as Baoxu explained, is
usually because the event queue is full and the listener begins to drop
events. In this case we are dropping the "submitted" event. This particular
exception should be fixed in the latest master, as we now check for whether
the key exists before indexing directly into it. Unfortunately, this is not
in Spark 1.0.1, but will be fixed in Spark 1.1. There is currently no
bullet-proof workaround for this issue, but you might try to reduce the
number of concurrently running tasks (partitions) to avoid emitting too
many events. The root cause of the listener queue taking too much time to
process events is recorded in SPARK-2316, which we also intend to fix by
Spark 1.1.

Andrew


2014-07-21 10:23 GMT-07:00 mrm <ma...@skimlinks.com>:

> I have the same error! Did you manage to fix it?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/LiveListenerBus-throws-exception-and-weird-web-UI-bug-tp8330p10324.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>