You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/12/15 02:57:59 UTC
[jira] [Assigned] (SPARK-18838) High latency of event processing
for large jobs
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-18838:
------------------------------------
Assignee: (was: Apache Spark)
> High latency of event processing for large jobs
> -----------------------------------------------
>
> Key: SPARK-18838
> URL: https://issues.apache.org/jira/browse/SPARK-18838
> Project: Spark
> Issue Type: Improvement
> Affects Versions: 2.0.0
> Reporter: Sital Kedia
>
> Currently we are observing the issue of very high event processing delay in driver's `ListenerBus` for large jobs with many tasks. Many critical component of the scheduler like `ExecutorAllocationManager`, `HeartbeatReceiver` depend on the `ListenerBus` events and these delay is causing job failure. For example, a significant delay in receiving the `SparkListenerTaskStart` might cause `ExecutorAllocationManager` manager to remove an executor which is not idle. The event processor in `ListenerBus` is a single thread which loops through all the Listeners for each event and processes each event synchronously https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala#L94.
>
> The single threaded processor often becomes the bottleneck for large jobs. In addition to that, if one of the Listener is very slow, all the listeners will pay the price of delay incurred by the slow listener.
> To solve the above problems, we plan to have a per listener single threaded executor service and separate event queue. That way we are not bottlenecked by the single threaded event processor and also critical listeners will not be penalized by the slow listeners. The downside of this approach is separate event queue per listener will increase the driver memory footprint.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org