You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Rohit Agarwal (JIRA)" <ji...@apache.org> on 2015/08/11 02:44:46 UTC
[jira] [Created] (SPARK-9806) Don't share ReplayListenerBus between
multiple applications
Rohit Agarwal created SPARK-9806:
------------------------------------
Summary: Don't share ReplayListenerBus between multiple applications
Key: SPARK-9806
URL: https://issues.apache.org/jira/browse/SPARK-9806
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 1.4.0
Reporter: Rohit Agarwal
Priority: Minor
Currently, we are sharing {{ReplayListenerBus}} for replaying the event logs of various apps.
https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L226
While replaying the event log for a particular app, we add an {{ApplicationEventListener}} to the bus.
https://github.com/apache/spark/blob/v1.4.0/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L391
But we never remove it.
This result in something like the following:
{code}
15/08/11 00:04:00 log-replay-executor INFO FsHistoryProvider: Replaying log path: hdfs://localhost:9000/spark-history/application_1438993108319_0146_1.snappy
15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:01 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Elapsed time: 39.730114407s
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Application log application_1438993108319_0146_1.snappy loaded successfully.
15/08/11 00:04:40 log-replay-executor INFO FsHistoryProvider: Replaying log path: hdfs://localhost:9000/spark-history/application_1438993108319_0126_1.snappy
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:04:40 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Elapsed time: 20.483128154s
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Application log application_1438993108319_0126_1.snappy loaded successfully.
15/08/11 00:05:00 log-replay-executor INFO FsHistoryProvider: Replaying log path: hdfs://localhost:9000/spark-history/application_1438993108319_0116_1.snappy
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onEnvUpdate
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:05:00 log-replay-executor INFO ApplicationEventListener: onApplicationStart
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO ApplicationEventListener: onApplicationEnd
15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Elapsed time: 29.110070845s
15/08/11 00:05:29 log-replay-executor INFO FsHistoryProvider: Application log application_1438993108319_0116_1.snappy loaded successfully.
{code}
We should either remove the listener from the bus or create a new bus for each app.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org