You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2016/08/11 21:56:10 UTC
spark git commit: [SPARK-16868][WEB UI] Fix executor be both dead and
alive on executor ui.
Repository: spark
Updated Branches:
refs/heads/master 1c9a386c6 -> 4ec5c360c
[SPARK-16868][WEB UI] Fix executor be both dead and alive on executor ui.
## What changes were proposed in this pull request?
In a heavy pressure of the spark application, since the executor will register it to driver block manager twice(because of heart beats), the executor will show as picture show:
## How was this patch tested?
NA
Details in: [SPARK-16868](https://issues.apache.org/jira/browse/SPARK-16868)
Author: huangzhaowei <ca...@gmail.com>
Closes #14530 from SaintBacchus/SPARK-16868.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4ec5c360
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4ec5c360
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4ec5c360
Branch: refs/heads/master
Commit: 4ec5c360ce2045a9bdecb3c5277ba519bf0f44ae
Parents: 1c9a386
Author: huangzhaowei <ca...@gmail.com>
Authored: Thu Aug 11 14:56:03 2016 -0700
Committer: Marcelo Vanzin <va...@cloudera.com>
Committed: Thu Aug 11 14:56:03 2016 -0700
----------------------------------------------------------------------
.../scala/org/apache/spark/storage/StorageStatusListener.scala | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/4ec5c360/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala
index 3008520..798658a 100644
--- a/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala
+++ b/core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala
@@ -77,6 +77,10 @@ class StorageStatusListener(conf: SparkConf) extends SparkListener {
val maxMem = blockManagerAdded.maxMem
val storageStatus = new StorageStatus(blockManagerId, maxMem)
executorIdToStorageStatus(executorId) = storageStatus
+
+ // Try to remove the dead storage status if same executor register the block manager twice.
+ deadExecutorStorageStatus.zipWithIndex.find(_._1.blockManagerId.executorId == executorId)
+ .foreach(toRemoveExecutor => deadExecutorStorageStatus.remove(toRemoveExecutor._2))
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org