[spark] branch branch-3.1 updated: [SPARK-33893][CORE] Exclude fallback block manager from executorList

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.1 by this push:
     new 2bde0bd  [SPARK-33893][CORE] Exclude fallback block manager from executorList
2bde0bd is described below

commit 2bde0bd883d0d9e11eae16bfd291b26de5d0aef8
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Wed Dec 23 15:31:56 2020 -0800

    [SPARK-33893][CORE] Exclude fallback block manager from executorList
    
    ### What changes were proposed in this pull request?
    
    This PR aims to exclude fallback block manager from `executorList` function.
    
    ### Why are the changes needed?
    
    When a fallback storage is used, the executors UI tab hangs because the executor list REST API result doesn't have `peakMemoryMetrics` of `ExecutorMetrics`. The root cause is that the block manager id used by fallback storage is included in the API result and it doesn't have `peakMemoryMetrics` because it's populated during HeartBeat reporting. We should hide it.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a bug fix on UI.
    
    ### How was this patch tested?
    
    Manual. Run the following and visit Spark `executors` tab UI with browser.
    ```
    bin/spark-shell -c spark.storage.decommission.fallbackStorage.path=file:///tmp/spark-storage/
    ```
    
    Closes #30911 from dongjoon-hyun/SPARK-33893.
    
    Authored-by: Dongjoon Hyun <dh...@apple.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
    (cherry picked from commit d467d817260d6ca605c34f493e68d0877209170f)
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 core/src/main/scala/org/apache/spark/status/AppStatusStore.scala | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
index affa85b..b9cc914 100644
--- a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
+++ b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
@@ -24,6 +24,7 @@ import scala.collection.mutable.HashMap
 
 import org.apache.spark.{JobExecutionStatus, SparkConf}
 import org.apache.spark.status.api.v1
+import org.apache.spark.storage.FallbackStorage.FALLBACK_BLOCK_MANAGER_ID
 import org.apache.spark.ui.scope._
 import org.apache.spark.util.Utils
 import org.apache.spark.util.kvstore.{InMemoryStore, KVStore}
@@ -88,7 +89,7 @@ private[spark] class AppStatusStore(
     } else {
       base
     }
-    filtered.asScala.map(_.info).toSeq
+    filtered.asScala.map(_.info).filter(_.id != FALLBACK_BLOCK_MANAGER_ID.executorId).toSeq
   }
 
   def executorSummary(executorId: String): v1.ExecutorSummary = {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org