(spark) branch master updated: [SPARK-46330] Loading of Spark UI blocks for a long time when HybridStore enabled

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new cf54e8f9a51b [SPARK-46330] Loading of Spark UI blocks for a long time when HybridStore enabled
cf54e8f9a51b is described below

commit cf54e8f9a51bf54e8fa3e1011ac370e46134b134
Author: zhouyifan279 <zh...@gmail.com>
AuthorDate: Wed Dec 20 16:50:38 2023 +0800

    [SPARK-46330] Loading of Spark UI blocks for a long time when HybridStore enabled
    
    ### What changes were proposed in this pull request?
    Move `LoadedAppUI` invalidate operation out of `FsHistoryProvider` synchronized block.
    
    ### Why are the changes needed?
    When closing a HybridStore of a `LoadedAppUI` with a lot of data waiting to be written to disk, loading of other Spark UIs will be blocked for a long time.
    
    See more details at https://issues.apache.org/jira/browse/SPARK-46330
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Passed existing tests.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No.
    
    Closes #44260 from zhouyifan279/SPARK-46330.
    
    Authored-by: zhouyifan279 <zh...@gmail.com>
    Signed-off-by: Kent Yao <ya...@apache.org>
---
 .../org/apache/spark/deploy/history/FsHistoryProvider.scala   | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 73fb0086b338..8f64de0847ec 100644
--- a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -926,11 +926,12 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
    * UI lifecycle.
    */
   private def invalidateUI(appId: String, attemptId: Option[String]): Unit = {
-    synchronized {
-      activeUIs.get((appId, attemptId)).foreach { ui =>
-        ui.invalidate()
-        ui.ui.store.close()
-      }
+    val uiOption = synchronized {
+      activeUIs.get((appId, attemptId))
+    }
+    uiOption.foreach { ui =>
+      ui.invalidate()
+      ui.ui.store.close()
     }
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org