[spark] branch master updated: [SPARK-31621][CORE] Fixing Spark Master UI Issue when application is waiting for workers to launch driver

[tg...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

tgraves pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new c71198a  [SPARK-31621][CORE] Fixing Spark Master UI Issue when application is waiting for workers to launch driver
c71198a is described below

commit c71198ab6c8c9ded6a52eb97859b39dc2119b5fd
Author: Akshat Bordia <ak...@gmail.com>
AuthorDate: Tue May 5 08:58:37 2020 -0500

    [SPARK-31621][CORE] Fixing Spark Master UI Issue when application is waiting for workers to launch driver
    
    ### What changes were proposed in this pull request?
    Fixing an issue where Spark Master UI Fails to load if the application is waiting for workers to launch driver.
    
    **Root Cause:**
    This is happening due to the fact that the submitted application is waiting for a worker to be free to run the driver. Due to this resource is set to null in the formatResourcesAddresses method and this is running into null pointer exception.
    ![image](https://user-images.githubusercontent.com/31816865/80801557-77ee9300-8bca-11ea-92b7-b8df58b68de3.png)
    
    **Fix:**
    Added a null check before forming a resource address and display "None" if the driver isn't launched yet.
    
    ### Why are the changes needed?
    
    Spark Master UI should load as expected when applications are waiting for workers to run driver.
    
    ### Does this PR introduce _any_ user-facing change?
    The worker column in Spark Master UI will show "None" if the driver hasn't been launched yet.
    ![image](https://user-images.githubusercontent.com/31816865/80801671-be43f200-8bca-11ea-86c3-381925f82cc7.png)
    
    ### How was this patch tested?
    Tested on a local setup. Launched 2 applications and ensured that Spark Master UI loads fine.
    ![image](https://user-images.githubusercontent.com/31816865/80801883-5b9f2600-8bcb-11ea-8a1a-cc597aabc4c2.png)
    
    Closes #28429 from akshatb1/MasterUIBug.
    
    Authored-by: Akshat Bordia <ak...@gmail.com>
    Signed-off-by: Thomas Graves <tg...@apache.org>
---
 core/src/main/scala/org/apache/spark/deploy/master/DriverInfo.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/core/src/main/scala/org/apache/spark/deploy/master/DriverInfo.scala b/core/src/main/scala/org/apache/spark/deploy/master/DriverInfo.scala
index bf68ba8..252e704 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/DriverInfo.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/DriverInfo.scala
@@ -37,7 +37,7 @@ private[deploy] class DriverInfo(
   @transient var worker: Option[WorkerInfo] = None
   // resources(e.f. gpu/fpga) allocated to this driver
   // map from resource name to ResourceInformation
-  private var _resources: Map[String, ResourceInformation] = _
+  private var _resources: Map[String, ResourceInformation] = Map.empty
 
   init()
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org