[jira] [Created] (SPARK-31908) Spark UI shows wrong driver memory configurations if driver memory is provided at run time

["Rahul Kumar (Jira)" <ji...@apache.org>]

Rahul Kumar created SPARK-31908:
-----------------------------------

             Summary: Spark UI shows wrong driver memory configurations if driver memory is provided at run time
                 Key: SPARK-31908
                 URL: https://issues.apache.org/jira/browse/SPARK-31908
             Project: Spark
          Issue Type: Bug
          Components: Web UI
    Affects Versions: 2.4.3
            Reporter: Rahul Kumar


I discovered that in cluster mode , when driver memory is provided via spark.driver.memory configuration at run time after creating spark session, spark doesn't pick this configurations at run time as application master is already launched by that time and picks the default spark driver memory configuration(1GB). 
However, on spark UI page, in environment tab, it still shows driver memory as the value passed via configurations at run time, which makes identifying and debugging this scenario more difficult. Driver memory should be shown as the value which spark is actually using in job.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org