[GitHub] [spark] panbingkun commented on a diff in pull request #36273: [SPARK-38960][CORE]Spark should fail fast if initial memory too large…

[GitBox <gi...@apache.org>]

panbingkun commented on code in PR #36273:
URL: https://github.com/apache/spark/pull/36273#discussion_r853849065


##########
core/src/main/scala/org/apache/spark/SparkConf.scala:
##########
@@ -529,6 +529,17 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging with Seria
           s"(was '$javaOpts'). Use spark.executor.memory instead."
         throw new Exception(msg)
       }
+      if (javaOpts.contains("-Xms") && contains(EXECUTOR_MEMORY)) {

Review Comment:
   > `EXECUTOR_MEMORY` has default value, so I don't think `contains(EXECUTOR_MEMORY)` necessary condition
   
   in SparkContext Initialization process，code as follow:
   [    _conf = config.clone()
       _conf.validateSettings()
       ...
       ...](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L390-L393)
   
      [     _executorMemory = _conf.getOption(EXECUTOR_MEMORY.key)
         .orElse(Option(System.getenv("SPARK_EXECUTOR_MEMORY")))
         .orElse(Option(System.getenv("SPARK_MEM"))
         .map(warnSparkMem))
         .map(Utils.memoryStringToMb)
         .getOrElse(1024)
         ....](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L527-L532)
   
   The patch will throw exception when set spark.executor.extraJavaOptions & spark.executor.memory by manual.
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org