[spark] branch master updated: [SPARK-35013][CORE] Don't allow to set spark.driver.cores=0

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new b3c916e  [SPARK-35013][CORE] Don't allow to set spark.driver.cores=0
b3c916e is described below

commit b3c916e5a58cc6993aa41928757d2d983b37ee8b
Author: shahid <sh...@gmail.com>
AuthorDate: Wed May 12 12:45:55 2021 -0700

    [SPARK-35013][CORE] Don't allow to set spark.driver.cores=0
    
    ### What changes were proposed in this pull request?
    Currently spark is not allowing to set spark.driver.memory, spark.executor.cores, spark.executor.memory to 0, but allowing driver cores to 0. This PR checks for driver core size as well. Thanks Oleg Lypkan for finding this.
    
    ### Why are the changes needed?
    To make the configuration check consistent.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Manual testing
    
    Closes #32504 from shahidki31/shahid/drivercore.
    
    Lead-authored-by: shahid <sh...@gmail.com>
    Co-authored-by: Hyukjin Kwon <gu...@gmail.com>
    Co-authored-by: Shahid <sh...@gmail.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
index 9da1a73..692e7ea 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
@@ -253,6 +253,9 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
         && Try(JavaUtils.byteStringAsBytes(executorMemory)).getOrElse(-1L) <= 0) {
       error("Executor memory must be a positive number")
     }
+    if (driverCores != null && Try(driverCores.toInt).getOrElse(-1) <= 0) {
+      error("Driver cores must be a positive number")
+    }
     if (executorCores != null && Try(executorCores.toInt).getOrElse(-1) <= 0) {
       error("Executor cores must be a positive number")
     }

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org