[GitHub] [incubator-livy] lukatera commented on a change in pull request #167: [LIVY-588][WIP]: Full support for Spark on Kubernetes

[GitBox <gi...@apache.org>]

lukatera commented on a change in pull request #167: [LIVY-588][WIP]: Full support for Spark on Kubernetes
URL: https://github.com/apache/incubator-livy/pull/167#discussion_r287238491
 
 

 ##########
 File path: server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala
 ##########
 @@ -86,14 +86,25 @@ object InteractiveSession extends Logging {
       val builderProperties = prepareBuilderProp(conf, request.kind, livyConf)
 
       val userOpts: Map[String, Option[String]] = Map(
-        "spark.driver.cores" -> request.driverCores.map(_.toString),
         SparkLauncher.DRIVER_MEMORY -> request.driverMemory.map(_.toString),
-        SparkLauncher.EXECUTOR_CORES -> request.executorCores.map(_.toString),
         SparkLauncher.EXECUTOR_MEMORY -> request.executorMemory.map(_.toString),
         "spark.executor.instances" -> request.numExecutors.map(_.toString),
-        "spark.app.name" -> request.name.map(_.toString),
-        "spark.yarn.queue" -> request.queue
-      )
+        "spark.app.name" -> request.name.map(_.toString)
+      ) ++ {
+        if (livyConf.isRunningOnKubernetes()) {
+          Map(
+            "spark.kubernetes.executor.request.cores" -> request.executorCores.map(_.toString),
 
 Review comment:
   i think this block of extra kubernetes parameters is not required. Spark will allocate resources just fine on Kubernetes with options non-specific to kubernetes. Moreover, user would sometime want to tune these kubernetes specific options if they know they're running on Kubernetes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services