You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/22 23:58:21 UTC

[GitHub] [spark] nssalian opened a new pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

nssalian opened a new pull request #29844:
URL: https://github.com/apache/spark/pull/29844


   
   ### What changes were proposed in this pull request?
   
   Similar patch to https://github.com/apache/spark/pull/24748 but applied to the branch-2.4.
   Backporting the fix to releases 2.4.x.
   Please let me know if I missed some step; I haven't contributed to spark in a long time.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698661308






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698651112


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699152244


   Sorry guys, @erikerlandson and @nssalian . 
   
   @erikerlandson . Could you revert this commit?
   
   @nssalian . You can make a correct PR if you want SPARK-27872. Please don't piggy-back the other improvements silently. It breaks Apache Spark release policy.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698650598


   **[Test build #129096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129096/testReport)** for PR 29844 at commit [`0a47a8e`](https://github.com/apache/spark/commit/0a47a8e7adf08cda46ec07b14b9bfc67d95f479e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043404


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698627620






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642517






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698422594


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494579858



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -89,9 +95,9 @@ private[spark] object Config extends Logging {
 
   val KUBERNETES_DRIVER_SUBMIT_CHECK =
     ConfigBuilder("spark.kubernetes.submitInDriver")
-    .internal()
-    .booleanConf
-    .createWithDefault(false)
+      .internal()

Review comment:
       fixed it. ide formated.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642523


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129093/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698661303


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33717/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642508


   **[Test build #129093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129093/testReport)** for PR 29844 at commit [`cfed34c`](https://github.com/apache/spark/commit/cfed34c05a061267ef41e299e0ddae79b87261fc).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652354


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33715/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699175248


   @dongjoon-hyun @erikerlandson  I added another PR to fix these changes, easier than a revert. -> #29876


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698663592






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494580391



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
   }
 
   def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+  def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       I changed this to addServiceAccount but this differs from the accepted patch in 3.0.0
   https://github.com/apache/spark/pull/24748/files 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699152244


   Sorry guys, @erikerlandson and @nssalian . 
   
   @erikerlandson . Could you revert this commit?
   
   @nssalian . You can make a correct PR if you want SPARK-27872. Please don't piggy-backing all the other improvements silently. It breaks Apache Spark release policy.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698562513


   **[Test build #129085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129085/testReport)** for PR 29844 at commit [`1e6c509`](https://github.com/apache/spark/commit/1e6c5092555f0fb387d9328244906e96a44bf4ae).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699141429


   @erikerlandson Does this conf exist in Apache Spark 3.0.1?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612614






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425317


   **[Test build #129079 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129079/testReport)** for PR 29844 at commit [`853e1c7`](https://github.com/apache/spark/commit/853e1c734b7735d2743c3d6ccf3615339e85debd).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425341


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646283


   **[Test build #129094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129094/testReport)** for PR 29844 at commit [`46bcdff`](https://github.com/apache/spark/commit/46bcdffc5dee6e5abb5fabdaed90287d97fb7245).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698433579


   Is there a test for executor SA that is different than the driver SA?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425341






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699154543


   cc @viirya , too. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612046


   **[Test build #129091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129091/testReport)** for PR 29844 at commit [`9cb5851`](https://github.com/apache/spark/commit/9cb585124160edc3e06dc933784c41838247a0dc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642157


   **[Test build #129093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129093/testReport)** for PR 29844 at commit [`cfed34c`](https://github.com/apache/spark/commit/cfed34c05a061267ef41e299e0ddae79b87261fc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642157


   **[Test build #129093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129093/testReport)** for PR 29844 at commit [`cfed34c`](https://github.com/apache/spark/commit/cfed34c05a061267ef41e299e0ddae79b87261fc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698624015


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33712/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698659901


   **[Test build #129097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129097/testReport)** for PR 29844 at commit [`81205a5`](https://github.com/apache/spark/commit/81205a58cf53042017123c7b8d54778347ef608c).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698660054






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425341






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494418469



##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
##########
@@ -123,12 +123,12 @@ private[spark] class KubernetesSuite extends SparkFunSuite
   }
 
   protected def runSparkPiAndVerifyCompletion(
-      appResource: String = containerLocalSparkDistroExamplesJar,
-      driverPodChecker: Pod => Unit = doBasicDriverPodCheck,
-      executorPodChecker: Pod => Unit = doBasicExecutorPodCheck,
-      appArgs: Array[String] = Array.empty[String],
-      appLocator: String = appLocator,
-      isJVM: Boolean = true ): Unit = {
+                                               appResource: String = containerLocalSparkDistroExamplesJar,
+                                               driverPodChecker: Pod => Unit = doBasicDriverPodCheck,
+                                               executorPodChecker: Pod => Unit = doBasicExecutorPodCheck,
+                                               appArgs: Array[String] = Array.empty[String],
+                                               appLocator: String = appLocator,
+                                               isJVM: Boolean = true ): Unit = {

Review comment:
       @nssalian . Could you revert this wrong change?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698651116


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129096/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698658661


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33718/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698663577


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33718/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646772


   **[Test build #129094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129094/testReport)** for PR 29844 at commit [`46bcdff`](https://github.com/apache/spark/commit/46bcdffc5dee6e5abb5fabdaed90287d97fb7245).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698650598


   **[Test build #129096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129096/testReport)** for PR 29844 at commit [`0a47a8e`](https://github.com/apache/spark/commit/0a47a8e7adf08cda46ec07b14b9bfc67d95f479e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698647535


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33714/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698435928


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33700/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698424261


   **[Test build #129079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129079/testReport)** for PR 29844 at commit [`853e1c7`](https://github.com/apache/spark/commit/853e1c734b7735d2743c3d6ccf3615339e85debd).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698662928


   @erikerlandson, tests passed. Thanks for your review.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494579765



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
       .stringConf
       .createOptional
 
+  val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
+    ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
+      .doc("Service account that is used when running the executor pod." +
+        "If this parameter is not setup, the fallback logic will use the driver's service account.")

Review comment:
       agreed.

##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -89,9 +95,9 @@ private[spark] object Config extends Logging {
 
   val KUBERNETES_DRIVER_SUBMIT_CHECK =
     ConfigBuilder("spark.kubernetes.submitInDriver")
-    .internal()
-    .booleanConf
-    .createWithDefault(false)
+      .internal()

Review comment:
       fixed it. ide formated.

##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
   }
 
   def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+  def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       I changed this to addServiceAccount but this differs from the accepted patch in 3.0.0
   https://github.com/apache/spark/pull/24748/files 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652493






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698610691


   Thanks @dongjoon-hyun and @erikerlandson for the review. Let me know if you have more comments that I can address.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425352


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129079/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652663


   **[Test build #129097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129097/testReport)** for PR 29844 at commit [`81205a5`](https://github.com/apache/spark/commit/81205a58cf53042017123c7b8d54778347ef608c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698576719






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698442134






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698663592


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698422594


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494608730



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
   }
 
   def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+  def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       sorry, I forgot that this was being pulled back to 2.4 as a bug-fix. Should probably keep it consistent w/ 3.0




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699180769


   Since @erikerlandson seems to be busy, I'll revert the problematic commit first.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699107396


   @nssalian merged to branch-2.4, thanks for the patch!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698651112






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652482


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33714/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652493






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698656131






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698424261






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698576719






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698571611


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33706/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494428684



##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
##########
@@ -84,6 +84,13 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
       })
   }
 
+  test("All pods have the same service account by default", k8sTestTag) {

Review comment:
       should discuss whether desired behavior is "executor SA defaults to driver" or "defaults to none"




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652663


   **[Test build #129097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129097/testReport)** for PR 29844 at commit [`81205a5`](https://github.com/apache/spark/commit/81205a58cf53042017123c7b8d54778347ef608c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698561799


   **[Test build #129085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129085/testReport)** for PR 29844 at commit [`1e6c509`](https://github.com/apache/spark/commit/1e6c5092555f0fb387d9328244906e96a44bf4ae).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642517


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698656131






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698424261






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646784


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129094/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698561799


   **[Test build #129085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129085/testReport)** for PR 29844 at commit [`1e6c509`](https://github.com/apache/spark/commit/1e6c5092555f0fb387d9328244906e96a44bf4ae).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698651103


   **[Test build #129096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129096/testReport)** for PR 29844 at commit [`0a47a8e`](https://github.com/apache/spark/commit/0a47a8e7adf08cda46ec07b14b9bfc67d95f479e).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494609081



##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
##########
@@ -84,6 +84,13 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
       })
   }
 
+  test("All pods have the same service account by default", k8sTestTag) {

Review comment:
       ignore this, since it is being pulled back from 3.0




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699158526


   @dongjoon-hyun wasn't aware of the PR you mentioned. I can correct this. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043140


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698656746


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33717/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698610691






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494423864



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
       .stringConf
       .createOptional
 
+  val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
+    ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
+      .doc("Service account that is used when running the executor pod." +
+        "If this parameter is not setup, the fallback logic will use the driver's service account.")

Review comment:
       It would be more backward compatible to default the executor SA to "none"

##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -89,9 +95,9 @@ private[spark] object Config extends Logging {
 
   val KUBERNETES_DRIVER_SUBMIT_CHECK =
     ConfigBuilder("spark.kubernetes.submitInDriver")
-    .internal()
-    .booleanConf
-    .createWithDefault(false)
+      .internal()

Review comment:
       this is a nit, but pure formatting changes are not good PR practice.

##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
   }
 
   def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+  def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       nit: would call this operation "add service account"

##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
##########
@@ -84,6 +84,13 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
       })
   }
 
+  test("All pods have the same service account by default", k8sTestTag) {

Review comment:
       should discuss whether desired behavior is "executor SA defaults to driver" or "defaults to none"

##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
   }
 
   def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+  def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       Can the `pod` parameter here be null, or empty?

##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
   }
 
   def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+  def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       sorry, I forgot that this was being pulled back to 2.4 as a bug-fix. Should probably keep it consistent w/ 3.0

##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
##########
@@ -84,6 +84,13 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
       })
   }
 
+  test("All pods have the same service account by default", k8sTestTag) {

Review comment:
       ignore this, since it is being pulled back from 3.0




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043140






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612626


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129091/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612046


   **[Test build #129091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129091/testReport)** for PR 29844 at commit [`9cb5851`](https://github.com/apache/spark/commit/9cb585124160edc3e06dc933784c41838247a0dc).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698660054






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494579765



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
       .stringConf
       .createOptional
 
+  val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
+    ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
+      .doc("Service account that is used when running the executor pod." +
+        "If this parameter is not setup, the fallback logic will use the driver's service account.")

Review comment:
       agreed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699180039


   Thanks for swift replying, @nssalian . However, your addition PR isn't better than a clean revert in the perspective of downstreams. We had better have a clean and correct commit after reverting.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699152244


   Sorry guys, @erikerlandson and @nssalian . I'll revert this first. 
   
   @nssalian . You can make a correct PR if you want SPARK-27872. Please don't piggy-backing all the other improvements silently. It breaks Apache Spark release policy.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699185924


   Yep. It looks like that. I understand the situation, @erikerlandson . The committers had better be more careful because the contributors make this kind of mistakes sometime. BTW, I'm not against SPARK-27872. I'm against SPARK-30122 based on the PR content.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698562528






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646782


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699185924


   Yep. It looks like that. I understand the situation, @erikerlandson . The committers had better be more careful because the contributors make this kind of mistakes sometime. BTW, I'm not against SPARK-27872. I'm against for SPARK-30122 based on the PR content.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698656119


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33715/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612614


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698627613


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33712/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698627620






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699180039


   Thanks for swift replying, @nssalian . However, your addition PR isn't not better than a clean revert in the perspective of downstreams. We had better have a clean and correct commit after reverting.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043140


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698576699


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33706/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646283


   **[Test build #129094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129094/testReport)** for PR 29844 at commit [`46bcdff`](https://github.com/apache/spark/commit/46bcdffc5dee6e5abb5fabdaed90287d97fb7245).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699150690


   This merge silently creates a future regression because Apache Spark 3.0 doesn't have this conf.
   ```
   ~s:master $ git grep KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName
   resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala:    ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
   
   ~s:branch-3.0 $ git grep KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName
   
   ~s:branch-2.4 $ git grep KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName
   resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala:    ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043404


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699183582


   the changes backported correspond to https://github.com/apache/spark/pull/24748, and SPARK-27872 was tagged as a bug fix (although a credible argument could be made it was a new feature).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494418469



##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
##########
@@ -123,12 +123,12 @@ private[spark] class KubernetesSuite extends SparkFunSuite
   }
 
   protected def runSparkPiAndVerifyCompletion(
-      appResource: String = containerLocalSparkDistroExamplesJar,
-      driverPodChecker: Pod => Unit = doBasicDriverPodCheck,
-      executorPodChecker: Pod => Unit = doBasicExecutorPodCheck,
-      appArgs: Array[String] = Array.empty[String],
-      appLocator: String = appLocator,
-      isJVM: Boolean = true ): Unit = {
+                                               appResource: String = containerLocalSparkDistroExamplesJar,
+                                               driverPodChecker: Pod => Unit = doBasicDriverPodCheck,
+                                               executorPodChecker: Pod => Unit = doBasicExecutorPodCheck,
+                                               appArgs: Array[String] = Array.empty[String],
+                                               appLocator: String = appLocator,
+                                               isJVM: Boolean = true ): Unit = {

Review comment:
       @nssalian . Could you revert this wrong change?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646782






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612597


   **[Test build #129091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129091/testReport)** for PR 29844 at commit [`9cb5851`](https://github.com/apache/spark/commit/9cb585124160edc3e06dc933784c41838247a0dc).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698638579


   @nssalian this LGTM - it looks like you exceeded 100 character line length in a few places and that needs to be fixed to pass the build checks.  Once it passes the linting and testing I can merge


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698424261


   **[Test build #129079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129079/testReport)** for PR 29844 at commit [`853e1c7`](https://github.com/apache/spark/commit/853e1c734b7735d2743c3d6ccf3615339e85debd).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698663600


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33718/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698562528


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698442134






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698433579






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698562540


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129085/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494424834



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -89,9 +95,9 @@ private[spark] object Config extends Logging {
 
   val KUBERNETES_DRIVER_SUBMIT_CHECK =
     ConfigBuilder("spark.kubernetes.submitInDriver")
-    .internal()
-    .booleanConf
-    .createWithDefault(false)
+      .internal()

Review comment:
       this is a nit, but pure formatting changes are not good PR practice.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494423864



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
       .stringConf
       .createOptional
 
+  val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
+    ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
+      .doc("Service account that is used when running the executor pod." +
+        "If this parameter is not setup, the fallback logic will use the driver's service account.")

Review comment:
       It would be more backward compatible to default the executor SA to "none"




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698442108


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33700/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494429924



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
   }
 
   def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+  def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       Can the `pod` parameter here be null, or empty?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r495216161



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
       .stringConf
       .createOptional
 
+  val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =

Review comment:
       Ur, this looks wrong.
   
   Why do we backport an improvement JIRA, SPARK-30122 Allow setting serviceAccountName for executor pods, here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043140


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043404






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697849257


   @erikerlandson  , could you help review this, when you get a chance?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
erikerlandson edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699183582


   @dongjoon-hyun the changes backported correspond to https://github.com/apache/spark/pull/24748, and SPARK-27872 was tagged as a bug fix (although a credible argument could be made it was a new feature).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4

Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494425934



##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
   }
 
   def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+  def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {

Review comment:
       nit: would call this operation "add service account"




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698661308






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] erikerlandson closed pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency

Posted by GitBox <gi...@apache.org>.
erikerlandson closed pull request #29844:
URL: https://github.com/apache/spark/pull/29844


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org