You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/22 23:58:21 UTC
[GitHub] [spark] nssalian opened a new pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
nssalian opened a new pull request #29844:
URL: https://github.com/apache/spark/pull/29844
### What changes were proposed in this pull request?
Similar patch to https://github.com/apache/spark/pull/24748 but applied to the branch-2.4.
Backporting the fix to releases 2.4.x.
Please let me know if I missed some step; I haven't contributed to spark in a long time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698661308
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698651112
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699152244
Sorry guys, @erikerlandson and @nssalian .
@erikerlandson . Could you revert this commit?
@nssalian . You can make a correct PR if you want SPARK-27872. Please don't piggy-back the other improvements silently. It breaks Apache Spark release policy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698650598
**[Test build #129096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129096/testReport)** for PR 29844 at commit [`0a47a8e`](https://github.com/apache/spark/commit/0a47a8e7adf08cda46ec07b14b9bfc67d95f479e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043404
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698627620
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642517
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698422594
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494579858
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -89,9 +95,9 @@ private[spark] object Config extends Logging {
val KUBERNETES_DRIVER_SUBMIT_CHECK =
ConfigBuilder("spark.kubernetes.submitInDriver")
- .internal()
- .booleanConf
- .createWithDefault(false)
+ .internal()
Review comment:
fixed it. ide formated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642523
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129093/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698661303
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33717/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642508
**[Test build #129093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129093/testReport)** for PR 29844 at commit [`cfed34c`](https://github.com/apache/spark/commit/cfed34c05a061267ef41e299e0ddae79b87261fc).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652354
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33715/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699175248
@dongjoon-hyun @erikerlandson I added another PR to fix these changes, easier than a revert. -> #29876
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698663592
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494580391
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
}
def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+ def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {
Review comment:
I changed this to addServiceAccount but this differs from the accepted patch in 3.0.0
https://github.com/apache/spark/pull/24748/files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699152244
Sorry guys, @erikerlandson and @nssalian .
@erikerlandson . Could you revert this commit?
@nssalian . You can make a correct PR if you want SPARK-27872. Please don't piggy-backing all the other improvements silently. It breaks Apache Spark release policy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698562513
**[Test build #129085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129085/testReport)** for PR 29844 at commit [`1e6c509`](https://github.com/apache/spark/commit/1e6c5092555f0fb387d9328244906e96a44bf4ae).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699141429
@erikerlandson Does this conf exist in Apache Spark 3.0.1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612614
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425317
**[Test build #129079 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129079/testReport)** for PR 29844 at commit [`853e1c7`](https://github.com/apache/spark/commit/853e1c734b7735d2743c3d6ccf3615339e85debd).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425341
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646283
**[Test build #129094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129094/testReport)** for PR 29844 at commit [`46bcdff`](https://github.com/apache/spark/commit/46bcdffc5dee6e5abb5fabdaed90287d97fb7245).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698433579
Is there a test for executor SA that is different than the driver SA?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425341
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699154543
cc @viirya , too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612046
**[Test build #129091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129091/testReport)** for PR 29844 at commit [`9cb5851`](https://github.com/apache/spark/commit/9cb585124160edc3e06dc933784c41838247a0dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642157
**[Test build #129093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129093/testReport)** for PR 29844 at commit [`cfed34c`](https://github.com/apache/spark/commit/cfed34c05a061267ef41e299e0ddae79b87261fc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642157
**[Test build #129093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129093/testReport)** for PR 29844 at commit [`cfed34c`](https://github.com/apache/spark/commit/cfed34c05a061267ef41e299e0ddae79b87261fc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698624015
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33712/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698659901
**[Test build #129097 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129097/testReport)** for PR 29844 at commit [`81205a5`](https://github.com/apache/spark/commit/81205a58cf53042017123c7b8d54778347ef608c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698660054
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425341
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494418469
##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
##########
@@ -123,12 +123,12 @@ private[spark] class KubernetesSuite extends SparkFunSuite
}
protected def runSparkPiAndVerifyCompletion(
- appResource: String = containerLocalSparkDistroExamplesJar,
- driverPodChecker: Pod => Unit = doBasicDriverPodCheck,
- executorPodChecker: Pod => Unit = doBasicExecutorPodCheck,
- appArgs: Array[String] = Array.empty[String],
- appLocator: String = appLocator,
- isJVM: Boolean = true ): Unit = {
+ appResource: String = containerLocalSparkDistroExamplesJar,
+ driverPodChecker: Pod => Unit = doBasicDriverPodCheck,
+ executorPodChecker: Pod => Unit = doBasicExecutorPodCheck,
+ appArgs: Array[String] = Array.empty[String],
+ appLocator: String = appLocator,
+ isJVM: Boolean = true ): Unit = {
Review comment:
@nssalian . Could you revert this wrong change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698651116
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129096/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698658661
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33718/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698663577
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33718/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646772
**[Test build #129094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129094/testReport)** for PR 29844 at commit [`46bcdff`](https://github.com/apache/spark/commit/46bcdffc5dee6e5abb5fabdaed90287d97fb7245).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698650598
**[Test build #129096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129096/testReport)** for PR 29844 at commit [`0a47a8e`](https://github.com/apache/spark/commit/0a47a8e7adf08cda46ec07b14b9bfc67d95f479e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698647535
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33714/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698435928
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33700/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698424261
**[Test build #129079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129079/testReport)** for PR 29844 at commit [`853e1c7`](https://github.com/apache/spark/commit/853e1c734b7735d2743c3d6ccf3615339e85debd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698662928
@erikerlandson, tests passed. Thanks for your review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494579765
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
.stringConf
.createOptional
+ val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
+ ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
+ .doc("Service account that is used when running the executor pod." +
+ "If this parameter is not setup, the fallback logic will use the driver's service account.")
Review comment:
agreed.
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -89,9 +95,9 @@ private[spark] object Config extends Logging {
val KUBERNETES_DRIVER_SUBMIT_CHECK =
ConfigBuilder("spark.kubernetes.submitInDriver")
- .internal()
- .booleanConf
- .createWithDefault(false)
+ .internal()
Review comment:
fixed it. ide formated.
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
}
def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+ def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {
Review comment:
I changed this to addServiceAccount but this differs from the accepted patch in 3.0.0
https://github.com/apache/spark/pull/24748/files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652493
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698610691
Thanks @dongjoon-hyun and @erikerlandson for the review. Let me know if you have more comments that I can address.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698425352
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129079/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652663
**[Test build #129097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129097/testReport)** for PR 29844 at commit [`81205a5`](https://github.com/apache/spark/commit/81205a58cf53042017123c7b8d54778347ef608c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698576719
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698442134
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698663592
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698422594
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494608730
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
}
def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+ def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {
Review comment:
sorry, I forgot that this was being pulled back to 2.4 as a bug-fix. Should probably keep it consistent w/ 3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699180769
Since @erikerlandson seems to be busy, I'll revert the problematic commit first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699107396
@nssalian merged to branch-2.4, thanks for the patch!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698651112
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652482
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33714/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652493
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698656131
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698424261
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698576719
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698571611
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33706/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494428684
##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
##########
@@ -84,6 +84,13 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
})
}
+ test("All pods have the same service account by default", k8sTestTag) {
Review comment:
should discuss whether desired behavior is "executor SA defaults to driver" or "defaults to none"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698652663
**[Test build #129097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129097/testReport)** for PR 29844 at commit [`81205a5`](https://github.com/apache/spark/commit/81205a58cf53042017123c7b8d54778347ef608c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698561799
**[Test build #129085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129085/testReport)** for PR 29844 at commit [`1e6c509`](https://github.com/apache/spark/commit/1e6c5092555f0fb387d9328244906e96a44bf4ae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698642517
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698656131
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698424261
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646784
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129094/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698561799
**[Test build #129085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129085/testReport)** for PR 29844 at commit [`1e6c509`](https://github.com/apache/spark/commit/1e6c5092555f0fb387d9328244906e96a44bf4ae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698651103
**[Test build #129096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129096/testReport)** for PR 29844 at commit [`0a47a8e`](https://github.com/apache/spark/commit/0a47a8e7adf08cda46ec07b14b9bfc67d95f479e).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494609081
##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
##########
@@ -84,6 +84,13 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
})
}
+ test("All pods have the same service account by default", k8sTestTag) {
Review comment:
ignore this, since it is being pulled back from 3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699158526
@dongjoon-hyun wasn't aware of the PR you mentioned. I can correct this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043140
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698656746
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33717/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698610691
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494423864
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
.stringConf
.createOptional
+ val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
+ ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
+ .doc("Service account that is used when running the executor pod." +
+ "If this parameter is not setup, the fallback logic will use the driver's service account.")
Review comment:
It would be more backward compatible to default the executor SA to "none"
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -89,9 +95,9 @@ private[spark] object Config extends Logging {
val KUBERNETES_DRIVER_SUBMIT_CHECK =
ConfigBuilder("spark.kubernetes.submitInDriver")
- .internal()
- .booleanConf
- .createWithDefault(false)
+ .internal()
Review comment:
this is a nit, but pure formatting changes are not good PR practice.
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
}
def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+ def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {
Review comment:
nit: would call this operation "add service account"
##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
##########
@@ -84,6 +84,13 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
})
}
+ test("All pods have the same service account by default", k8sTestTag) {
Review comment:
should discuss whether desired behavior is "executor SA defaults to driver" or "defaults to none"
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
}
def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+ def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {
Review comment:
Can the `pod` parameter here be null, or empty?
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
}
def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+ def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {
Review comment:
sorry, I forgot that this was being pulled back to 2.4 as a bug-fix. Should probably keep it consistent w/ 3.0
##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
##########
@@ -84,6 +84,13 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
})
}
+ test("All pods have the same service account by default", k8sTestTag) {
Review comment:
ignore this, since it is being pulled back from 3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043140
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612626
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129091/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612046
**[Test build #129091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129091/testReport)** for PR 29844 at commit [`9cb5851`](https://github.com/apache/spark/commit/9cb585124160edc3e06dc933784c41838247a0dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698660054
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
nssalian commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494579765
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
.stringConf
.createOptional
+ val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
+ ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
+ .doc("Service account that is used when running the executor pod." +
+ "If this parameter is not setup, the fallback logic will use the driver's service account.")
Review comment:
agreed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699180039
Thanks for swift replying, @nssalian . However, your addition PR isn't better than a clean revert in the perspective of downstreams. We had better have a clean and correct commit after reverting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699152244
Sorry guys, @erikerlandson and @nssalian . I'll revert this first.
@nssalian . You can make a correct PR if you want SPARK-27872. Please don't piggy-backing all the other improvements silently. It breaks Apache Spark release policy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699185924
Yep. It looks like that. I understand the situation, @erikerlandson . The committers had better be more careful because the contributors make this kind of mistakes sometime. BTW, I'm not against SPARK-27872. I'm against SPARK-30122 based on the PR content.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698562528
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646782
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699185924
Yep. It looks like that. I understand the situation, @erikerlandson . The committers had better be more careful because the contributors make this kind of mistakes sometime. BTW, I'm not against SPARK-27872. I'm against for SPARK-30122 based on the PR content.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698656119
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33715/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612614
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698627613
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33712/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698627620
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699180039
Thanks for swift replying, @nssalian . However, your addition PR isn't not better than a clean revert in the perspective of downstreams. We had better have a clean and correct commit after reverting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043140
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698576699
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33706/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646283
**[Test build #129094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129094/testReport)** for PR 29844 at commit [`46bcdff`](https://github.com/apache/spark/commit/46bcdffc5dee6e5abb5fabdaed90287d97fb7245).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699150690
This merge silently creates a future regression because Apache Spark 3.0 doesn't have this conf.
```
~s:master $ git grep KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala: ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
~s:branch-3.0 $ git grep KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName
~s:branch-2.4 $ git grep KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala: ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043404
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699183582
the changes backported correspond to https://github.com/apache/spark/pull/24748, and SPARK-27872 was tagged as a bug fix (although a credible argument could be made it was a new feature).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494418469
##########
File path: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
##########
@@ -123,12 +123,12 @@ private[spark] class KubernetesSuite extends SparkFunSuite
}
protected def runSparkPiAndVerifyCompletion(
- appResource: String = containerLocalSparkDistroExamplesJar,
- driverPodChecker: Pod => Unit = doBasicDriverPodCheck,
- executorPodChecker: Pod => Unit = doBasicExecutorPodCheck,
- appArgs: Array[String] = Array.empty[String],
- appLocator: String = appLocator,
- isJVM: Boolean = true ): Unit = {
+ appResource: String = containerLocalSparkDistroExamplesJar,
+ driverPodChecker: Pod => Unit = doBasicDriverPodCheck,
+ executorPodChecker: Pod => Unit = doBasicExecutorPodCheck,
+ appArgs: Array[String] = Array.empty[String],
+ appLocator: String = appLocator,
+ isJVM: Boolean = true ): Unit = {
Review comment:
@nssalian . Could you revert this wrong change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698646782
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698612597
**[Test build #129091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129091/testReport)** for PR 29844 at commit [`9cb5851`](https://github.com/apache/spark/commit/9cb585124160edc3e06dc933784c41838247a0dc).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698638579
@nssalian this LGTM - it looks like you exceeded 100 character line length in a few places and that needs to be fixed to pass the build checks. Once it passes the linting and testing I can merge
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698424261
**[Test build #129079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129079/testReport)** for PR 29844 at commit [`853e1c7`](https://github.com/apache/spark/commit/853e1c734b7735d2743c3d6ccf3615339e85debd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698663600
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33718/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698562528
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698442134
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698433579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698562540
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129085/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494424834
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -89,9 +95,9 @@ private[spark] object Config extends Logging {
val KUBERNETES_DRIVER_SUBMIT_CHECK =
ConfigBuilder("spark.kubernetes.submitInDriver")
- .internal()
- .booleanConf
- .createWithDefault(false)
+ .internal()
Review comment:
this is a nit, but pure formatting changes are not good PR practice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494423864
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
.stringConf
.createOptional
+ val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
+ ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
+ .doc("Service account that is used when running the executor pod." +
+ "If this parameter is not setup, the fallback logic will use the driver's service account.")
Review comment:
It would be more backward compatible to default the executor SA to "none"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698442108
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33700/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494429924
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
}
def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+ def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {
Review comment:
Can the `pod` parameter here be null, or empty?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r495216161
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -81,6 +80,13 @@ private[spark] object Config extends Logging {
.stringConf
.createOptional
+ val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
Review comment:
Ur, this looks wrong.
Why do we backport an improvement JIRA, SPARK-30122 Allow setting serviceAccountName for executor pods, here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043140
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697043404
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nssalian commented on pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
nssalian commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-697849257
@erikerlandson , could you help review this, when you get a chance?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson edited a comment on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
erikerlandson edited a comment on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-699183582
@dongjoon-hyun the changes backported correspond to https://github.com/apache/spark/pull/24748, and SPARK-27872 was tagged as a bug fix (although a credible argument could be made it was a new feature).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson commented on a change in pull request #29844: [SPARK-27872][K8s] Fix executor service account inconsistency for branch-2.4
Posted by GitBox <gi...@apache.org>.
erikerlandson commented on a change in pull request #29844:
URL: https://github.com/apache/spark/pull/29844#discussion_r494425934
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -60,4 +63,15 @@ private[spark] object KubernetesUtils {
}
def parseMasterUrl(url: String): String = url.substring("k8s://".length)
+
+ def buildPodWithServiceAccount(serviceAccount: Option[String], pod: SparkPod): Option[Pod] = {
Review comment:
nit: would call this operation "add service account"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29844:
URL: https://github.com/apache/spark/pull/29844#issuecomment-698661308
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] erikerlandson closed pull request #29844: [SPARK-27872][K8s][2.4] Fix executor service account inconsistency
Posted by GitBox <gi...@apache.org>.
erikerlandson closed pull request #29844:
URL: https://github.com/apache/spark/pull/29844
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org