You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/28 19:31:10 UTC
[GitHub] [spark] ayudovin opened a new pull request #27034: [SPARK-30122] -
Allow setting serviceAccountName for executor pods
ayudovin opened a new pull request #27034: [SPARK-30122] - Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Currently, it doesn't seem to be possible to have Spark Driver set the serviceAccountName for executor pods it launches.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
it will allow settings serviceAccountName for executors pods.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
It was tested manually.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#discussion_r372196470
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
// if not setup by the pod template fallback to the driver's sa,
// last option is the default sa.
Review comment:
Could you revise the comment according to the new logic?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581140368
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin edited a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
ayudovin edited a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-577879919
> Hi, @ayudovin .
>
> 1. In general, new feature should not create a breaking change. For example, this PR enforce service account configuration additionally. If executor service account is not given, we had better fallback to use driver service account.
> 2. Do you think we can have a test case for this? Test case is important to protect your contribution from the future changes.
@dongjoon-hyun , Unfortunately, I can not find a way to add a unit test for it. Maybe, you can give advice or tell where I can find an example of tests for these properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27034:
[SPARK-30122][Resource-Manager][Kubernetes] - Allow setting
serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27034: [SPARK-30122][Resource-Manager][Kubernetes] - Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572686632
cc @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582639788
Thank you so much, @ayudovin , @ecapoccia, @liyinan926 . Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 edited a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 edited a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582630983
> @liyinan926 let me try to explain the use case. Assume you have to process a bunch of files stored in S3; the executor pods are the ones reading them, so they need to have the AWS credentials to access S3. In order to use pod identity (https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts) you need to provide the executors with their own service account --> associated with a certain role.
Got it. It's similar to the way GCP's [workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) works, and this will be useful for users using GCS/BigQuery through the workload identity also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-577879919
> Hi, @ayudovin .
>
> 1. In general, new feature should not create a breaking change. For example, this PR enforce service account configuration additionally. If executor service account is not given, we had better fallback to use driver service account.
> 2. Do you think we can have a test case for this? Test case is important to protect your contribution from the future changes.
Unfortunately, I can not find a way to add a unit test for it. Maybe, you can give advice or tell where I can find an example of tests for these properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572695004
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116408/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][Resource-Manager][Kubernetes] - Allow setting
serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][Resource-Manager][Kubernetes] - Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-569445270
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572696474
**[Test build #116409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116409/testReport)** for PR 27034 at commit [`25595b8`](https://github.com/apache/spark/commit/25595b835e1e0a9004d81e40d13b00e35fc4b60a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145123
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22506/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375542844
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
Yep. I agree. If the current behavior is wrong, we can fix that in a separate JIRA and PR since the default behavior was orthogonal.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375535036
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
Oh, right. Then we should stick to the current behavior.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ecapoccia commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582635959
@liyinan926 I think this PR is superseded by the pod template feature in Spark 3+. Unless you plan to backport it to 2.4, in which case makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145583
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22505/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581142955
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22506/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572690333
Also, cc @vanzin
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581139296
**[Test build #117745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117745/testReport)** for PR 27034 at commit [`3aaffa8`](https://github.com/apache/spark/commit/3aaffa839b2853534dd59a9e7a2f856125b65539).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578524738
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22186/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#discussion_r372196587
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -118,6 +118,12 @@ private[spark] object Config extends Logging {
.stringConf
.createOptional
+ val KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME =
+ ConfigBuilder(s"$KUBERNETES_AUTH_EXECUTOR_CONF_PREFIX.serviceAccountName")
+ .doc("Service account that is used when running the executor pod.")
Review comment:
Please add additional explanation for the fallback logic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572717957
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21198/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582637744
> BTW, thank you for sharing your use cases, @ecapoccia . BTW, we don't backport new features. If we merge this, this will be Apache Spark 3.1 because we already in `Feature Freeze` stage for `3.0`.
>
> How do you think about that, @ayudovin ?
ok, that makes sense
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375534441
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
Hi, @liyinan926 and @ecapoccia . The current behavior is fallback to `driverServiceAccount` first, isn't it? This PR only add `executorServiceAccount` at the front of the chain.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581139675
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ecapoccia commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-583735410
I managed to get the full picture on the above.
TL;RD
Long story short, I confirm that in 2.4.4 the executors do *not* take the driver's service account if set. They are on "default" regardless.
Long version.
The change that introduces the new behaviour is this one https://issues.apache.org/jira/browse/SPARK-27872 and is *not* back ported to 2.4.4. So in essence, @ayudovin 's commit does not change the behaviour compared to the codebase he had when he started the work, but Stravos' previous commit (27872) *did* change the behaviour compared to v2.4.4.
As I see it, this is a breaking change for applications when upgrading from 2.4.4 to 3.0.0. All of a sudden if you upgrade Spark, executors that used to be on default will start to pick up a different service account. Which on paper is formally ok, given the major version change; however as I already pointed out
1) it is a bit surprising
2) not sure is the correct thing to do, as the permissions for drivers and executors are different.
However, I think this can be accounted for in the cost of the migration from 2.4.4 to 3.0.0.
A final note on general software crafting practice, @dongjoon-hyun @liyinan926. I think that exposing the configuration is yet another area where the DRY principles apply.
Now you're exposing two completely different ways of setting the executors' service account, one via the pod template, the other via the direct Spark configuration parameters. They are competing with one another, if there are inconsistencies (I set A on the pod template and B via the conf param) I think the pod template prevails (in the current implementation), but then you'll have users trying to change the service account via the config parameter in vain.
I'd rather go for killing the ability to set the executors service accounts via individual properties, given 3.0.0 is not yet released and given the general initiative to expose a template rather than individual parameters. And deprecating the existing method of setting the driver service account directly with a plan to eliminate it in a future release.
Hope the above makes sense, although I am not sure if it's worth raising a JIRA for such a big change.
Happy to hear your views on the subject, at this stage I'm pretty happy with the back porting that I have done to 2.4.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581139644
**[Test build #117744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117744/testReport)** for PR 27034 at commit [`44cd90d`](https://github.com/apache/spark/commit/44cd90daf64cbcaeeaee81ee3762cb040c23153b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375542218
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
Actually I don't think it makes much sense to default to the driver's service account given that the executors don't manage pods/services or any other K8S resources. Defaulting to the `default` service account should be the right behavior.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572705537
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21198/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572694997
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581139677
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117744/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145580
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572717295
Agreed with @dongjoon-hyun that it should fall back to using the driver service account that is the current behavior. Also what is the primary use case for needing to set the service account for executor pods, which normally don't talk to the API server?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572694997
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581140329
**[Test build #117745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117745/testReport)** for PR 27034 at commit [`3aaffa8`](https://github.com/apache/spark/commit/3aaffa839b2853534dd59a9e7a2f856125b65539).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375542151
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
If then, could you file a JIRA for that, @ecapoccia ? It sounds like worth to have a test case at least.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ecapoccia commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375544810
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
@liyinan926 agree with you. they are very different, and should have different identities.
Also -- apologies in advance I am betraying the fact that I'm old here..:)
It violates the principle of least astonishment https://en.wikipedia.org/wiki/Principle_of_least_astonishment
If one sets the driver service account to XYZ and says nothing about the executor service account, I don't think that one expects the executor's service account to be XYZ.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145162
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22506/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578518372
**[Test build #117427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117427/testReport)** for PR 27034 at commit [`ec8bed7`](https://github.com/apache/spark/commit/ec8bed7fa51bfabdb6a14d7c52410880a4d1ad9a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ExecutorKubernetesCredentialsFeatureStepTest extends SparkFunSuite with BeforeAndAfter `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#discussion_r372198992
##########
File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStepTest.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s.features
+
+import org.scalatest.BeforeAndAfter
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.deploy.k8s.{KubernetesExecutorConf, KubernetesTestConf, SparkPod}
+import org.apache.spark.deploy.k8s.Config._
+
+class ExecutorKubernetesCredentialsFeatureStepTest extends SparkFunSuite with BeforeAndAfter {
+
+ private var baseConf: SparkConf = _
+
+ before {
+ baseConf = new SparkConf(false)
+ }
+
+ private def newExecutorConf(
+ environment: Map[String, String] = Map.empty): KubernetesExecutorConf = {
+ KubernetesTestConf.createExecutorConf(
+ sparkConf = baseConf,
+ environment = environment)
+ }
+
+ test("configure spark pod with executor service account") {
+ baseConf.set(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME, "executor-name")
+ val step = new ExecutorKubernetesCredentialsFeatureStep(newExecutorConf())
+ val spec = step
+ .configurePod(SparkPod.initialPod())
+ .pod
+ .getSpec
+
+ val serviceAccountName = spec.getServiceAccountName
+ val accountName = spec.getServiceAccount
+ assert(serviceAccountName.equals("executor-name"))
+ assert(accountName.equals("executor-name"))
Review comment:
Shall we remove the duplicated logic in these three tests by adding a helper function?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578524752
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22186/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-580028887
Yes. @ayudovin , could you answer to @liyinan926 question? You had better put it to the PR description.
> In case you haven't seen my question above. What is the primary use case for needing to set the service account for executor pods, which normally don't talk to the API server?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-580028887
Yes. Could you answer to @liyinan926 question? You had better put it to the PR description.
> In case you haven't seen my question above. What is the primary use case for needing to set the service account for executor pods, which normally don't talk to the API server?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578518405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117427/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582638744
Sorry missed @dongjoon-hyun's comment on backporting. In that case, we will get this into Spark 3.x.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578517311
**[Test build #117427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117427/testReport)** for PR 27034 at commit [`ec8bed7`](https://github.com/apache/spark/commit/ec8bed7fa51bfabdb6a14d7c52410880a4d1ad9a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145580
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145159
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578524744
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572717957
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21198/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#discussion_r372199242
##########
File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStepTest.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s.features
+
+import org.scalatest.BeforeAndAfter
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.deploy.k8s.{KubernetesExecutorConf, KubernetesTestConf, SparkPod}
+import org.apache.spark.deploy.k8s.Config._
+
+class ExecutorKubernetesCredentialsFeatureStepTest extends SparkFunSuite with BeforeAndAfter {
Review comment:
`ExecutorKubernetesCredentialsFeatureStepTest` -> `ExecutorKubernetesCredentialsFeatureStepSuite`?
Please follow the naming style of the other files in this directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375534441
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
Hi, @liyinan926 . The current behavior is fallback to `driverServiceAccount` first, isn't it? This PR only add add `executorServiceAccount` at the front of the chain.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#discussion_r372196291
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
// if not setup by the pod template fallback to the driver's sa,
// last option is the default sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
+ buildPodWithServiceAccount(executorServiceAccount
+ .orElse(driverServiceAccount), pod).getOrElse(pod.pod)
Review comment:
Thank you for adding fallback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581139296
**[Test build #117745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117745/testReport)** for PR 27034 at commit [`3aaffa8`](https://github.com/apache/spark/commit/3aaffa839b2853534dd59a9e7a2f856125b65539).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581140370
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117745/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582557592
Currently, the following fails consistently. I made a PR to fix the failure first (https://github.com/apache/spark/pull/27465)
```
- Launcher client dependencies *** FAILED ***
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375534441
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
Hi, @liyinan926 and @ecapoccia . The current behavior is fallback to `driverServiceAccount` first, isn't it? This PR only add add `executorServiceAccount` at the front of the chain.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581140368
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-569445403
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145583
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22505/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375534441
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
Hi, @liyinan926 . The current behavior is fallback to `driverServiceAccount` first, isn't it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582559040
BTW, thank you for sharing your use cases, @ecapoccia . BTW, we don't backport new features. If we merge this, this will be Apache Spark 3.1 because we already in `Feature Freeze` stage for `3.0`.
How do you think about that, @ayudovin ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581140370
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117745/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145162
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22506/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572717920
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21198/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578518400
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-583786064
@ecapoccia Please ping on `SPARK-27872` instead of this PR (SPARK-30122) for your concerns. I'm worrying about that your all concerns are wasted here without any result.
> The change that introduces the new behaviour is this one https://issues.apache.org/jira/browse/SPARK-27872 and is not back ported to 2.4.4.
And, I want to confirm again that the other relevant concerns on this PR are listened carefully by me and @liancheng . Thanks for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
ayudovin removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-577879919
> Hi, @ayudovin .
>
> 1. In general, new feature should not create a breaking change. For example, this PR enforce service account configuration additionally. If executor service account is not given, we had better fallback to use driver service account.
> 2. Do you think we can have a test case for this? Test case is important to protect your contribution from the future changes.
@dongjoon-hyun , Unfortunately, I can not find a way to add a unit test for it. Maybe, you can give advice or tell where I can find an example of tests for these properties.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581142420
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22505/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034:
[SPARK-30122][Resource-Manager][Kubernetes] - Allow setting
serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][Resource-Manager][Kubernetes] - Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-569445403
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578518400
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ecapoccia commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375540770
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
hmm. I think you're right by reading the code.
However I am pretty sure that in vanilla Spark if you set the driver service account name, the executors will be on default.
I think possibly BasicExecutorFeatureStep ignores the config? I think an integration test will help here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582559611
@ ecapoccia I agree with you that the default should really be `default` instead of the service account of the driver pod. I'm not sure I understand your use case though. Why would you need to use a different service account for accessing S3?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572717946
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375556683
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
Again, you are mentioning the existing behavior before this PR. Please file a JIRA. Otherwise, nothing will happen here for that. It's a way of open source. 😄
> If one sets the driver service account to XYZ and says nothing about the executor service account, I don't think that one expects the executor's service account to be XYZ.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572694852
**[Test build #116408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116408/testReport)** for PR 27034 at commit [`25595b8`](https://github.com/apache/spark/commit/25595b835e1e0a9004d81e40d13b00e35fc4b60a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ecapoccia commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582627637
@dongjoon-hyun RE: merge into 3.1. I think the PR should be cancelled then. Spark 3.0 features https://issues.apache.org/jira/browse/SPARK-24434 which in my opinion is superior to exposing individual properties.
My point writing a patch similar to this one (as I said, spare that in my code I default to "default") was that I had to support an existing application in production which features Spark 2.4.4 and cannot upgrade as yet - because of the compatibility with Apache Beam. Happy to live with my patch until we can migrate to 3.0+.
@liyinan926 let me try to explain the use case. Assume you have to process a bunch of files stored in S3; the executor pods are the ones reading them, so they need to have the AWS credentials to access S3. In order to use pod identity (https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts) you need to provide the executors with their own service account --> associated with a certain role.
Every distinct application will have its own service account, none of them is good with "default". The current vanilla Spark 2.4.4 *mandates* the default service account for executors. That's the issue for us -- at least this is my understanding, I'm happy to hear that there is a way around it that we did not consider.
Notice: the driver pod needs to have its own service account - distinct from the executors - as the requirements are different. Drivers do *not* need to access AWS resources, but they need the privileges to create other pods - which executors shouldn't have.
Hope the above is clearer / makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578524744
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581139677
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117744/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582638871
Yes. Unfortunately, we cannot backport to `branch-2.4.`. For `3.1`, this is also a valid way when we don't use `pod template`. I'd like to merge this to `master` as a convenience method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581138601
**[Test build #117744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117744/testReport)** for PR 27034 at commit [`44cd90d`](https://github.com/apache/spark/commit/44cd90daf64cbcaeeaee81ee3762cb040c23153b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#discussion_r372196921
##########
File path: resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStepTest.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.k8s.features
+
+import org.scalatest.BeforeAndAfter
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.deploy.k8s.{KubernetesExecutorConf, KubernetesTestConf, SparkPod}
+import org.apache.spark.deploy.k8s.Config._
+
+class ExecutorKubernetesCredentialsFeatureStepTest extends SparkFunSuite with BeforeAndAfter {
+
+ private var baseConf: SparkConf = _
+
+ before {
+ baseConf = new SparkConf(false)
+ }
+
+ private def newExecutorConf(
+ environment: Map[String, String] = Map.empty): KubernetesExecutorConf = {
Review comment:
indentation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572696833
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116409/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572696833
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116409/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-579603743
In case you haven't seen my question above. What is the primary use case for needing to set the service account for executor pods, which normally don't talk to the API server?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582636945
@ecapoccia yes, the pod template support works for Spark 3.x, but we can backport this to 2.4.x. @dongjoon-hyun thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on a change in pull request #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on a change in pull request #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#discussion_r375542218
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala
##########
@@ -17,19 +17,23 @@
package org.apache.spark.deploy.k8s.features
import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod}
-import org.apache.spark.deploy.k8s.Config.KUBERNETES_SERVICE_ACCOUNT_NAME
+import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME, KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME}
import org.apache.spark.deploy.k8s.KubernetesUtils.buildPodWithServiceAccount
private[spark] class ExecutorKubernetesCredentialsFeatureStep(kubernetesConf: KubernetesConf)
extends KubernetesFeatureConfigStep {
- private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_SERVICE_ACCOUNT_NAME)
+
+ private lazy val driverServiceAccount = kubernetesConf.get(KUBERNETES_DRIVER_SERVICE_ACCOUNT_NAME)
+ private lazy val executorServiceAccount =
+ kubernetesConf.get(KUBERNETES_EXECUTOR_SERVICE_ACCOUNT_NAME)
override def configurePod(pod: SparkPod): SparkPod = {
pod.copy(
- // if not setup by the pod template fallback to the driver's sa,
- // last option is the default sa.
+ // if not setup by the pod template, fallback to the executor's sa,
+ // if executor's sa is not setup, the last option is driver's sa.
pod = if (Option(pod.pod.getSpec.getServiceAccount).isEmpty) {
- buildPodWithServiceAccount(driverServiceAccount, pod).getOrElse(pod.pod)
Review comment:
Actually I don't think it makes much sense to default to the driver's service account given that the executors don't manage pods/services or any other K8S resources. Defaulting to the `default` service account should be the right behavior. Please create a Jira issue for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581139675
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034:
[SPARK-30122][Resource-Manager][Kubernetes] - Allow setting
serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][Resource-Manager][Kubernetes] - Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-569445270
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572691915
**[Test build #116409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116409/testReport)** for PR 27034 at commit [`25595b8`](https://github.com/apache/spark/commit/25595b835e1e0a9004d81e40d13b00e35fc4b60a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582639475
> Yes. Unfortunately, we cannot backport to `branch-2.4.`. For `3.1`, this is also a valid way when we don't use `pod template`. I'd like to merge this to `master` as a convenience method.
Agreed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582639521
/lgtm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-579606412
Also, @ayudovin . Could you describe a little bit more about your use case for this feature in the PR description, please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572688751
**[Test build #116408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116408/testReport)** for PR 27034 at commit [`25595b8`](https://github.com/apache/spark/commit/25595b835e1e0a9004d81e40d13b00e35fc4b60a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27034:
[SPARK-30122][Resource-Manager][Kubernetes] - Allow setting
serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27034: [SPARK-30122][Resource-Manager][Kubernetes] - Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-570820424
@liyinan926, Could you please review this pull request?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572688751
**[Test build #116408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116408/testReport)** for PR 27034 at commit [`25595b8`](https://github.com/apache/spark/commit/25595b835e1e0a9004d81e40d13b00e35fc4b60a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572717946
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578518405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117427/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow
setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578521196
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22186/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ecapoccia edited a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
ecapoccia edited a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-583735410
I managed to get the full picture on the above.
TL;DR
Long story short, I confirm that in 2.4.4 the executors do *not* take the driver's service account if set. They are on "default" regardless.
Long version.
The change that introduces the new behaviour is this one https://issues.apache.org/jira/browse/SPARK-27872 and is *not* back ported to 2.4.4. So in essence, @ayudovin 's commit does not change the behaviour compared to the codebase he had when he started the work, but Stravos' previous commit (27872) *did* change the behaviour compared to v2.4.4.
As I see it, this is a breaking change for applications when upgrading from 2.4.4 to 3.0.0. All of a sudden if you upgrade Spark, executors that used to be on default will start to pick up a different service account. Which on paper is formally ok, given the major version change; however as I already pointed out
1) it is a bit surprising
2) not sure is the correct thing to do, as the permissions for drivers and executors are different.
However, I think this can be accounted for in the cost of the migration from 2.4.4 to 3.0.0.
A final note on general software crafting practice, @dongjoon-hyun @liyinan926. I think that exposing the configuration is yet another area where the DRY principles apply.
Now you're exposing two completely different ways of setting the executors' service account, one via the pod template, the other via the direct Spark configuration parameters. They are competing with one another, if there are inconsistencies (I set A on the pod template and B via the conf param) I think the pod template prevails (in the current implementation), but then you'll have users trying to change the service account via the config parameter in vain.
I'd rather go for killing the ability to set the executors service accounts via individual properties, given 3.0.0 is not yet released and given the general initiative to expose a template rather than individual parameters. And deprecating the existing method of setting the driver service account directly with a plan to eliminate it in a future release.
Hope the above makes sense, although I am not sure if it's worth raising a JIRA for such a big change.
Happy to hear your views on the subject, at this stage I'm pretty happy with the back porting that I have done to 2.4.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27034:
[SPARK-30122][Resource-Manager][Kubernetes] - Allow setting
serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27034: [SPARK-30122][Resource-Manager][Kubernetes] - Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572688551
ok to test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145159
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 edited a comment on issue #27034:
[SPARK-30122][K8S] Support
spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 edited a comment on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582559611
@ecapoccia I agree with you that the default should really be `default` instead of the service account of the driver pod. I'm not sure I understand your use case though. Why would you need to use a different service account for accessing S3?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581138601
**[Test build #117744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117744/testReport)** for PR 27034 at commit [`44cd90d`](https://github.com/apache/spark/commit/44cd90daf64cbcaeeaee81ee3762cb040c23153b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] liyinan926 commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
liyinan926 commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582630983
> @liyinan926 let me try to explain the use case. Assume you have to process a bunch of files stored in S3; the executor pods are the ones reading them, so they need to have the AWS credentials to access S3. In order to use pod identity (https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts) you need to provide the executors with their own service account --> associated with a certain role.
Got it. It's similar to the way GCP's [workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) works, and this will be useful for users using GCS/BigQuery through the workload identity also.
@ayudovin if you could make the change to make the fallback service account the `default` one, this PR looks good to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578524752
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22186/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ecapoccia commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581116487
Interestingly, I am reading this as I was about to submit an almost identical pull request. I should have checked in advance, too bad.
In my implementation however the fallback for the executor service account name is not the driver's one but rather the default -- as far as I understand, this is the current functionality.
The use case is quite simple, my worker nodes need to access AWS resources (primarily S3) and so far I had to use the node identity on the cluster nodes to set their roles. This violates the least privilege principle, as the role is now shared with every single pod than runs on the node. Hope this explain the reason why we worked out this solution @liyinan926 @dongjoon-hyun .
I know in Spark 3 there is the plan to expose the entire pod template for the executor, which is a more reasonable solution than exposing the properties individually.
However, we need to support the current applications running on Spark 2.4 (and in my case Scala 2.11 as I use Beam).
Not sure what to do with my implementation thou, I will keep it for myself probably since this PR is already on its way. I think however that in order to be a like-for-like if the executor serviceAccountName is not set the default should be "default".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-578517311
**[Test build #117427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117427/testReport)** for PR 27034 at commit [`ec8bed7`](https://github.com/apache/spark/commit/ec8bed7fa51bfabdb6a14d7c52410880a4d1ad9a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27034: [SPARK-30122][K8S]
Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572695004
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116408/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27034:
[SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27034: [SPARK-30122][K8S] Allow setting serviceAccountName for executor pods
URL: https://github.com/apache/spark/pull/27034#issuecomment-572691915
**[Test build #116409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116409/testReport)** for PR 27034 at commit [`25595b8`](https://github.com/apache/spark/commit/25595b835e1e0a9004d81e40d13b00e35fc4b60a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27034: [SPARK-30122][K8S]
Support spark.kubernetes.authenticate.executor.serviceAccountName
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-581145359
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22505/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org