You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anshulbaliga7 (via GitHub)" <gi...@apache.org> on 2026/09/15 06:45:09 UTC

[PR] [SPARK-59518][K8S] Fall back to the driver service account when the executor one is empty [spark]

anshulbaliga7 opened a new pull request, #58807:
URL: https://github.com/apache/spark/pull/58807

   ### What changes were proposed in this pull request?
   
   `ExecutorKubernetesCredentialsFeatureStep` now treats an explicitly empty `spark.kubernetes.authenticate.executor.serviceAccountName` (and `spark.kubernetes.authenticate.driver.serviceAccountName`) as unset. Both optional configs are filtered with `.filter(_.nonEmpty)` before the fallback chain runs, so:
   
   - an empty executor account falls back to the driver account, as documented;
   - if both are empty, the executor pod spec is left untouched instead of receiving `""`.
   
   The config description in `docs/running-on-kubernetes.md` is updated to say "not set or is empty". The change is confined to the two config reads so it stays independent of #58343 (SPARK-58910), which reworks the pod-template check in the same step.
   
   ### Why are the changes needed?
   
   The executor service account config is `createOptional`, so setting it to an empty string yields `Some("")`. `executorServiceAccount.orElse(driverServiceAccount)` then never falls back, and `buildPodWithServiceAccount` writes an empty `serviceAccount` / `serviceAccountName` into the executor pod. That contradicts the documented behavior ("if this parameter is not setup, the fallback logic will use the driver's service account") and produces a pod that names no usable
   account. An empty value is easy to hit from templating or `--conf key=` in submission scripts.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, a bug fix. Before, `--conf spark.kubernetes.authenticate.executor.serviceAccountName=` with a driver account configured wrote `serviceAccountName: ""` to executor pods. Now such executors use the driver's service account, matching the behavior when the config is absent. Non-empty values behave exactly as before.
   
   ### How was this patch tested?
   
   Two new tests in `ExecutorKubernetesCredentialsFeatureStepSuite`:
   
   - empty executor account plus a driver account resolves to the driver account;
   - empty executor and driver accounts leave `serviceAccount` / `serviceAccountName` null.
   
   ```
   build/sbt -Pkubernetes "kubernetes/testOnly org.apache.spark.deploy.k8s.features.ExecutorKubernetesCredentialsFeatureStepSuite"
   ```
   
   All 5 tests pass (3 existing, 2 new).
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Fable 5.1)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59518][K8S] Fall back to the driver service account when the executor one is empty [spark]

Posted by "anshulbaliga7 (via GitHub)" <gi...@apache.org>.
anshulbaliga7 commented on PR #58807:
URL: https://github.com/apache/spark/pull/58807#issuecomment-6065999026

   Rebased on SPARK-58910 and refreshed the description. This is a small one, two filter calls plus tests. @LuciferYang since you filed the ticket, or @dongjoon-hyun @peter-toth if either of you has a moment. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59518][K8S] Fall back to the driver service account when the executor one is empty [spark]

Posted by "anshulbaliga7 (via GitHub)" <gi...@apache.org>.
anshulbaliga7 commented on PR #58807:
URL: https://github.com/apache/spark/pull/58807#issuecomment-5710263171

   @LuciferYang I have updated to filter both lazy vals as described in the jira description. Can you PTAL? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59518][K8S] Fall back to the driver service account when the executor one is empty [spark]

Posted by "anshulbaliga7 (via GitHub)" <gi...@apache.org>.
anshulbaliga7 commented on PR #58807:
URL: https://github.com/apache/spark/pull/58807#issuecomment-5677996323

   @LuciferYang @dongjoon-hyun could you please take a look when you have a moment? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org