[GitHub] [spark] ecapoccia commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName

[GitBox <gi...@apache.org>]

ecapoccia commented on issue #27034: [SPARK-30122][K8S] Support spark.kubernetes.authenticate.executor.serviceAccountName
URL: https://github.com/apache/spark/pull/27034#issuecomment-582627637
 
 
   @dongjoon-hyun RE: merge into 3.1. I think the PR should be cancelled then. Spark 3.0 features https://issues.apache.org/jira/browse/SPARK-24434 which in my opinion is superior to exposing individual properties. 
   My point writing a patch similar to this one (as I said, spare that in my code I default to "default") was that I had to support an existing application in production which features Spark 2.4.4 and cannot upgrade as yet - because of the compatibility with Apache Beam. Happy to live with my patch until we can migrate to 3.0+.
   
   @liyinan926 let me try to explain the use case. Assume you have to process a bunch of files stored in S3; the executor pods are the ones reading them, so they need to have the AWS credentials to access S3. In order to use pod identity (https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts) you need to provide the executors with their own service account --> associated with a certain role. 
   
   Every distinct application will have its own service account, none of them is good with "default". The current vanilla Spark 2.4.4 *mandates* the default service account for executors. That's the issue for us -- at least this is my understanding, I'm happy to hear that there is a way around it that we did not consider. 
   
   Notice: the driver pod needs to have its own service account - distinct from the executors - as the requirements are different. Drivers do *not* need to access AWS resources, but they need the privileges to create other pods - which executors shouldn't have.
   
   Hope the above is clearer / makes sense.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org