You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by "ashokkumarrathore (via GitHub)" <gi...@apache.org> on 2024/11/15 13:53:51 UTC
[I] Spark on Kubernetes job fails if service account doesn't have permission to all namespaces. [incubator-livy]
ashokkumarrathore opened a new issue, #461:
URL: https://github.com/apache/incubator-livy/issues/461
In a multi tenant Kubernetes cluster, it will not always be the case that livy service account has access to all namespaces. Since we look for job in all namespaces, it fails if the permission is not there for some namespace.
Ideally we should only look for job in the namespace it was submitted to. Relevant code is here, i think : https://github.com/apache/incubator-livy/blob/1f6bd7ab55db3e5b08ff133d554b32db83a23890/server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala#L446
Also this,
https://github.com/apache/incubator-livy/blob/1f6bd7ab55db3e5b08ff133d554b32db83a23890/server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala#L689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Spark on Kubernetes job fails if service account doesn't have permission to all namespaces. [incubator-livy]
Posted by "ashokkumarrathore (via GitHub)" <gi...@apache.org>.
ashokkumarrathore commented on issue #461:
URL: https://github.com/apache/incubator-livy/issues/461#issuecomment-2478894355
@askhatri @jahstreet fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Spark on Kubernetes job fails if service account doesn't have permission to all namespaces. [incubator-livy]
Posted by "askhatri (via GitHub)" <gi...@apache.org>.
askhatri commented on issue #461:
URL: https://github.com/apache/incubator-livy/issues/461#issuecomment-2550314562
Hi @ashokkumarrathore,
Thank you for your efforts. I agree with the suggested fix.
CC: @jahstreet , @gyogal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Spark on Kubernetes job fails if service account doesn't have permission to all namespaces. [incubator-livy]
Posted by "ashokkumarrathore (via GitHub)" <gi...@apache.org>.
ashokkumarrathore commented on issue #461:
URL: https://github.com/apache/incubator-livy/issues/461#issuecomment-2550300786
The potential fix is to provide a namespace argument and use that in GetApplications() from Kubernetes. So, rather than looking for app in any namespace [here](https://github.com/apache/incubator-livy/blob/1f6bd7ab55db3e5b08ff133d554b32db83a23890/server/src/main/scala/org/apache/livy/utils/SparkKubernetesApp.scala#L689), it should be namespaced.
The namespace need to be passed from here:
https://github.com/apache/incubator-livy/blob/1f6bd7ab55db3e5b08ff133d554b32db83a23890/server/src/main/scala/org/apache/livy/server/batch/BatchSession.scala#L109
There are couple of options on how we do this:
1. Add a namespace argument to SparkApp.create() call. This changes the create() function definition which is generic for both YarnApp and KubernetesApp.
2. To avoid modifying create() call, we can put namespace as part of LivyConf and pass a copy of this rather than actual ref. This will have minimal changes but technically namespace should not be part of LivyConf.
@jahstreet @askhatri Please let me know your thoughts and i can implement it accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Spark on Kubernetes job fails if service account doesn't have permission to all namespaces. [incubator-livy]
Posted by "ashokkumarrathore (via GitHub)" <gi...@apache.org>.
ashokkumarrathore commented on issue #461:
URL: https://github.com/apache/incubator-livy/issues/461#issuecomment-2567005302
@askhatri @jahstreet Changes for namespace support in Livy for Spark on k8s. https://github.com/apache/incubator-livy/pull/462
Can you guys please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Spark on Kubernetes job fails if service account doesn't have permission to all namespaces. [incubator-livy]
Posted by "askhatri (via GitHub)" <gi...@apache.org>.
askhatri commented on issue #461:
URL: https://github.com/apache/incubator-livy/issues/461#issuecomment-2479075067
Thank you @ashokkumarrathore for the findings. @jahstreet, please review and share your insights.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Spark on Kubernetes job fails if service account doesn't have permission to all namespaces. [incubator-livy]
Posted by "jahstreet (via GitHub)" <gi...@apache.org>.
jahstreet commented on issue #461:
URL: https://github.com/apache/incubator-livy/issues/461#issuecomment-2611017347
Reviewed the MR. Valid point 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@livy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org