You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Yu-Lin Chen (Jira)" <ji...@apache.org> on 2023/11/06 14:55:00 UTC
[jira] [Commented] (YUNIKORN-1961) User Info annotation incorrect for executor pods
[ https://issues.apache.org/jira/browse/YUNIKORN-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783242#comment-17783242 ]
Yu-Lin Chen commented on YUNIKORN-1961:
---------------------------------------
Hi [~mitdesai],
The spark-submit sequentially create driver/executor pods. However, from YuniKorn’s perspective, these are two independent pods without any inherent connection. If user information is not set in the pod, the YuniKorn admission controller simply adds the user information based on the authenticated user interacting with the API server.
In your wordcount example, the authenticated user who requests the driver/executor pod is not matched. The users accounts are:
# Create Driver Pod with your user account(mdesai@visa.com)
# Create Executor Pod with your service account (spark)
We can config spark-submit to make sure the user-info consistent.
Below is the valid solutioins from my test:
# Set up spark-submit to request the driver pod using ‘spark’ service account, so driver/executor have same user-info.
ex:
a. Add ‘spark’ to kubectl context,
b. Change current context to spark or config 'spark.kubernetes.context' in spark-submit
# Set up spark-submit to request executor pod with ‘mdesai@visa.com’ user account, so driver/executor have same user-info. The configs are:
a. Add 'spark.kubernetes.authenticate.driver.clientCertFile' in spark-submit
b. Add spark.kubernetes.authenticate.driver.clientKeyFile in spark-submit
c. remove spark.kubernetes.authenticate.driver.serviceAccountName from spark-submit
(The driver pod will request executor with ‘mdesai@visa.com’ account.)
# Set up spark-submit to add user-info annotation in driver/executor directly. ex:
{code:java}
--conf spark.kubernetes.driver.annotation.yunikorn\.apache\.org/user\.info="{\"user\": \"system:serviceaccount:spark-ns:spark\",\"groups\": [\"system:serviceaccounts\",\"system:serviceaccounts:spark-ns\",\"system:authenticated\"]}" \
--conf spark.kubernetes.executor.annotation.yunikorn\.apache\.org/user\.info="{\"user\": \"system:serviceaccount:spark-ns:spark\",\"groups\": [\"system:serviceaccounts\",\"system:serviceaccounts:spark-ns\",\"system:authenticated\"]}" \ {code}
(But it is required to add your accounts to 'admissionController.accessControl.systemUsers'. I don’t think this is a good solution.[2] ex:
{code:java}
admissionController.accessControl.systemUsers: "^system:serviceaccount:kube-system:,mdesai@visa.com,system:serviceaccount:mdesai:spark"
{code}
)
You can find spark-submit configuration in below link [3].
In my opinion, this issue was caused by the spark application side. if we can configure spark-submit to ensure user-info consistent, there should be no need for code changes in YuniKorn.
Reference Link:
[1] [https://stackoverflow.com/questions/42170380/how-to-add-users-to-kubernetes-kubectl]
[2] [https://yunikorn.apache.org/docs/next/user_guide/service_config/#admissioncontrolleraccesscontrolsystemusers]
[3] [https://spark.apache.org/docs/3.3.1/running-on-kubernetes.html#configuration]
> User Info annotation incorrect for executor pods
> ------------------------------------------------
>
> Key: YUNIKORN-1961
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1961
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: Mit Desai
> Assignee: Yu-Lin Chen
> Priority: Major
>
> UserInfo annotation is getting populated correctly for driver pods. But for the executor pods, _spark_ service account information is populated. Ideally, this should be same as the driver pod where we have the information of the user who originally submitted the job.
> Here is the example of pod describe when running a worcount job with 2 executors:
> {noformat}
> Name: md-wc-94ec368a6d3b30cd-driver
> Namespace: mdesai
> Priority: 0
> Service Account: spark
> Node: *** node1 ***
> Start Time: Wed, 06 Sep 2023 18:21:59 -0700
> Labels: spark-app-selector=spark-c9315eb5d3bf4552b30c77090a328249
> spark-role=driver
> Annotations: otherAnnotations: omited
> yunikorn.apache.org/allow-preemption: true
> yunikorn.apache.org/user.info:
> {"user":"mdesai@visa.com","groups":["group1","group2","...
> Status: Running {noformat}
>
> {noformat}
> Name: javawordcount-ca2f0f8a6d3b64e6-exec-1
> Namespace: mdesai
> Priority: 0
> Service Account: spark
> Node: *** node2 ***
> Start Time: Wed, 06 Sep 2023 18:22:06 -0700
> Labels: spark-app-selector=spark-c9315eb5d3bf4552b30c77090a328249
> spark-exec-id=1
> spark-exec-resourceprofile-id=0
> spark-role=executor
> Annotations: otherAnnotations: omited
> yunikorn.apache.org/allow-preemption: true
> yunikorn.apache.org/user.info:
> {"user":"system:serviceaccount:mdesai:spark","groups":["system:serviceaccounts","system:serviceaccounts:mdesai","system:authenticated"]} {noformat}
>
> {noformat}
> Name: javawordcount-ca2f0f8a6d3b64e6-exec-2
> Namespace: mdesai
> Priority: 0
> Service Account: spark
> Node: *** node3 ***
> Start Time: Wed, 06 Sep 2023 18:22:07 -0700
> Labels: spark-app-selector=spark-c9315eb5d3bf4552b30c77090a328249
> spark-exec-id=2
> spark-exec-resourceprofile-id=0
> spark-role=executor
> Annotations: otherAnnotations: omited
> yunikorn.apache.org/allow-preemption: true
> yunikorn.apache.org/user.info:
> {"user":"system:serviceaccount:mdesai:spark","groups":["system:serviceaccounts","system:serviceaccounts:mdesai","system:authenticated"]}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org