You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tiago Reis (Jira)" <ji...@apache.org> on 2021/07/30 22:54:00 UTC
[jira] [Updated] (SPARK-36366) Google Kubernetes Engine
authentication fails
[ https://issues.apache.org/jira/browse/SPARK-36366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tiago Reis updated SPARK-36366:
-------------------------------
Description:
When connecting to a Google Kubernetes Engine, a command {{gcloud container clusters get-credentials}} is used that generates a {{~/.kube/config}} file. The distinctive trait in this config file is that it uses an {{auth-provider}} relying on {{gcloud}} to inject the keys {{expiry}} and {{access-token}} from the general Google SDK auth config, as seen here:
{code:json}
users:
- name: gke_my-project_my-region_my-cluster
user:
auth-provider:
config:
cmd-args: config config-helper --format=json
cmd-path: /Users/user/google-cloud-sdk/bin/gcloud
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
{code}
{{kubectl}}, because it uses {{client-go}}, supports the auth-provider and fetches the token and expiry from the json returne by config-helper. As Spark is using the fabric8 client, this is yet to be supported, breaking when running spark-submit:
{code:java}
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://my-endpoint/api/v1/namespaces/my-namespace/pods. Message: Forbidden! User gke_my-project_my-region_my-cluster doesn't have permission. pods is forbidden: User "system:anonymous" cannot create resource "pods" in API group "" in the namespace "my-namespace".
{code}
was:
When connecting to a Google Kubernetes Engine, a command {{gcloud container clusters get-credentials}} is used that generates a {{~/.kube/config}} file. The distinctive trait in this config file is that it uses an {{auth-provider}} relying on {{gcloud}} to inject the keys {{expiry}} and {{access-token}} from the general Google SDK auth config, as seen here:
{code:json}
users:
- name: gke_my-project_my-region_my-cluster
user:
auth-provider:
config:
cmd-args: config config-helper --format=json
cmd-path: /Users/reist01/google-cloud-sdk/bin/gcloud
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
{code}
{{kubectl}}, because it uses {{client-go}}, supports the auth-provider and fetches the token and expiry from the json returne by config-helper. As Spark is using the fabric8 client, this is yet to be supported, breaking when running spark-submit:
{code:java}
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://my-endpoint/api/v1/namespaces/my-namespace/pods. Message: Forbidden! User gke_my-project_my-region_my-cluster doesn't have permission. pods is forbidden: User "system:anonymous" cannot create resource "pods" in API group "" in the namespace "my-namespace".
{code}
> Google Kubernetes Engine authentication fails
> ---------------------------------------------
>
> Key: SPARK-36366
> URL: https://issues.apache.org/jira/browse/SPARK-36366
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.1.2
> Environment: {code}
> $ kubectl version
> Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.15", GitCommit:"73dd5c840662bb066a146d0871216333181f4b64", GitTreeState:"clean", BuildDate:"2021-01-13T13:22:41Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"}
> Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.19-gke.1701", GitCommit:"d7cecefb99b58e8968f59b59d76448eb1e6ea403", GitTreeState:"clean", BuildDate:"2021-06-23T21:51:59Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}
> $ spark-submit --version
> version 3.1.2
> Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 11.0.10
> {code}
> Reporter: Tiago Reis
> Priority: Minor
> Labels: google, kubernetes, kubernetesexecutor, newbie
>
> When connecting to a Google Kubernetes Engine, a command {{gcloud container clusters get-credentials}} is used that generates a {{~/.kube/config}} file. The distinctive trait in this config file is that it uses an {{auth-provider}} relying on {{gcloud}} to inject the keys {{expiry}} and {{access-token}} from the general Google SDK auth config, as seen here:
> {code:json}
> users:
> - name: gke_my-project_my-region_my-cluster
> user:
> auth-provider:
> config:
> cmd-args: config config-helper --format=json
> cmd-path: /Users/user/google-cloud-sdk/bin/gcloud
> expiry-key: '{.credential.token_expiry}'
> token-key: '{.credential.access_token}'
> {code}
> {{kubectl}}, because it uses {{client-go}}, supports the auth-provider and fetches the token and expiry from the json returne by config-helper. As Spark is using the fabric8 client, this is yet to be supported, breaking when running spark-submit:
> {code:java}
> Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://my-endpoint/api/v1/namespaces/my-namespace/pods. Message: Forbidden! User gke_my-project_my-region_my-cluster doesn't have permission. pods is forbidden: User "system:anonymous" cannot create resource "pods" in API group "" in the namespace "my-namespace".
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org