You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/04 11:56:36 UTC
[GitHub] [airflow] raphaelauv opened a new issue, #22727: gcp provider - GKEStartPodOperator without gcloud to get conf
raphaelauv opened a new issue, #22727:
URL: https://github.com/apache/airflow/issues/22727
### Description
Retrieve the GKE conf with python code and not by running a `gcloud` subprocess cmd
### Use case/motivation
It's currently necessary to have gcloud to use the GKEStartPodOperator , but it look like it's possible without by only using python libs
https://github.com/googleapis/python-container/issues/6#issuecomment-970746358
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lukas-at-harren commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
lukas-at-harren commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1140439827
@rbiegacz I wanted to post a potential solution (one that we use in our codebase). But no, I cannot work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1140534674
@rbiegacz Are you interested to take care of this? I will be happy to help with the implementation of this issue, but I do not have the capacity to check all edge cases especially handling internal IP addresses (`use_internal_ip` parameter in ` GKEStartPodOperator`), handling impersonation.
We can be inspired by how `google-github-actions/get-gke-credentials` action is works
https://github.com/google-github-actions/get-gke-credentials/blob/fa30d95d807daeddd45dd78b8f594d56c0737a73/src/gkeClient.ts#L186-L222
We also have access to the source code of gcloud:
https://github.com/google-cloud-sdk-unofficial/google-cloud-sdk/blob/151b8e4a9d82014cc9ef4b10a3a0be5a1bb95cc0/lib/surface/container/clusters/get_credentials.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rbiegacz commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
rbiegacz commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1140880165
Yes, I can work on this item. Please, assign it to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lukas-at-harren commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
lukas-at-harren commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1136440722
```python
from typing import Sequence
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from tempfile import NamedTemporaryFile
from google.cloud.container_v1 import ClusterManagerClient
import yaml
class GKEJobOperator(KubernetesPodOperator):
template_fields: Sequence[str] = (
'image',
'cmds',
'arguments',
'env_vars',
'labels',
'config_file',
'pod_template_file',
'namespace',
)
def __init__(
self,
*,
project_id = None,
location = None,
cluster_name = None,
config_file = None,
**kwargs,
) -> None:
super().__init__(**{'in_cluster': False, **kwargs})
self.project_id = project_id
self.location = location
self.cluster_name = cluster_name
self.config_file = config_file
def execute(self, context):
if self.config_file is None:
self.config_file = self.tmp_config_file()
return super().execute(context)
def tmp_config_file(self):
cluster_manager = ClusterManagerClient()
cluster = cluster_manager.get_cluster(name=f'projects/{self.project_id}/locations/{self.location}/clusters/{self.cluster_name}')
kubeconfig = {
'apiVersion': 'v1',
'current-context': 'this-cluster',
'clusters': [
{
'name': 'this-cluster',
'cluster': {
'certificate-authority-data': cluster.master_auth.cluster_ca_certificate,
'server': f'https://{cluster.endpoint}'
}
}
],
'contexts': [
{'name': 'this-cluster', 'context': {'cluster': 'this-cluster', 'user': 'this-user'}}
],
'users': [
{
'name': 'this-user',
'user': {
'auth-provider': {'name': 'gcp'}
}
}
]
}
tf = NamedTemporaryFile(delete=False)
with open(tf.name, 'w') as f:
yaml.dump(kubeconfig, f)
return tf.name
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rbiegacz commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
rbiegacz commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1140255347
@lukas-at-harren - are you planning to work on this item to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lukas-at-harren commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
lukas-at-harren commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1285095593
Here is the documentation on how to provide identity to a Kubernetes workload (e.g. Deployment).
https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
We use workload identity to provide a Google Service Account to our Airflow workers.
That SA has the rights to list the cluster and obtain its credentials.
Therefore if you call the Google API from within a worker pod that has this workload identity, it will obtain its credentials from so called application default credentials.
Read more on application default credentials here: https://cloud.google.com/docs/authentication/application-default-credentials
So the „chain“ looks like:
Google Service Account for Airflow workers
> Has the rights to list and get credentials for the other cluster
Airflow cluster (GKE)
> provides workload identity to all Airflow workers (using the SA)
> Airflow worker then uses application default credentials
> The GKEJpbOperator can list the other cluster and obtain its credentials
> The GKEJobOperator can schedule a Pod on the other cluster therefore
Kind regards,
Lukas Rieder
––
Data & Cloud Engineer (ext.)
On 20. Oct 2022 at 09:44 +0200, jholowaty ***@***.***>, wrote:
> Can you give me and example how get the credentials without the cli to the GKEStartPodOperator? Thanks!!
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lukas-at-harren commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
lukas-at-harren commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1285096899
@jholowaty I literally posted the code in the same thread above here. You can read it.
Kind regards,
Lukas Rieder
––
Data & Cloud Engineer (ext.)
On 20. Oct 2022 at 09:44 +0200, jholowaty ***@***.***>, wrote:
> Can you give me and example how get the credentials without the cli to the GKEStartPodOperator? Thanks!!
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1435730575
@rbiegacz are you working on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv closed issue #22727: GKEStartPodOperator without gcloud to get conf
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv closed issue #22727: GKEStartPodOperator without gcloud to get conf
URL: https://github.com/apache/airflow/issues/22727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jholowaty commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
jholowaty commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1285080389
Hi!! when do you think this could be resolved? In the meantime we build a custom airflow image with the google-cloud-cli and google-cloud-sdk-gke-gcloud-auth-plugin inside. Thanks!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jholowaty commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
jholowaty commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1285086835
Can you give me and example how get the credentials without the cli to the GKEStartPodOperator? Thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #22727: GKEStartPodOperator without gcloud to get conf
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1691572525
using
```
apache-airflow-providers-google==10.6.0
apache-airflow-providers-cncf-kubernetes==7.4.2
```
GKEStartPodOperator is working whitout `gcloud` cli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1087548656
It is worth remembering that the token may expire and we need to refresh it. We can be inspired by how it is done in EKS. https://github.com/apache/airflow/pull/17951
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lukas-at-harren commented on issue #22727: google provider - GKEStartPodOperator without gcloud to get conf
Posted by GitBox <gi...@apache.org>.
lukas-at-harren commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1285084571
@jholowaty
You must understand that everything at Google is an API, and you can call it. Even the listing of clusters and getting its credentials.
Kind regards,
Lukas Rieder
––
Data & Cloud Engineer (ext.)
On 20. Oct 2022 at 09:38 +0200, jholowaty ***@***.***>, wrote:
> Hi!! when do you think this could be resolved? In the meantime we build a custom airflow image with the google-cloud-cli and google-cloud-sdk-gke-gcloud-auth-plugin inside. Thanks!!!
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on issue #22727: GKEStartPodOperator without gcloud to get conf
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on issue #22727:
URL: https://github.com/apache/airflow/issues/22727#issuecomment-1537793304
maybe https://github.com/apache/airflow/pull/29266 handle it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org