You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/27 09:13:15 UTC
[GitHub] [airflow] olchas opened a new issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator
olchas opened a new issue #10596:
URL: https://github.com/apache/airflow/issues/10596
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
`GKEStartPodOperator`, `DataflowCreateJavaJobOperator` and `DataflowCreatePythonJobOperator` do not support [direct impersonation of a service account in Google services](https://cloud.google.com/iam/docs/understanding-service-accounts#directly_impersonating_a_service_account).
**Use case / motivation**
`GKEStartPodOperator`, `DataflowCreateJavaJobOperator` and `DataflowCreatePythonJobOperator`, in contrary to other Google operators, do not use `Credentials` class for authentication, so they require individual approach to support direct impersonation in them.
In case of `GKEStartPodOperator` it seems it should suffice to add [`--impersonate-service-account`](https://cloud.google.com/sdk/gcloud/reference#--impersonate-service-account) to the [`gcloud container clusters get-credentials` command](https://github.com/apache/airflow/blob/bb3e45c2509bba760f720d60b008f2f71e1024e2/airflow/providers/google/cloud/operators/kubernetes_engine.py#L324). However, this way we will not be able to use chain of service accounts, like in the rest of Google operators.
In case of `DataflowCreateJavaJobOperator` and `DataflowCreatePythonJobOperator`, some changes in [`provide_gcp_credential_file_as_context`](https://github.com/apache/airflow/blob/master/airflow/providers/google/common/hooks/base_google.py#L396) will probably be needed.
**Related Issues**
https://github.com/apache/airflow/issues/8803
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rajatsri28 commented on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator
Posted by GitBox <gi...@apache.org>.
rajatsri28 commented on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-703460034
Hi @olchas ,
Can I pick this up?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rajatsri28 commented on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator
Posted by GitBox <gi...@apache.org>.
rajatsri28 commented on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-709240745
Hi,
For `GKEStartPodOperator`, I did not find any way to use a chain of service accounts for impersonation. It seems it's only available via the API. So I am thinking of following what is suggested here, i.e. to add `--impersonate-service-account` to the `gcloud container clusters get-credentials` command.
For `DataflowCreateJavaJobOperator` and `DataflowCreatePythonJobOperator`, I am a bit confused since there are not many options to update ADC. Are we looking for something like running `gcloud config set auth/impersonate_service_account` command to declare the impersonation? However, we will not be able to specify a chain of accounts then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-703480940
@rajatsri28 Awesome! I assigned you to this ticket. 🐈
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-703465040
@rajatsri28 Do you have experience with Google Cloud? This task is trivial from an Airflow perspective, but requires above-average knowledge of the Google Cloud platform. I don't want you to have a bad experience after contributing to this project.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] abhishekbafna commented on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator
Posted by GitBox <gi...@apache.org>.
abhishekbafna commented on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-907141204
The DataflowCreateJavaJobOperator is documented as deprecated and users are suggested to use the BeamRunJavaPipelineOperator. I tried that and I believe, I was not able to impersonate the different service account in that as well. The reason could be that the underline logic might be the same between the two operators. We should add these Beam operators also into the above list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-703480940
@rajatsri28 Awesome! I assigned you to this ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rajatsri28 commented on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator
Posted by GitBox <gi...@apache.org>.
rajatsri28 commented on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-703476903
Hi @mik-laj , Yes I have some experience with Google Cloud. At Twitter, we migrated from on-premise offering to provide Airflow on GKE (GCP) so I can somewhat relate to this issue as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10596: Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10596:
URL: https://github.com/apache/airflow/issues/10596#issuecomment-712146597
@rajatsri28 I think support for a single service account is sufficient in most cases. `gcloud config set auth/impersonate_service_account` The gcloud command should work for all operators that use gcloud, so all you have to do is add account name forwarding in `airflow.providers.google.common.hooks.base_google.GoogleBaseHook.provide_authorized_gcloud` method.
`gcloud config set auth/impersonate_service_account` command and `--impersonate-service-account` argument behave similarly, but I think the first method will be simpler to implement. I am not sure about this and think it is worth checking for all the gcloud supported authentication methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org