You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ginolegigot (via GitHub)" <gi...@apache.org> on 2023/10/23 16:41:24 UTC
[PR] Fix service account impersonation with computeEngineSSHHook (google provider) [airflow]
ginolegigot opened a new pull request, #35136:
URL: https://github.com/apache/airflow/pull/35136
Hello !
This PR aims to solve the issue where we can't use service account impersonation when we want to connect with ssh between 2 Google Compute Engine instances.
It is a new version of an old PR i submitted but i did not have the proper specs/money/time to test the code. Now I have them, am willing to learn on the testing side given how newbie i am on this part.
I'll check the airflow developers guideline again meanwhile, i submit this piece of code which add service account impersonation support for ssh connections between 2 GCE instances using IAP tunneling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]
Posted by "ginolegigot (via GitHub)" <gi...@apache.org>.
ginolegigot commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1826372468
Thanks a lot for your reviews and information.
Great day!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix service account impersonation with computeEngineSSHHook (google provider) [airflow]
Posted by "ginolegigot (via GitHub)" <gi...@apache.org>.
ginolegigot commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1775674632
Roger!
I'll try to understand how to write some unit tests in my particular context using airflow tests libs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]
Posted by "ginolegigot (via GitHub)" <gi...@apache.org>.
ginolegigot commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1819606385
Hey!
I think i understood how to setup a proper airflow dev environment, launch tests and I think my unit test even though it might not optimal at least is functional (first time i code unit test).
One more point, usually an impersonation chain is chosen in google operators, so either a service acount email as a string might be used or a list of strings of service account emails.
Given the particular nature of the ComputeEngineSSHHook, (at least in my understanding), it creates a shell command based on gcloud. And I don't know how to pass multiple service accounts to the "--impersonate-service-account=" argument. I did not test yet (so maybe it's trivial) but i'm not sure that's the scope of this PR. I'm opened to discuss it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix service account impersonation with computeEngineSSHHook (google provider) [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1778263226
The simplest way have a look how it done in other Google Provider test, for example
https://github.com/apache/airflow/blob/6c62985055e7f9a715c3ae47f6ff584ad8378e2a/tests/providers/google/marketing_platform/operators/test_analytics.py#L56-L77
L65: Provide some value to operator
L73: Check that hook (mocked) was call with this value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1825394266
@shahar1 can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1826173774
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1826173929
> What are the next steps ? I'm not familiar with the PR process. Are there additional steps from you ? Or shall i do something in addition ?
Just wait until PR gets green and merged by a committer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]
Posted by "ginolegigot (via GitHub)" <gi...@apache.org>.
ginolegigot commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1823990626
Hello @potiuk and @Taragolis!
May i have a review ? Or call someone who can do it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]
Posted by "ginolegigot (via GitHub)" <gi...@apache.org>.
ginolegigot commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1825976483
Hello thanks for your review!
What are the next steps ? I'm not familiar with the PR process. Are there additional steps from you ? Or shall i do something in addition ?
I'm afraid the update branch button cancelled the approval sorry for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add support for service account impersonation with computeEngineSSHHook (google provider) and IAP tunnel [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #35136:
URL: https://github.com/apache/airflow/pull/35136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix service account impersonation with computeEngineSSHHook (google provider) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35136:
URL: https://github.com/apache/airflow/pull/35136#issuecomment-1775670817
It will need unit test. We rarely accept changes without corresponding unit tests - mostly when they are difficult to write. This one seems quite reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org