You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/30 14:13:46 UTC
[GitHub] [airflow] TobKed opened a new pull request #19894: CI - OpenID Connect authorication to AWS
TobKed opened a new pull request #19894:
URL: https://github.com/apache/airflow/pull/19894
Hardening security by using OIDC to authorize access to AWS.
Requires to set up `DOCS_AWS_ROLE_TO_ASSUME` in GA secrets, `DOCS_AWS_ACCESS_KEY_ID` and `DOCS_AWS_SECRET_ACCESS_KEY` secrets will not be necessary anymore and could be deleted.
# TODO:
- [ ] - trust and policy json files
- [ ] - instructions to setup OIDC on AWS
### Links
Instructions based on : https://github.com/TobKed/github_actions_oidc/
More information about GItHub Actions and OIDC:
- [The GitHub Blog: GitHub Actions: Secure cloud deployments with OpenID Connect](https://github.blog/changelog/2021-10-27-github-actions-secure-cloud-deployments-with-openid-connect/)
- [GitHub Docs - About security hardening with OpenID Connect](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect)
- [GitHub Docs - Configuring OpenID Connect in Amazon Web Services](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)
- [AWS - Creating OpenID Connect (OIDC) identity providers](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html)
- [AWS - Creating a role for web identity or OpenID connect federation (console) ](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html)
- [AWS - Obtaining the thumbprint for an OpenID Connect Identity Provider](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html)
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-982690196
We'll need to configure our AWS account to trust this too right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-989264721
Cool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-982681566
Really nice @ashb - we can get rid of the AWS secrets this way :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TobKed commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-986096307
With @mik-laj we tested OIDC in other project and [`permissions`](https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idpermissions) should be tested before merging. I will do it in incoming week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-989992102
This hasn't worked https://github.com/apache/airflow/runs/4472124262?check_suite_focus=true -- and I've added `sts:AssumeRole` to the instance profile permission, but even after 15 minutes (more than enough time for IAM to propagate changes) I can't even get `aws sts assume-role` to work form one of the AWS self-hosted runners.
Hmmmm.
Oh the trust relationship conditions is why ManualRunnerRole can't assume that role (only Github OIDC can). But anyway, since we are running on AWS hosted runners with an instance role we don't need any of this. As it's blocking main I'm just going to remove the cred section entirely, it's not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TobKed edited a comment on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
TobKed edited a comment on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-982700123
I've just updated instructions. I don't have too much experience with AWS and I am not 100% confident about policies. In case they could be tweaked let me know, I will improve intructions in my side repo as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #19894:
URL: https://github.com/apache/airflow/pull/19894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-989992102
This hasn't worked https://github.com/apache/airflow/runs/4472124262?check_suite_focus=true -- and I've added `sts:AssumeRole` to the instance profile permission, but even after 15 minutes (more than enough time for IAM to propagate changes) I can't even get `aws sts assume-role` to work form one of the AWS self-hosted runners.
Hmmmm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-983000768
Thanks for doing this task I thought for a long time to do it, but I couldn't find the time to do it. If you want it, I can call you over the weekend and set up an AWS account to check it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-982694037
> We'll need to configure our AWS account to trust this too right?
Yep. @TobKed provided instructions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-989210221
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-990000209
Corection, the instance profile _doesn't_ have the perms on the S3 bucket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TobKed commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-982700123
I've just updated instructions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-982689445
Oh yeah, I saw something about this the other week, then forgot it entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #19894: CI - OpenID Connect authorication to AWS
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #19894:
URL: https://github.com/apache/airflow/pull/19894#issuecomment-989210536
I updated the ARN so now I'm merging to test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org