You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/03 13:56:19 UTC
[GitHub] [airflow] vladiceanu opened a new issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
vladiceanu opened a new issue #12035:
URL: https://github.com/apache/airflow/issues/12035
**Description**
Support of [AssumeRoleWithWebIdentity](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role_with_web_identity) for AWS provider when running Airflow workers in EKS.
**Use case / motivation**
This feature will allow us to use [IRSA](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/) with Airflow Pods running on EKS and x-account _assumeRole_.
A connection type aws with empty username & password and the following extra parameters
```json
{
"role_arn": "arn:aws:iam::708874716777:role/eu-relevance-airflow-eu-west-1",
"region_name": "eu-west-1",
"aws_session_token": "file://$AWS_WEB_IDENTITY_TOKEN_FILE"
}
```
will retrieve temporary credentials using method `sts-assume-role-with-web-identity` (see [boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role_with_web_identity) of this method)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hammad13060 commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
hammad13060 commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-742678909
@mik-laj Is anyone working on it? I'd like to give it a shot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-720515825
Is it related? https://github.com/apache/airflow/issues/9873 If so, I will start working on it this week or next.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-888042695
@kloty Thanks for checking out this solution. Would you like to contribute to this change in documentation? I'm happy to help with review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vladiceanu commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
vladiceanu commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-739808307
sorry folks, I've got a lot on my plate right now and couldn't work on this. I'll unnasign in case if anyone wants to work on it and come back to it by the end of December if nobody picks it up. Cheers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-742682819
@vladiceanu @hammad13060 Isn't it enough to configure an empty AWS connection to use [the boto3 credential search flow](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) and set the `AWS_WEB_IDENTITY_TOKEN_FILE` environment variable?
Reference:
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-723648721
My PR has been merged. Do you need help to add integrations with EKS?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-720515825
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kloty commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
kloty commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-887944378
> @vladiceanu @hammad13060 Isn't it enough to configure an empty AWS connection to use [the boto3 credential search flow](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) and set the `AWS_WEB_IDENTITY_TOKEN_FILE` environment variable?
>
> Reference:
> https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
>
> CC: @besson, @arobinski, @kokojumbo, @SadatAnwar, @Swalloow, and @giannisbetas
For me, this method worked with the correct settings. I'll leave my case for the next person.
I installed Airflow 2.1.2 with helm and attached the required role-arn to the airflow service account as an annotation. The role Includes s3 read write policy for logging and other aws resource related policies.
I also checked IRSA related Environment Variable (`AWS_ROLE_ARN`, `AWS_WEB_IDENTITY_TOKEN_FILE`) were automatically attached to the Airflow-related pods well.
According to `airflow/providers/amazon/aws/hooks/base_aws.py`, if `role-arn` does not specify in the aws connection's extra parameter, a base session declared using an empty parameter will be used.
So I declared AWS Connection with name aws_default without any additional parameters, and I confirmed that everything worked fined in my project.
However, there was some confusion when reading and working on documentation.
:https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#amazon-web-services-connection
Because there are only the GCP related contents are written.
I don't know if above should be written down on documentation, but if that part is resolved, I think this issue can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vladiceanu commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
vladiceanu commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-728184933
> My PR has been merged. Do you need help to add integrations with EKS?
I'm going to work on it by the end of this week. Thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #12035:
URL: https://github.com/apache/airflow/issues/12035
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-888119194
@kloty - happy to review as well :). It's super-easy - jus use the "suggest change on this page" button on the bottom right and it will open the PR with GitHub UI and you do not even need developer environment for that :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kloty edited a comment on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
kloty edited a comment on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-887944378
> @vladiceanu @hammad13060 Isn't it enough to configure an empty AWS connection to use [the boto3 credential search flow](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) and set the `AWS_WEB_IDENTITY_TOKEN_FILE` environment variable?
>
> Reference:
> https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
>
> CC: @besson, @arobinski, @kokojumbo, @SadatAnwar, @Swalloow, and @giannisbetas
For me, this method worked with the correct settings. I'll leave my case for the next person.
I installed Airflow 2.1.2 with helm and attached the required role-arn to the airflow service account as an annotation. The role Includes s3 read write policy for logging and other aws resource related policies.
I also checked IRSA related Environment Variable (`AWS_ROLE_ARN`, `AWS_WEB_IDENTITY_TOKEN_FILE`) were automatically attached to the Airflow-related pods well.
According to `airflow/providers/amazon/aws/hooks/base_aws.py`, if `role-arn` does not specify in the aws connection's extra parameter, a base session declared using an empty parameter will be used.
So I declared AWS Connection with name aws_default without any additional parameters, and I confirmed that everything worked fined in my project.
However, there was some confusion when reading and working on documentation.
:https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#amazon-web-services-connection
Because there are only the GCP related contents are written when working with web identity provider.
I don't know if above should be written down on documentation, but if that part is resolved, I think this issue can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kloty commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
kloty commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-888169161
@mik-laj @potiuk I just added the relevant documentation and I would appreciate for reviewing it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vladiceanu commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
vladiceanu commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-720530525
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-720401831
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kloty edited a comment on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
kloty edited a comment on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-887944378
> @vladiceanu @hammad13060 Isn't it enough to configure an empty AWS connection to use [the boto3 credential search flow](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) and set the `AWS_WEB_IDENTITY_TOKEN_FILE` environment variable?
>
> Reference:
> https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
>
> CC: @besson, @arobinski, @kokojumbo, @SadatAnwar, @Swalloow, and @giannisbetas
For me, this method worked with the correct settings. I'll leave my case for the next person.
I installed Airflow 2.1.2 with helm and attached the required role-arn to the airflow service account as an annotation. The role Includes s3 read write policy for logging and other aws resource related policies.
I also checked IRSA related Environment Variable (`AWS_ROLE_ARN`, `AWS_WEB_IDENTITY_TOKEN_FILE`) were automatically attached to the Airflow-related pods well.
According to `airflow/providers/amazon/aws/hooks/base_aws.py`, if `role-arn` does not specify in the aws connection's extra parameter, a base session declared using an empty parameter will be used.
So I declared AWS Connection with name aws_default without any additional parameters, and I confirmed that everything worked fined in my project.
However, there was some confusion when reading and working on documentation.
:https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#amazon-web-services-connection
Because there are only the GCP related contents are written when working with assume-role-with-web-identity.
I don't know if above should be written down on documentation, but if that part is resolved, I think this issue can be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-742682819
@vladiceanu @hammad13060 Isn't it enough to configure an empty AWS connection to use [the boto3 credential search flow](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) and set the `AWS_WEB_IDENTITY_TOKEN_FILE` environment variable?
Reference:
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
CC: @besson, @arobinski, @kokojumbo, @SadatAnwar, @Swalloow, and @giannisbetas
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12035: Support AssumeRoleWithWebIdentity for AWS provider
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12035:
URL: https://github.com/apache/airflow/issues/12035#issuecomment-742682819
@vladiceanu @hammad13060 Isn't it enough to configure an empty AWS connection to use [the boto3 credential search flow](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) and set the AWS_WEB_IDENTITY_TOKEN_FILE environment variable?
Reference:
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org