You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/14 19:54:21 UTC
[GitHub] [airflow] snazzyfox opened a new issue #8301: Support Duration in
AwsHook when using assume role
snazzyfox opened a new issue #8301: Support Duration in AwsHook when using assume role
URL: https://github.com/apache/airflow/issues/8301
**Description**
Boto3 `assume_role` [supports a `DurationSeconds` option](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role) to generate assumed role credentials valid for a specified duration, instead of the default of 1 hour.
Airflow should also support specifying this option for `AwsHook`.
**Use case / motivation**
When an `AwsHook` is created with a connection that specifies a role, the hook internally calls STS and generates temporary credentials to use. These credentials are always valid for one hour since the hook does not pass the `DurationSeconds` parameter.
Many airflow AWS operators perform long-polling, such as `ECSOperator` (to wait for task completion), `AwsAthenaOperator`, `DynamoDBToS3Operator` (to stream data in chunks), and many more. These operations can run for more than one hour on larger jobs.
Since the credentials generated at hook creation time is only valid for one hour, these jobs will fail one hour after start because credentials in the hook are no longer valid.
**Proposal**
Add a `duration_seconds` attribute to connection extras for AWS Connections. In AwsHook, if using assumed role credentials and this extra exists, pass it through to the boto3 `assume_role` call.
**Related Issues**
None found
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #8301: Support Duration in
AwsHook when using assume role
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8301: Support Duration in AwsHook when using assume role
URL: https://github.com/apache/airflow/issues/8301#issuecomment-613676708
Would you like to work on this issue? Apache Airflow is an open-source project without paid technical support. Each problem is solved by community members - other Airflow users. If this issue is important to you, it is best to take action yourself.
If you are interested, I invite you to read:
Contributor Guide: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst
Development Environment Guide: https://github.com/apache/airflow/blob/master/BREEZE.rst
Testing Guide: https://github.com/apache/airflow/blob/master/TESTING.rst
I hope this information will be helpful for you and I look forward to your PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #8301: Support Duration
in AwsHook when using assume role
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #8301: Support Duration in AwsHook when using assume role
URL: https://github.com/apache/airflow/issues/8301#issuecomment-613676708
Would you like to work on this issue? Apache Airflow is an open-source project without paid technical support. Each problem is solved by community members - other Airflow users. If this issue is important to you, it is best to take action yourself.
If you are interested, I invite you to read the following guides:
Contributor Guide: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst
Development Environment Guide: https://github.com/apache/airflow/blob/master/BREEZE.rst
Testing Guide: https://github.com/apache/airflow/blob/master/TESTING.rst
I hope this information will be helpful for you and I look forward to your PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] snazzyfox commented on issue #8301: Support Duration in
AwsHook when using assume role
Posted by GitBox <gi...@apache.org>.
snazzyfox commented on issue #8301: Support Duration in AwsHook when using assume role
URL: https://github.com/apache/airflow/issues/8301#issuecomment-613734429
After digging a little into the code, it appears that #7619 introduced the ability to set arbitrary kwargs for the assume role function call. That solves this issue, but the change is only on master. Any chance we can backport it to 1.10?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8301: Support
Duration in AwsHook when using assume role
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8301: Support Duration in AwsHook when using assume role
URL: https://github.com/apache/airflow/issues/8301#issuecomment-613649035
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #8301: Support Duration in
AwsHook when using assume role
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8301: Support Duration in AwsHook when using assume role
URL: https://github.com/apache/airflow/issues/8301#issuecomment-613735539
We don't cheery pick new changes related to operators to Airflow 1.10, but we want to release operators from Ariflow 2.0 as a pip package.
More information: https://github.com/apache/airflow/blob/master/README.md#using-hooks-and-operators-from-master-in-airflow-110
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj closed issue #8301: Support Duration in AwsHook
when using assume role
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #8301: Support Duration in AwsHook when using assume role
URL: https://github.com/apache/airflow/issues/8301
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services