You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/02 08:50:37 UTC
[GitHub] [airflow] baolsen opened a new issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
baolsen opened a new issue #16770:
URL: https://github.com/apache/airflow/issues/16770
**Apache Airflow version**: 1.10.8 (Patched with latest AWS Hook)
**Environment**:
- **Cloud provider or hardware configuration**: 4 VCPU 8GB RAM VM
- **OS** (e.g. from /etc/os-release): RHEL 7.7
- **Kernel** (e.g. `uname -a`): Linux 3.10.0-957.el7.x86_64
- **Install tools**:
- **Others**:
The AWS Hook functionality for AssumeRoleWithSAML is not available in this version, we manually added it via patching the hook file.
**What happened**:
We've been using this hook for a while now with this issue, basically sts.assume_role and sts.assume_role_with_saml will return temporary credentials that are only valid for eg 1 hour by default. Eventually with long running operators / hooks / sensors some of them fail because the credentials have expired.
Example error messages
An error occurred (ExpiredTokenException) when calling the AssumeRole operation: Response has expired
An error occurred (ExpiredTokenException) when calling the AssumeRoleWithSAML operation: Response has expired
botocore.exceptions.ClientError: An error occurred (ExpiredTokenException) when calling the <any operation here> operation: The security token included in the request is expired
**What you expected to happen**:
AWS hook should be updated to use boto3 RefreshableCredentials when temporary credentials are in use.
**How to reproduce it**:
Use any of the assume role methods with the AWS Hook, create a session, wait 1 hour (or whatever expiry period applies to your role), and try and use the hook again.
**Anything else we need to know**:
I have a solution, please self-assign this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ss-nef edited a comment on issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
Posted by GitBox <gi...@apache.org>.
ss-nef edited a comment on issue #16770:
URL: https://github.com/apache/airflow/issues/16770#issuecomment-944298277
Hello,
I am not able to understand how to enable the automatic refresh when using the ECSOperator to orchestrate Fargate Containers.
The container I am running is taking more than one hour to complete its task, which is why I am getting the following error when using the ECSOperator:
```
ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 220, in execute
self._wait_for_task_ended()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 293, in _wait_for_task_ended
waiter.wait(cluster=self.cluster, tasks=[self.arn])
File "/home/airflow/.local/lib/python3.8/site-packages/botocore/waiter.py", line 53, in wait
Waiter.wait(self, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/botocore/waiter.py", line 334, in wait
raise WaiterError(
botocore.exceptions.WaiterError: Waiter TasksStopped failed: An error occurred (ExpiredTokenException): The security token included in the request is expired
```
what setting do I need to set to enable automtic refreshing, so that the waiter can wait for longer than one hour?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ss-nef commented on issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
Posted by GitBox <gi...@apache.org>.
ss-nef commented on issue #16770:
URL: https://github.com/apache/airflow/issues/16770#issuecomment-946591174
Alright thank you for clearing that up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ss-nef commented on issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
Posted by GitBox <gi...@apache.org>.
ss-nef commented on issue #16770:
URL: https://github.com/apache/airflow/issues/16770#issuecomment-944298277
Hello,
I am not able to understand how to enable the automatic refresh when using the ECSOperator to orchestrate Fargate Containers.
The container I am running is taking more than one hour to complete its task, which is why I am getting the following error when using the ECSOperator:
```ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 220, in execute
self._wait_for_task_ended()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 293, in _wait_for_task_ended
waiter.wait(cluster=self.cluster, tasks=[self.arn])
File "/home/airflow/.local/lib/python3.8/site-packages/botocore/waiter.py", line 53, in wait
Waiter.wait(self, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/botocore/waiter.py", line 334, in wait
raise WaiterError(
botocore.exceptions.WaiterError: Waiter TasksStopped failed: An error occurred (ExpiredTokenException): The security token included in the request is expired```
what setting do I need to set to enable automtic refreshing, so that the waiter can wait for longer than one hour?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16770:
URL: https://github.com/apache/airflow/issues/16770#issuecomment-946590278
Docker images for Airflow 2.1.0 has been shipped with ``apache-airflow-providers-amazon==2.0.0``,
https://github.com/apache/airflow/blob/constraints-2.1.2/constraints-3.9.txt#L49
so if you are using this image then you need to update the package.
For details, see: http://airflow.apache.org/docs/docker-stack/build.html#example-of-upgrading-airflow-provider-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #16770:
URL: https://github.com/apache/airflow/issues/16770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16770:
URL: https://github.com/apache/airflow/issues/16770#issuecomment-946587821
This is not tied to the Airflow release but to Amazon provider package release. See: http://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/index.html
This change has been releasesd in apache-airflow-providers-amazon v2.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ss-nef commented on issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
Posted by GitBox <gi...@apache.org>.
ss-nef commented on issue #16770:
URL: https://github.com/apache/airflow/issues/16770#issuecomment-946550142
Thank you for your quick reply,
is it possible that the changes you made in airflow/providers/amazon/aws/hooks/base_aws.py were actually not shipped with Airflow version v2.1.2?
Because I am looking through your code changes and it seems like your changes are missing in v2.1.
I am asking because in this issue https://github.com/apache/airflow/issues/8301 it is said the changes should have been already come with v2.1.
Maybe I just need to upgrade airflow to v2.2 on our system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] baolsen commented on issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
Posted by GitBox <gi...@apache.org>.
baolsen commented on issue #16770:
URL: https://github.com/apache/airflow/issues/16770#issuecomment-944336930
Hi there
If you are using an "assume_role"/similar with the aws hook then the
credentials _should_ automatically refresh.
I tested it using normal boto3 clients but have not tested using a waiter,
like what the ECS operator uses.
Maybe try add some logging lines to the aws base hook if you can, to see if
the expected assume role and refresh credentials methods are being called.
Regards
On Fri, 15 Oct 2021, 15:22 ss-nef, ***@***.***> wrote:
> Hello,
>
> I am not able to understand how to enable the automatic refresh when using
> the ECSOperator to orchestrate Fargate Containers.
>
> The container I am running is taking more than one hour to complete its
> task, which is why I am getting the following error when using the
> ECSOperator:
>
> Traceback (most recent call last):
> File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
> self._prepare_and_execute_task_with_callbacks(context, task)
> File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
> result = self._execute_task(context, task_copy)
> File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
> result = task_copy.execute(context=context)
> File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 220, in execute
> self._wait_for_task_ended()
> File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 293, in _wait_for_task_ended
> waiter.wait(cluster=self.cluster, tasks=[self.arn])
> File "/home/airflow/.local/lib/python3.8/site-packages/botocore/waiter.py", line 53, in wait
> Waiter.wait(self, **kwargs)
> File "/home/airflow/.local/lib/python3.8/site-packages/botocore/waiter.py", line 334, in wait
> raise WaiterError(
> botocore.exceptions.WaiterError: Waiter TasksStopped failed: An error occurred (ExpiredTokenException): The security token included in the request is expired```
>
>
> what setting do I need to set to enable automtic refreshing, so that the waiter can wait for longer than one hour?
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/16770#issuecomment-944298277>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGLEIY2ML7DTDURJDTVAH5LUHATKDANCNFSM47WJHO5A>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ss-nef edited a comment on issue #16770: AWS hook should automatically refresh credentials when using temporary credentials
Posted by GitBox <gi...@apache.org>.
ss-nef edited a comment on issue #16770:
URL: https://github.com/apache/airflow/issues/16770#issuecomment-946550142
Thank you for your quick reply,
is it possible that the changes you made in airflow/providers/amazon/aws/hooks/base_aws.py were actually not shipped with Airflow version v2.1.2?
Because I am looking through your code changes and it seems like your changes are missing in v2.1.
I am asking because in this issue https://github.com/apache/airflow/issues/8301 it is said the changes should have already come with v2.1.
Maybe I just need to upgrade airflow to v2.2 on our system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org