You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "tim-habitat (via GitHub)" <gi...@apache.org> on 2023/07/13 07:56:20 UTC
[GitHub] [airflow] tim-habitat opened a new issue, #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`
tim-habitat opened a new issue, #32580:
URL: https://github.com/apache/airflow/issues/32580
### Apache Airflow version
2.6.3
### What happened
It seems that with the new `aws` provider package, when using the `deferable` keyword in the `EcsRunTaskOperator` - the `execution_timeout` is ignored and the task is killed from another timeout, the trigger timeout seems to be `timeout=timedelta(seconds=self.waiter_max_attempts * self.waiter_delay + 60)`.
Also, it seems when the `trigger` fires that timeout - it seems the task return "success" even though it hasn't finished.
### What you think should happen instead
The `execution_timeout` should be used in the trigger timeout, or at least a warning if that timeout is overriden or is smaller
### How to reproduce
Run an `EcsRunTaskOperator` task with deferable mode, put a large `execution_timeout` and a small number of `waiter_retries`. The task should terminates based on the `trigger` timing out before the `execution_timeout` is up.
### Operating System
linux ubuntu
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable` [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-3209276807
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vandonr-amz commented on issue #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`
Posted by "vandonr-amz (via GitHub)" <gi...@apache.org>.
vandonr-amz commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-1686781218
using the search, it seems like it was reported a looong time ago in #19929 (though at the same time as an other issue too)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable` [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`
URL: https://github.com/apache/airflow/issues/32580
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tim-habitat commented on issue #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`
Posted by "tim-habitat (via GitHub)" <gi...@apache.org>.
tim-habitat commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-1683635652
It seems this https://github.com/apache/airflow/pull/32589 pr has fixed the fact that the task doesn't return success anymore, however - it doesn't seem to have solved the "killing of the task part".
If the trigger times out by "number of retries" the ecs task is not killed. I should expect that if the airflow task fail, the ecs task gets killed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nathadfield commented on issue #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-1634030921
#31881 was the PR that introduced this. @vandonr-amz are you able to determine whether this is indeed a bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable` [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-2933792680
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable` [airflow]
Posted by "thesuperzapper (via GitHub)" <gi...@apache.org>.
thesuperzapper commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-2144022586
Here is the generic issue for the fact that `on_kill` is not supported when using deferable operators:
- https://github.com/apache/airflow/issues/36090
There are also some workarounds described in that issue for provider/operator developers before its fully fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-1683139404
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vandonr-amz commented on issue #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`
Posted by "vandonr-amz (via GitHub)" <gi...@apache.org>.
vandonr-amz commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-1686777657
ah ok, I missed the `It seems this doesn't kill the task either.` line. Several issues in one here !
It is a known issue yeah that the `on_kill` of operators doesn't get called in deferrable mode :/
Maybe it'd be worth opening a new more general issue about this. We can try to fix it in every operator by piecing code together to replicate the expected behavior, but it's a suboptimal solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vandonr-amz commented on issue #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`
Posted by "vandonr-amz (via GitHub)" <gi...@apache.org>.
vandonr-amz commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-1634660611
After testing it, I can confirm that there was a bug where success was returned after max_attempts had been reached
however, setting a short `execution_timeout` works well and returns an error even when the task is in the triggerer
and setting a long execution timeout works well too, but it doesn't "override" the max attempts.
And I don't think it should be the case, except maybe if `max_attempts` is not specified but `execution timeout` is ? But it's a complex behavior, and I'm not sure it'd be a good idea.
Maybe another way to see this is that the defaults that are set for the ecs operator are too short, and we should increase those values so that less people would run into timeouts "out of the box".
We currently allow 100 pings with 6 sec delay by default, so any task taking more than 10 minutes needs an override of those params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vandonr-amz commented on issue #32580: `execution_timeout` seems to be ignored in `EcsRunTaskOperator` when using `deferable`
Posted by "vandonr-amz (via GitHub)" <gi...@apache.org>.
vandonr-amz commented on issue #32580:
URL: https://github.com/apache/airflow/issues/32580#issuecomment-1634580545
Hmm that's interesting because `execution_timeout` is a property of the base operator (i.e. of all operators), which is usually handled "automatically", but I don't think that plays well with deferrable...
Some operators have included it manually in the `defer()` call, with varying logic, some others don't consider it.
I'm also confused by what's the expected behavior in this report. Setting a large `execution_timeout` is not going to make the trigger run longer than `waiter_retries`.
I actually see 3 bugs being reported:
- tasks are successful when triggers time out (probably not what we want ?)
- we hit the trigger timeout even though it should end on max_attempts by itself 60 seconds before ?
- `execution_timeout` is not handled properly on defer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org