You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/14 12:40:40 UTC
[GitHub] [airflow] danielito-orteguita opened a new issue #21562: Do not retry task if non-retriable error occurs
danielito-orteguita opened a new issue #21562:
URL: https://github.com/apache/airflow/issues/21562
### Description
There should exist a mechanism to easily tell airflow whether a given failed task should be retried or not.
My ideal solution would be for the Python operator to have a parameter to specify a list of non-retriable exceptions. If the task fails with any of those, it should be marked as failed and not be retried.
Eg:
`
class MyCustomException(Exception):
pass
task = PythonOperator(
task_id = "my_task",
python_callable = my_function,
retries = 3,
retry_delay = datetime.timedelta(hours=24),
non_retriable_exceptions = [ TypeError, MyCustomException ]
)
`
### Use case/motivation
I have a DAG whose tasks may fail due to mostly two kinds of errors.
One of them is retriable, because it is caused by a daily quota enforced by an external API I make calls to, and which resets after 24h. It thus makes sense to retry the task with a 24h delay when this error is encountered, because the quota will have been refreshed by then.
The other error is non-retriable and requires human intervention, as it occurs because of "dirty" external data out of my control. However, if I configure the DAG retries according to the first error, the DAG won't continue its work until [24h x retries] later, which is far too long. This is especially inconvenient because it prevents further task and DAG executions from starting, because I can only have one concurrent task at a time (because of the limited quota in the aforementioned API). If the tasks simply failed and not be retried, the task could fail, and airflow could move on.
A possible workaround would be adding branching operators and deciding whether to go on or not depending on what exception is thrown. However, this would only conceal the problem. I want the task to actually fail (and the DAG at large).
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21562: Do not retry task if non-retriable error occurs
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21562:
URL: https://github.com/apache/airflow/issues/21562#issuecomment-1039040353
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #21562: Do not retry task if non-retriable error occurs
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #21562:
URL: https://github.com/apache/airflow/issues/21562#issuecomment-1039044996
This functionality already exist.
Use [`AirflowFailException`](https://github.com/apache/airflow/blob/28378d867afaac497529bd2e1d2c878edf66f460/airflow/exceptions.py#L93-L94)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] danielito-orteguita commented on issue #21562: Do not retry task if non-retriable error occurs
Posted by GitBox <gi...@apache.org>.
danielito-orteguita commented on issue #21562:
URL: https://github.com/apache/airflow/issues/21562#issuecomment-1039086072
Just what I needed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #21562: Do not retry task if non-retriable error occurs
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #21562:
URL: https://github.com/apache/airflow/issues/21562#issuecomment-1039044996
Use [`AirflowFailException`](https://github.com/apache/airflow/blob/28378d867afaac497529bd2e1d2c878edf66f460/airflow/exceptions.py#L93-L94)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #21562: Do not retry task if non-retriable error occurs
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #21562:
URL: https://github.com/apache/airflow/issues/21562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org