You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "karenbraganz (via GitHub)" <gi...@apache.org> on 2024/12/12 20:49:09 UTC
[I] Task retries are not run with soft_fail=True [airflow]
karenbraganz opened a new issue, #44893:
URL: https://github.com/apache/airflow/issues/44893
### Apache Airflow version
2.10.3
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Using `soft_fail=True` with sensors does not allow retries.
If a sensor is going to fail and `soft_fail=True`, the task will skip instead of failing. However, if retries are set for the task, those are not executed since an [AirflowSkipExecution is raised in the code](https://github.com/apache/airflow/blob/35087d7d10714130cc3e9e9730e34b07fc56938d/airflow/sensors/base.py#L293), and the task's status is marked as `skipped`, not `up_for_retry`.
### What you think should happen instead?
If the task has retries left, its status should be set to `up_for_retry`. The `AirflowSkipException` should only be raised if no retries are left.
### How to reproduce
I have seen a few reports of this and was also able to reproduce it with the below DAG code.
```
import datetime
from airflow.decorators import dag
from airflow.decorators import task
from airflow.sensors.external_task import ExternalTaskSensor
@dag(start_date=datetime.datetime(2024, 10, 1), schedule="* * * * *", catchup=False)
def dag_2():
ets = ExternalTaskSensor(
task_id="ets",
external_dag_id="non_existent_dag",
soft_fail=True,
retries=3,
poke_interval=10,
timeout=5,
)
ets
dag_2()
```
### Operating System
Debian GNU/Linux 12
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task retries are not run with soft_fail=True [airflow]
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield closed issue #44893: Task retries are not run with soft_fail=True
URL: https://github.com/apache/airflow/issues/44893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task retries are not run with soft_fail=True [airflow]
Posted by "karenbraganz (via GitHub)" <gi...@apache.org>.
karenbraganz commented on issue #44893:
URL: https://github.com/apache/airflow/issues/44893#issuecomment-2558306645
@nathadfield You're right- timeouts are probably not a good application for retries. I thought that all failures including intermittent failures would result in the task getting skipped even when retries are set. After reviewing the code and running a few tests to see if this applies to intermittent issues (by deliberately getting tasks stuck in queued and failing), it seems this isn't the case. When the task got stuck in queued and failed, it was able to retry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task retries are not run with soft_fail=True [airflow]
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on issue #44893:
URL: https://github.com/apache/airflow/issues/44893#issuecomment-2545486339
@karenbraganz I think, in the scenario you describe, this is the desired behaviour. The reason why the task is not going into an `UP_FOR_RETRY` state is because the we have reached the [timeout](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#timeouts).
When this occurs, the sensor task will immediately raise a `AirflowSensorTimeout` exception and fail without retrying.
```
[2024-12-16, 12:09:25 GMT] {external_task.py:274} INFO - Poking for DAG 'non_existent_dag' on 2024-12-16T12:07:00+00:00 ...
[2024-12-16, 12:09:25 GMT] {taskinstance.py:301} INFO - Sensor has timed out; run duration of 10.012994088 seconds exceeds the specified timeout of 5.0.
```
It makes no sense to retry a sensor just for a timeout; instead the timeout should be longer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org