You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "waterWang (via GitHub)" <gi...@apache.org> on 2026/08/06 19:57:28 UTC
[PR] fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail [airflow]
waterWang opened a new pull request, #71256:
URL: https://github.com/apache/airflow/pull/71256
## Description
When `soft_fail=True`, deferred sensors currently skip on **all** exceptions, while non-deferred (`poke`/`reschedule` mode) sensors only skip on timeout-related exceptions (`AirflowSensorTimeout`, `AirflowTaskTimeout`, `AirflowFailException`) and let unexpected errors propagate.
This behavior gap means operators cannot distinguish between a sensor timeout (legitimate skip) and a sensor misconfiguration or unexpected error when using deferred mode with `soft_fail=True`.
## Fix
In `BaseSensorOperator.resume_execution()`, split the exception handling into two tiers matching the `execute()` method:
1. **Timeout-related exceptions** (`AirflowSensorTimeout`, `AirflowTaskTimeout`, `AirflowFailException`) → `soft_fail` and `never_fail` apply (skip)
2. **Other exceptions** (`AirflowException`, `TaskDeferralError`) → only `never_fail` applies (not `soft_fail`)
## Related issue
Closes #71255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail [airflow]
Posted by "raghavan-arvind (via GitHub)" <gi...@apache.org>.
raghavan-arvind commented on code in PR #71256:
URL: https://github.com/apache/airflow/pull/71256#discussion_r3766899139
##########
task-sdk/src/airflow/sdk/bases/sensor.py:
##########
@@ -258,12 +258,16 @@ def resume_execution(self, next_method: str, next_kwargs: dict[str, Any] | None,
return super().resume_execution(next_method, next_kwargs, context)
except TaskDeferralTimeout as e:
raise AirflowSensorTimeout(*e.args) from e
- except (AirflowException, TaskDeferralError) as e:
+ except (AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException) as e:
if self.soft_fail:
raise AirflowSkipException("Skipping due to soft_fail is set to True.") from e
if self.never_fail:
raise AirflowSkipException("Skipping due to never_fail is set to True.") from e
raise
+ except (AirflowException, TaskDeferralError) as e:
+ if self.never_fail:
+ raise AirflowSkipException("Skipping due to never_fail is set to True.") from e
+ raise
Review Comment:
Using logic here for reference:
https://github.com/apache/airflow/blob/84d873a36f6089be9fc1358b1a718ebe8fa42331/task-sdk/src/airflow/sdk/bases/sensor.py#L205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail [airflow]
Posted by "raghavan-arvind (via GitHub)" <gi...@apache.org>.
raghavan-arvind commented on PR #71256:
URL: https://github.com/apache/airflow/pull/71256#issuecomment-5208768814
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail [airflow]
Posted by "raghavan-arvind (via GitHub)" <gi...@apache.org>.
raghavan-arvind commented on PR #71256:
URL: https://github.com/apache/airflow/pull/71256#issuecomment-5295620302
cc: @raphaelauv if you have any thoughts, I see you upvoted as well, running into issues in #71625 because of this parity mismatch I believe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #71256:
URL: https://github.com/apache/airflow/pull/71256#issuecomment-5259038769
Closing this as part of a cleanup of a large batch of PRs opened in quick succession from this account.
18 PRs have been opened here in the past two weeks and none have merged. Several show signs of being generated and submitted without review: #71432 and #71433 are the same change across the same five files, opened two minutes apart, and several titles carry a leaked agent identifier that other contributors already flagged as garbled text on #70629 and #71322.
Airflow is maintained by volunteers. Every PR costs reviewer time and CI capacity, so a high volume of unvetted submissions has a real cost to the project.
You are welcome to keep contributing. Please open one change at a time, run it locally against the tests, and read the [contributors' guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst) before submitting. If you think a specific change here is correct, comment with the reasoning and a maintainer can reopen it.
Note for @raghavan-arvind: this PR carried your approval. It is being closed as part of the batch cleanup above, not on the merits of the diff. Happy to reopen if you would rather see it land.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail [airflow]
Posted by "raghavan-arvind (via GitHub)" <gi...@apache.org>.
raghavan-arvind commented on PR #71256:
URL: https://github.com/apache/airflow/pull/71256#issuecomment-5267532359
Hi @kaxil, can you take another look at this PR? It does seem to be a parity mismatch worth fixing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail [airflow]
Posted by "raghavan-arvind (via GitHub)" <gi...@apache.org>.
raghavan-arvind commented on code in PR #71256:
URL: https://github.com/apache/airflow/pull/71256#discussion_r3766914063
##########
task-sdk/src/airflow/sdk/bases/sensor.py:
##########
@@ -258,12 +258,16 @@ def resume_execution(self, next_method: str, next_kwargs: dict[str, Any] | None,
return super().resume_execution(next_method, next_kwargs, context)
except TaskDeferralTimeout as e:
raise AirflowSensorTimeout(*e.args) from e
- except (AirflowException, TaskDeferralError) as e:
+ except (AirflowSensorTimeout, AirflowTaskTimeout, AirflowFailException) as e:
if self.soft_fail:
raise AirflowSkipException("Skipping due to soft_fail is set to True.") from e
if self.never_fail:
raise AirflowSkipException("Skipping due to never_fail is set to True.") from e
raise
+ except (AirflowException, TaskDeferralError) as e:
+ if self.never_fail:
+ raise AirflowSkipException("Skipping due to never_fail is set to True.") from e
+ raise
Review Comment:
Considering whether we should have full parity, just catch "Exception" in the last block, and filter out `silent_failure`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed pull request #71256: fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail
URL: https://github.com/apache/airflow/pull/71256
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(sensors): only skip timeout-related exceptions on deferred sensor soft_fail [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #71256:
URL: https://github.com/apache/airflow/pull/71256#issuecomment-5295814851
this is one of the thing I wanted to be fix before airflow v3 https://github.com/apache/airflow/pull/41463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org