You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nathadfield (via GitHub)" <gi...@apache.org> on 2026/01/27 16:37:49 UTC

[PR] Fix deferrable sensors not respecting soft_fail on timeout [airflow]

nathadfield opened a new pull request, #61132:
URL: https://github.com/apache/airflow/pull/61132

   ## Description                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                          
   Fixes a regression in Airflow 3.x where deferrable sensors with `soft_fail=True` fail with `AirflowSensorTimeout` instead of being marked as SKIPPED when they timeout. This breaks compatibility with Airflow 2.x behavior and user expectations.                                                                     
                                                                                                                                                                                                                                                                                                                          
   ## Root Cause                                                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                                          
   In `BaseSensorOperator.resume_execution()`, the exception handling order was incorrect:                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                          
   1. `TaskDeferralTimeout` was caught and immediately converted to `AirflowSensorTimeout`                                                                                                                                                                                                                                
   2. The `soft_fail`/`never_fail` checks only caught `AirflowException` and `TaskDeferralError`                                                                                                                                                                                                                          
   3. By the time these checks ran, the timeout was already converted, causing the flags to be ignored                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                          
   ## Solution                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                          
   - Use nested try-except blocks to convert `TaskDeferralTimeout` → `AirflowSensorTimeout` while ensuring both exceptions are caught by the outer handler                                                                                                                                                                
   - Check `soft_fail` and `never_fail` flags before raising `AirflowSensorTimeout`                                                                                                                                                                                                                                       
   - Align error messages with the `execute()` method for consistency                                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                                                                                                          
   ## Changes                                                                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                                                                          
   **`task-sdk/src/airflow/sdk/bases/sensor.py`:**                                                                                                                                                                                                                                                                        
   - Modified `resume_execution()` to use nested exception handling                                                                                                                                                                                                                                                       
   - Changed to separate `soft_fail` and `never_fail` checks with explicit error messages                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                          
   **`task-sdk/tests/task_sdk/bases/test_sensor.py`:**                                                                                                                                                                                                                                                                    
   - Added tests for timeout scenarios with `soft_fail=True/False`                                                                                                                                                                                                                                                        
   - Added tests for timeout scenarios with `never_fail=True`                                                                                                                                                                                                                                                             
   - Added tests for trigger failure scenarios with `soft_fail=True/False`                                                                                                                                                                                                                                                
   - Added tests for trigger failure scenarios with `never_fail=True`                                                                                                                                                                                                                                                     
   - Fixed existing tests to use `TriggerFailureReason` enum values
   ##### Was generative AI tooling used to co-author this PR?                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                                                                          
   - [X] Yes (please specify the tool below)                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                          
   <!--                                                                                                                                                                                                                                                                                                                   
   Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix deferrable sensors not respecting soft_fail on timeout [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil merged PR #61132:
URL: https://github.com/apache/airflow/pull/61132


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix deferrable sensors not respecting soft_fail on timeout [airflow]

Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on code in PR #61132:
URL: https://github.com/apache/airflow/pull/61132#discussion_r2735567747


##########
task-sdk/src/airflow/sdk/bases/sensor.py:
##########
@@ -250,12 +250,15 @@ def run_duration() -> float:
 
     def resume_execution(self, next_method: str, next_kwargs: dict[str, Any] | None, context: Context):
         try:

Review Comment:
   Thanks! Comment added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix deferrable sensors not respecting soft_fail on timeout [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61132:
URL: https://github.com/apache/airflow/pull/61132#issuecomment-3844494660

   ### Backport successfully created: v3-1-test
   
   <table>
                   <tr>
                       <th>Status</th>
                       <th>Branch</th>
                       <th>Result</th>
                   </tr>
                   <tr>
                       <td>✅</td>
                       <td>v3-1-test</td>
                       <td><a href="https://github.com/apache/airflow/pull/61421"><img src="https://img.shields.io/badge/PR-61421-blue" alt="PR Link"></a></td>
                   </tr>
               </table>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix deferrable sensors not respecting soft_fail on timeout [airflow]

Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #61132:
URL: https://github.com/apache/airflow/pull/61132#discussion_r2733932165


##########
task-sdk/src/airflow/sdk/bases/sensor.py:
##########
@@ -250,12 +250,15 @@ def run_duration() -> float:
 
     def resume_execution(self, next_method: str, next_kwargs: dict[str, Any] | None, context: Context):
         try:

Review Comment:
   I would add a comment here explaining why there are nested try/except blocks. Perhaps something like:
   
   `Nested try/except ensures TaskDeferralTimeout conversion does not bypass soft_fail / never_fail handling.`
   
   Makes it easier to reason about. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix deferrable sensors not respecting soft_fail on timeout [airflow]

Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on PR #61132:
URL: https://github.com/apache/airflow/pull/61132#issuecomment-3818598860

   Hey folks! If any of you get the time to review this PR soon I'd be most appreciative. Many thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org