You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "kaxil (via GitHub)" <gi...@apache.org> on 2025/08/29 22:08:03 UTC

[PR] Fix scheduler processing of cleared running tasks stuck in `RESTARTING` state [airflow]

kaxil opened a new pull request, #55084:
URL: https://github.com/apache/airflow/pull/55084

closes #55045

Clearing a DAG with running tasks causes them to get stuck in `RESTARTING` state indefinitely, with continuous heartbeat timeout events. This is a regression in Airflow 3.x where the scheduler fails to process executor completion events for `RESTARTING` tasks.

### Error Symptoms
- Tasks remain in `RESTARTING` state after clearing
- Continuous heartbeat timeout events: `404 Not Found` or `409 Conflict`
- Tasks never transition to retry or completion
- Executor reports `SUCCESS` but task state doesn't update

## Root Cause

The scheduler's `_process_executor_events()` function has two gatekeeping mechanisms that filter which executor events get processed:

1. **First filter (lines 769-775)**: Only processes events for certain states
2. **Second filter (lines 859-864)**: Validates task instance state matches executor event

Both filters were missing `TaskInstanceState.RESTARTING`, causing the scheduler to ignore when tasks in `RESTARTING` state were successfully terminated by the executor.

### Historical Context

The `RESTARTING` state was introduced in Airflow 2.2.0 (PR #16681) to handle cleared running tasks gracefully:
- Prevent cleared tasks from transitioning to `FAILED` (like `SHUTDOWN` did)
- Allow tasks to be retried instead of marked as failed
- Originally relied on "zombie detection" mechanism in Airflow 2.x

In Airflow 3.x, the zombie detection mechanism was removed with the transition from Job-based to Task SDK architecture, but the scheduler was never updated to handle `RESTARTING` tasks in executor events.

### Why not immediately set state to None

The `RESTARTING` state serves a critical purpose in the clearing workflow:

1. **Race condition prevention**: When `clear_task_instances()` is called, it immediately:
- Generates a new `ti.id` via `prepare_db_for_next_try()`
- Sets state to `RESTARTING`
- Commits it to DB

2. **Running task process continues**: The task process keeps running with the old `ti.id` and attempts heartbeating, which fails with `404 Not Found` since the database now has a new `ti.id`.

3. **Executor termination**: The executor eventually kills the running process and reports `SUCCESS` for the terminated task.

4. **Scheduler coordination**: The `RESTARTING` state signals to the scheduler that this task was intentionally cleared and should be retried (not failed) when the executor confirms termination.

Setting the state directly to `None` during clearing would lose this coordination mechanism and could lead to race conditions where the scheduler processes a "completed" task before the executor has terminated the running process.

## Future Considerations

This fix resolves the immediate regression but raises design questions for future improvements:

1. **Callbacks**: Should `on_retry_callback` be triggered for cleared tasks?
2. **Email notifications**: Should `email_on_retry` be sent?

These questions are deferred to maintain scope and can be addressed in follow-up discussions if needed.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix scheduler processing of cleared running tasks stuck in `RESTARTING` state [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.

kaxil commented on PR #55084:
URL: https://github.com/apache/airflow/pull/55084#issuecomment-3239106445

   > >  Should on_retry_callback be triggered for cleared tasks?
   > 
   > 
   > 
   > In Airflow 2 when I cleared a task I did not expect on_retry_callback to be executed. I don't think I've seen anyone expecting this. Why would this be an issue specifically now in Airflow 3?
   
   Mainly cause in 2.x, it first went to UP_FOR_RETRY state after RESTARTING so on_retry_callback could have been executed if set.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix scheduler processing of cleared running tasks stuck in `RESTARTING` state [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.

kaxil commented on PR #55084:
URL: https://github.com/apache/airflow/pull/55084#issuecomment-3239105169

   > This will work, but is there any way we could do this without relying on executor events in the future?
   
   Yeah, possible to improve in future. Currently, we mainly need it to only run after the current task attempt is killed. Otherwise they could run in parallel i.e new attempt can start running before the first one is fully terminated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix scheduler processing of cleared running tasks stuck in `RESTARTING` state [airflow]

Posted by "rawwar (via GitHub)" <gi...@apache.org>.

rawwar commented on PR #55084:
URL: https://github.com/apache/airflow/pull/55084#issuecomment-3238934365

   > Callbacks: Should on_retry_callback be triggered for cleared tasks?
   Email notifications: Should email_on_retry be sent?
   
   I think we should do both only in cases where task was cleared while it was in running state.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix scheduler processing of cleared running tasks stuck in `RESTARTING` state [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.

kaxil merged PR #55084:
URL: https://github.com/apache/airflow/pull/55084


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix scheduler processing of cleared running tasks stuck in `RESTARTING` state [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #55084:
URL: https://github.com/apache/airflow/pull/55084#issuecomment-3239105678

   ### Backport failed to create: v3-0-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/17341674477'> Run details </a>
   
   <table>
       <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>❌</td>
           <td>v3-0-test</td>
           <td><a href="https://github.com/apache/airflow/commit/f9adaadfd55e0c7d3a34c6715d9b9f2739cedbf3"><img src='https://img.shields.io/badge/Commit-f9adaad-red' alt='Commit Link'></a></td>
       </tr>
   </table>
   
   You can attempt to backport this manually by running:
   
   ```bash
   cherry_picker f9adaad v3-0-test
   ```
   
   This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
   the files that need manual conflict resolution.
   
   After you have resolved the conflicts, you can continue the backport process by running:
   
   ```bash
   cherry_picker --continue
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix scheduler processing of cleared running tasks stuck in `RESTARTING` state [airflow]

Posted by "rawwar (via GitHub)" <gi...@apache.org>.

rawwar commented on PR #55084:
URL: https://github.com/apache/airflow/pull/55084#issuecomment-3238933591

   Tested this locally and it works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix scheduler processing of cleared running tasks stuck in `RESTARTING` state [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.

eladkal commented on PR #55084:
URL: https://github.com/apache/airflow/pull/55084#issuecomment-3239029097

   >  Should on_retry_callback be triggered for cleared tasks?
   
   In Airflow 2 when I cleared a task I did not expect on_retry_callback to be executed. I don't think I've seen anyone expecting this. Why would this be an issue specifically now in Airflow 3?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org