You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Andrushika (via GitHub)" <gi...@apache.org> on 2026/09/03 18:33:02 UTC

[PR] Enforce execution_timeout from the task supervisor [airflow]

Andrushika opened a new pull request, #72493:
URL: https://github.com/apache/airflow/pull/72493

   ## Why
   
   `execution_timeout` is only enforced by a SIGALRM handler inside the task process. When that process is stuck in native code, deadlocked on a lock inherited across fork, or hit by SIGSEGV, the handler never runs and the task stays running forever (#57174, #57712, #45930, #71707).
   
   ## What
   
   The task process sends a new `SetExecutionTimeout` message to the supervisor right before it arms its own alarm, so both clocks start at the same point. If the task has not reported a terminal state within 5 seconds after the timeout, the supervisor sends SIGTERM to the process group, and SIGKILL 5 seconds later.
   
   This keeps the task runner's SIGALRM handler instead of "moving it", which is a bit different from what #53337 asks for. The reason is that `on_kill`, the retry policy and the failure callbacks can only run inside the task process, and dropping the handler would skip them on every timeout.
   
   closes: #53337
   related: #57174, #57712, #45930, #71707
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Fable 5.1)
   
   Generated-by: Claude Code (Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enforce execution_timeout from the task supervisor [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.
seanmuth commented on PR #72493:
URL: https://github.com/apache/airflow/pull/72493#issuecomment-5716025936

   `SetExecutionTimeout` is only sent right before entering `execute()`:
   
   ```python
   if timeout_seconds <= 0:
       raise AirflowTaskTimeout()
   SUPERVISOR_COMMS.send(SetExecutionTimeout(timeout_seconds=timeout_seconds))
   with timeout(timeout_seconds):
       result = ctx.run(execute, context=context)
   ```
   
   So `_execution_timeout_enforce_at` doesn't exist until that message arrives — the supervisor-side backstop is inert for anything before `execute()` starts: bundle loading, parsing, bundle refresh. A wedge there (e.g. the fork-orphaned-lock case named directly in `_handle_execution_timeout_if_needed`'s own docstring, #71707) runs with no deadline at all regardless of `execution_timeout`, since the message establishing one hasn't been sent yet.
   
   Given the mechanism already exists and doesn't depend on the task process's cooperation, would it make sense for the supervisor to arm a (longer, generic) deadline of its own covering the pre-`execute()` window too — bundle load/parse start to `SetExecutionTimeout` arrival — rather than leaving that window fully unenforced? Not asking for scope creep on this PR necessarily, just flagging it doesn't close #71707 as fully as the `related:` line implies.
   
   ---
   Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enforce execution_timeout from the task supervisor [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #72493:
URL: https://github.com/apache/airflow/pull/72493#issuecomment-5839385820

   Hello @Andrushika - thank you for your contributions to Apache Airflow!
   
   The Airflow community has introduced a limit of **5 open pull requests at a time** for contributors without write access to the repository. You currently have 7 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
   
   * #69713 - Preserve AirflowTaskTimeout when timeout logging fails
   * #70135 - Fix mapped tasks does not validate task_id
   * #70630 - Reject consecutive dots in Dag and task IDs in the Task SDK
   * #72493 - Enforce execution_timeout from the task supervisor
   * #73241 - Add breeze verify to list local checks for changed files
   * #73464 - Add task state store API to the Java SDK
   
   These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
   
   * #69937 - Warn on suspicious Dag and task IDs at Java SDK build time
   
   This is **not** a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.
   
   What we ask you to do is to make your **first prioritization decision**: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or `gh pr reopen <PR_NUMBER> --repo apache/airflow`. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.
   
   While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.
   
   Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.
   
   ---
   Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enforce execution_timeout from the task supervisor [airflow]

Posted by "Andrushika (via GitHub)" <gi...@apache.org>.
Andrushika commented on PR #72493:
URL: https://github.com/apache/airflow/pull/72493#issuecomment-5728745614

   Good catch, you are right! I fixed the docstring and updated the description.
   
   Thanks for the idea of a deadline for the pre-`execute()` window -- it does sound useful. However, the supervisor does not know `execution_timeout` before the Dag is parsed, so it would need its own config value. 
   I would like to leave it in a separate PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enforce execution_timeout from the task supervisor [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #72493: Enforce execution_timeout from the task supervisor
URL: https://github.com/apache/airflow/pull/72493


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Enforce execution_timeout from the task supervisor [airflow]

Posted by "Andrushika (via GitHub)" <gi...@apache.org>.
Andrushika commented on PR #72493:
URL: https://github.com/apache/airflow/pull/72493#issuecomment-5530485575

   It may also fix related issues like #57174 and #57712, but I have not reproduced those yet, so that still needs confirmation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org