You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vgl-grin (via GitHub)" <gi...@apache.org> on 2025/11/14 10:35:03 UTC

[I] task_instance_heartbeat_timeout is ignored when >4 [airflow]

vgl-grin opened a new issue, #58307:
URL: https://github.com/apache/airflow/issues/58307

   ### Apache Airflow version
   
   Other Airflow 2/3 version (please specify below)
   
   ### If "Other Airflow 2/3 version" selected, which one?
   
   3.0.6
   
   ### What happened?
   
   i have a test sleep-10-min DAG.
   my settings were:
   task_instance_heartbeat_sec = 0
   **task_instance_heartbeat_timeout = 300**
   task_instance_heartbeat_timeout_detection_interval = 10.0
   
   i was running this DAG with values 300, 1, 2, 4, 5, 6, 10. between runs airflow containers were respawned.
   except values 1, 2 DAG was not terminated.
   for the cases 1 and 2 duration were around 9 - 19 seconds. 
   
   ### What you think should happen instead?
   
   tasks shoud be terminated after task_instance_heartbeat_timeout.
   
   ### How to reproduce
   
   DAG:
   `from airflow.sdk import dag
   from airflow.providers.standard.operators.bash import BashOperator
   from datetime import datetime
   
   
   @dag(start_date=datetime(2021, 1, 1), schedule="@once", catchup=False)
   def sleep_dag():
       t1 = BashOperator(
           task_id="sleep_10_minutes",
           bash_command="sleep 600",
       )
   
   
   sleep_dag()
   `
   be sure you have no further settings override on docker-compose.yaml
   set settings in airflow.cfg to section scheduler:
   my settings were:
   task_instance_heartbeat_sec = 0
   **task_instance_heartbeat_timeout = 300**
   task_instance_heartbeat_timeout_detection_interval = 10.0
   
   
   ### Operating System
   
   Ubuntu 24.04.3 LTS
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] task_instance_heartbeat_timeout is ignored when >2 [airflow]

Posted by "hwang-cadent (via GitHub)" <gi...@apache.org>.
hwang-cadent commented on issue #58307:
URL: https://github.com/apache/airflow/issues/58307#issuecomment-3533399247

   It seems the query in _find_task_instances_without_heartbeats() doesn't handle NULL last_heartbeat_at values.
   
   see line 2479 in scheduler_job_runner.py
   
   .where(
       TI.state.in_((TaskInstanceState.RUNNING, TaskInstanceState.RESTARTING)),
       TI.last_heartbeat_at < limit_dttm,  #  This excludes NULL values!
   )
   
   When task_instance_heartbeat_sec = 0, tasks don't send heartbeats, so last_heartbeat_at is NULL. In SQL, NULL < anything evaluates to NULL (not True), so rows with NULL last_heartbeat_at are excluded from the WHERE clause.The query should include tasks where last_heartbeat_at is NULL OR older than the timeout. like 
   
   .where(
       TI.state.in_((TaskInstanceState.RUNNING, TaskInstanceState.RESTARTING)),
       or_(
           TI.last_heartbeat_at.is_(None),  # Handle NULL case
           TI.last_heartbeat_at < limit_dttm
       ),
   )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] task_instance_heartbeat_timeout is ignored when >4 [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #58307:
URL: https://github.com/apache/airflow/issues/58307#issuecomment-3532086712

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] task_instance_heartbeat_timeout is ignored when >2 [airflow]

Posted by "mykola-shyshov (via GitHub)" <gi...@apache.org>.
mykola-shyshov commented on issue #58307:
URL: https://github.com/apache/airflow/issues/58307#issuecomment-3539133722

   as for me it looks like misconfiguration. 
   >    task_instance_heartbeat_sec:
         description: |
           The frequency (in seconds) at which the LocalTaskJob should send heartbeat signals to the
           scheduler to notify it's still alive. If this value is set to 0, the heartbeat interval will default
           to the value of ``[scheduler] task_instance_heartbeat_timeout``.
   
   So heartbeat is still going on so task should not be terminated. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] task_instance_heartbeat_timeout is ignored when >2 [airflow]

Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on issue #58307:
URL: https://github.com/apache/airflow/issues/58307#issuecomment-4438697061

   Investigating the heartbeat-timeout cleanup query in `_find_task_instances_without_heartbeats` — opened #66773 to fix one underlying SQL-predicate bug that can leave RUNNING TIs stuck. Captured trace below.
   
   `_find_task_instances_without_heartbeats` filters with `TI.last_heartbeat_at < limit_dttm`. In SQL three-valued logic that predicate evaluates to NULL (not TRUE) when the row's `last_heartbeat_at IS NULL`, so the row is never returned and the TI never gets purged. Deterministic reproducer captured by adding a regression test that creates a RUNNING TI with `last_heartbeat_at=NULL` and `start_date` past the timeout window:
   
   ```
   FAILED ::test_find_and_purge_task_instances_without_heartbeats_null_last_heartbeat
       AssertionError: assert ti.key not in MockExecutor.running
       (cleanup query silently skipped the NULL row; TI stays in RUNNING forever)
   ```
   
   A NULL `last_heartbeat_at` for a RUNNING TI happens on Airflow 2 → 3 upgrade (migration `0045` added the column as `nullable=True` without a backfill) and on restored DB dumps / manual DB intervention. The codebase already acknowledges this inside `scheduler_job_runner.py:2854-2856` (`adopt_or_reset_orphaned_tasks`); the PR adds the matching fallback to the heartbeat-cleanup path.
   
   If your reproducer was on a fresh 3.0.6 cluster with no upgrade history this may not fully cover what you're seeing — the timeout-ignored-for-values-greater-than-2 symptom could be a separate issue (e.g. heartbeat-cadence vs detection-interval interaction). If you can attach a snapshot of the stuck TI's row (`SELECT state, last_heartbeat_at, start_date FROM task_instance WHERE dag_id = 'sleep_dag' AND state = 'running'`) that would help confirm whether the NULL path is the same one you hit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org