You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vgl-grin (via GitHub)" <gi...@apache.org> on 2025/11/14 10:35:03 UTC
[I] task_instance_heartbeat_timeout is ignored when >4 [airflow]
vgl-grin opened a new issue, #58307:
URL: https://github.com/apache/airflow/issues/58307
### Apache Airflow version
Other Airflow 2/3 version (please specify below)
### If "Other Airflow 2/3 version" selected, which one?
3.0.6
### What happened?
i have a test sleep-10-min DAG.
my settings were:
task_instance_heartbeat_sec = 0
**task_instance_heartbeat_timeout = 300**
task_instance_heartbeat_timeout_detection_interval = 10.0
i was running this DAG with values 300, 1, 2, 4, 5, 6, 10. between runs airflow containers were respawned.
except values 1, 2 DAG was not terminated.
for the cases 1 and 2 duration were around 9 - 19 seconds.
### What you think should happen instead?
tasks shoud be terminated after task_instance_heartbeat_timeout.
### How to reproduce
DAG:
`from airflow.sdk import dag
from airflow.providers.standard.operators.bash import BashOperator
from datetime import datetime
@dag(start_date=datetime(2021, 1, 1), schedule="@once", catchup=False)
def sleep_dag():
t1 = BashOperator(
task_id="sleep_10_minutes",
bash_command="sleep 600",
)
sleep_dag()
`
be sure you have no further settings override on docker-compose.yaml
set settings in airflow.cfg to section scheduler:
my settings were:
task_instance_heartbeat_sec = 0
**task_instance_heartbeat_timeout = 300**
task_instance_heartbeat_timeout_detection_interval = 10.0
### Operating System
Ubuntu 24.04.3 LTS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] task_instance_heartbeat_timeout is ignored when >2 [airflow]
Posted by "hwang-cadent (via GitHub)" <gi...@apache.org>.
hwang-cadent commented on issue #58307:
URL: https://github.com/apache/airflow/issues/58307#issuecomment-3533399247
It seems the query in _find_task_instances_without_heartbeats() doesn't handle NULL last_heartbeat_at values.
see line 2479 in scheduler_job_runner.py
.where(
TI.state.in_((TaskInstanceState.RUNNING, TaskInstanceState.RESTARTING)),
TI.last_heartbeat_at < limit_dttm, # This excludes NULL values!
)
When task_instance_heartbeat_sec = 0, tasks don't send heartbeats, so last_heartbeat_at is NULL. In SQL, NULL < anything evaluates to NULL (not True), so rows with NULL last_heartbeat_at are excluded from the WHERE clause.The query should include tasks where last_heartbeat_at is NULL OR older than the timeout. like
.where(
TI.state.in_((TaskInstanceState.RUNNING, TaskInstanceState.RESTARTING)),
or_(
TI.last_heartbeat_at.is_(None), # Handle NULL case
TI.last_heartbeat_at < limit_dttm
),
)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] task_instance_heartbeat_timeout is ignored when >4 [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #58307:
URL: https://github.com/apache/airflow/issues/58307#issuecomment-3532086712
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] task_instance_heartbeat_timeout is ignored when >2 [airflow]
Posted by "mykola-shyshov (via GitHub)" <gi...@apache.org>.
mykola-shyshov commented on issue #58307:
URL: https://github.com/apache/airflow/issues/58307#issuecomment-3539133722
as for me it looks like misconfiguration.
> task_instance_heartbeat_sec:
description: |
The frequency (in seconds) at which the LocalTaskJob should send heartbeat signals to the
scheduler to notify it's still alive. If this value is set to 0, the heartbeat interval will default
to the value of ``[scheduler] task_instance_heartbeat_timeout``.
So heartbeat is still going on so task should not be terminated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] task_instance_heartbeat_timeout is ignored when >2 [airflow]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on issue #58307:
URL: https://github.com/apache/airflow/issues/58307#issuecomment-4438697061
Investigating the heartbeat-timeout cleanup query in `_find_task_instances_without_heartbeats` — opened #66773 to fix one underlying SQL-predicate bug that can leave RUNNING TIs stuck. Captured trace below.
`_find_task_instances_without_heartbeats` filters with `TI.last_heartbeat_at < limit_dttm`. In SQL three-valued logic that predicate evaluates to NULL (not TRUE) when the row's `last_heartbeat_at IS NULL`, so the row is never returned and the TI never gets purged. Deterministic reproducer captured by adding a regression test that creates a RUNNING TI with `last_heartbeat_at=NULL` and `start_date` past the timeout window:
```
FAILED ::test_find_and_purge_task_instances_without_heartbeats_null_last_heartbeat
AssertionError: assert ti.key not in MockExecutor.running
(cleanup query silently skipped the NULL row; TI stays in RUNNING forever)
```
A NULL `last_heartbeat_at` for a RUNNING TI happens on Airflow 2 → 3 upgrade (migration `0045` added the column as `nullable=True` without a backfill) and on restored DB dumps / manual DB intervention. The codebase already acknowledges this inside `scheduler_job_runner.py:2854-2856` (`adopt_or_reset_orphaned_tasks`); the PR adds the matching fallback to the heartbeat-cleanup path.
If your reproducer was on a fresh 3.0.6 cluster with no upgrade history this may not fully cover what you're seeing — the timeout-ignored-for-values-greater-than-2 symptom could be a separate issue (e.g. heartbeat-cadence vs detection-interval interaction). If you can attach a snapshot of the stuck TI's row (`SELECT state, last_heartbeat_at, start_date FROM task_instance WHERE dag_id = 'sleep_dag' AND state = 'running'`) that would help confirm whether the NULL path is the same one you hit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org