You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "karenbraganz (via GitHub)" <gi...@apache.org> on 2025/11/20 17:58:47 UTC
[I] Task duration keeps increasing for skipped tasks due to DAG timeout [airflow]
karenbraganz opened a new issue, #58536:
URL: https://github.com/apache/airflow/issues/58536
### Discussed in https://github.com/apache/airflow/discussions/53868
<div type='discussions-op-text'>
<sup>Originally posted by **Szjs-z** July 29, 2025</sup>
Hi Airflow community,
I've encountered an issue where task duration keeps increasing for tasks that are skipped due to DAG timeout:
**Problem Description:**
When a DAG run times out due to `dagrun_timeout`, the unfinished tasks are marked as `SKIPPED`. However, in the Web UI, the task duration continues to increase even though the task state is `SKIPPED` :
<img width="770" height="428" alt="Screenshot 2025-07-29 at 17 42 11" src="https://github.com/user-attachments/assets/720b243e-d915-4275-a615-b09a31bbbdb5" />
**Root Cause Analysis:**
From code inspection in `airflow/jobs/scheduler_job.py`, the `_schedule_dag_run` method handles DAG timeout by:
```python
if (
dag_run.start_date
and dag.dagrun_timeout
and dag_run.start_date < timezone.utcnow() - dag.dagrun_timeout
):
dag_run.set_state(DagRunState.FAILED)
unfinished_task_instances = session.scalars(
select(TI)
.where(TI.dag_id == dag_run.dag_id)
.where(TI.run_id == dag_run.run_id)
.where(TI.state.in_(State.unfinished))
)
for task_instance in unfinished_task_instances:
task_instance.state = TaskInstanceState.SKIPPED
session.merge(task_instance)
session.flush()
self.log.info("Run %s of %s has timed-out", dag_run.run_id, dag_run.dag_id)
```
1. Setting DAG run state to FAILED
2. Marking unfinished tasks as SKIPPED
3. But it doesn't set `end_date` for these task instances
I'm wondering if this is a known issue or if there's a recommended way to handle this.
Thanks,
Chris Zhao</div>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task duration keeps increasing for skipped tasks due to DAG timeout [airflow]
Posted by "steveahnahn (via GitHub)" <gi...@apache.org>.
steveahnahn commented on issue #58536:
URL: https://github.com/apache/airflow/issues/58536#issuecomment-3567041916
Opened a PR to address this issue in #58588
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task duration keeps increasing for skipped tasks due to DAG timeout [airflow]
Posted by "karenbraganz (via GitHub)" <gi...@apache.org>.
karenbraganz commented on issue #58536:
URL: https://github.com/apache/airflow/issues/58536#issuecomment-3559368087
I have also been able to reproduce this issue on Airflow 2.11. I agree with Chris' analysis on the root cause.
- The `set_state` method should be called for the unfinished task instances as well instead of directly setting the state in the DBso that the end_date is set.
- The task instance duration should also be set.
- Additionally, callbacks should be run because the task instance could have an on_skipped_callback.
- This code only skips the running task instances. Downstream task instances remain in the None state. Why aren't all task instances skipped?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task duration keeps increasing for skipped tasks due to DAG timeout [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #58536:
URL: https://github.com/apache/airflow/issues/58536#issuecomment-3566699784
See also https://github.com/apache/airflow/issues/30264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task duration keeps increasing for skipped tasks due to DAG timeout [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #58536: Task duration keeps increasing for skipped tasks due to DAG timeout
URL: https://github.com/apache/airflow/issues/58536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org