You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "YoannAbriel (via GitHub)" <gi...@apache.org> on 2026/03/10 10:12:35 UTC

[PR] fix(scheduler): set end_date on tasks skipped by dagrun timeout [airflow]

YoannAbriel opened a new pull request, #63250:
URL: https://github.com/apache/airflow/pull/63250

   ## Problem
   
   When a DAG run times out via `dagrun_timeout`, unfinished task instances are marked `SKIPPED` but their `end_date` is never set. The UI computes duration as `now - start_date`, so skipped tasks show a continuously increasing duration — confusing and incorrect.
   
   ## Root Cause
   
   In `SchedulerJobRunner._schedule_dag_run`, the timeout handler sets `task_instance.state = TaskInstanceState.SKIPPED` but doesn't touch `end_date`. Without an `end_date`, the UI falls back to computing live duration.
   
   ## Fix
   
   Set `end_date = timezone.utcnow()` alongside the state change for all tasks skipped by dagrun timeout. Added a test that creates a timed-out DAG run with running tasks and verifies `end_date` is populated after the scheduler processes it.
   
   Closes: #58536
   
   <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code
   
   Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   ---
   
   * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(scheduler): set end_date on tasks skipped by dagrun timeout [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63250:
URL: https://github.com/apache/airflow/pull/63250#issuecomment-4034429821

   Did you actually check if it works? Or is that change just PR-ed without actually running some real tests? Can you show screenshots and logs from the execution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(scheduler): set end_date on tasks skipped by dagrun timeout [airflow]

Posted by "YoannAbriel (via GitHub)" <gi...@apache.org>.
YoannAbriel commented on PR #63250:
URL: https://github.com/apache/airflow/pull/63250#issuecomment-4042253948

   Reproduced and verified on Airflow 3.1.8 (standalone, SequentialExecutor). Test DAG with `dagrun_timeout=timedelta(seconds=30)` and 3 tasks chained: `quick_task` (5s) -> `slow_task` (120s) -> `final_task` (5s).
   
   **Before fix (unpatched 3.1.8):**
   
   | Task | State | end_date |
   |------|-------|----------|
   | quick_task | success | 2026-03-11 21:05:05 |
   | slow_task | skipped | None |
   | final_task | None | None |
   
   **After fix:**
   
   | Task | State | end_date |
   |------|-------|----------|
   | quick_task | success | 2026-03-11 21:09:21 |
   | slow_task | skipped | 2026-03-11 21:09:45 |
   | final_task | None | None (never started, expected) |
   
   ![Test evidence](https://raw.githubusercontent.com/YoannAbriel/YoannAbriel.github.io/master/pr-63250-evidence-v2.png)
   
   Two lines in `_schedule_dag_run` — `now = timezone.utcnow()` and `task_instance.end_date = now` before merging skipped tasks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(scheduler): set end_date on tasks skipped by dagrun timeout [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63250:
URL: https://github.com/apache/airflow/pull/63250#issuecomment-4034433398

   explanation: This is a **vital** part of airflow in order to modify scheduler you need to know exactly what you are doing and actually test it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(scheduler): set end_date on tasks skipped by dagrun timeout [airflow]

Posted by "YoannAbriel (via GitHub)" <gi...@apache.org>.
YoannAbriel commented on PR #63250:
URL: https://github.com/apache/airflow/pull/63250#issuecomment-4034739694

   Fair — unit tests only so far. Will set up a local environment with a DAG that hits the timeout path and share logs/screenshots.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(scheduler): set end_date on tasks skipped by dagrun timeout [airflow]

Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #63250:
URL: https://github.com/apache/airflow/pull/63250#issuecomment-4764226387

   This should not set only `end_date`. The timeout path creates terminal skipped TIs but bypasses normal duration bookkeeping, so task-instance APIs and table views reading `TI.duration` can still show blank/stale duration. Please use the same state transition helper or explicitly set duration from `start_date` and `now`, and extend the test to assert `duration`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(scheduler): set end_date on tasks skipped by dagrun timeout [airflow]

Posted by "YoannAbriel (via GitHub)" <gi...@apache.org>.
YoannAbriel commented on PR #63250:
URL: https://github.com/apache/airflow/pull/63250#issuecomment-4042257078

   Reproduced and verified on Airflow 3.1.8 (standalone, SequentialExecutor). Test DAG with `dagrun_timeout=timedelta(seconds=30)` and 3 tasks chained: `quick_task` (5s) -> `slow_task` (120s) -> `final_task` (5s).
   
   **Before fix (unpatched 3.1.8):**
   
   | Task | State | end_date |
   |------|-------|----------|
   | quick_task | success | 2026-03-11 21:05:05 |
   | slow_task | skipped | None |
   | final_task | None | None |
   
   **After fix:**
   
   | Task | State | end_date |
   |------|-------|----------|
   | quick_task | success | 2026-03-11 21:09:21 |
   | slow_task | skipped | 2026-03-11 21:09:45 |
   | final_task | None | None (never started, expected) |
   
   Two lines in `_schedule_dag_run` — `now = timezone.utcnow()` and `task_instance.end_date = now` before merging skipped tasks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org