You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "developer-rpai (via GitHub)" <gi...@apache.org> on 2026/10/12 04:27:53 UTC

[PR] Fix Dag run timeout marking running mapped task instances as skipped [airflow]

developer-rpai opened a new pull request, #74567:
URL: https://github.com/apache/airflow/pull/74567

   When a Dag run times out, the scheduler marks all unfinished task instances SKIPPED. A mapped task whose inner tasks were interrupted then misleadingly reports success, hiding the timeout. This change marks running task instances (RUNNING, DEFERRED, UP_FOR_RESCHEDULE, AWAITING_INPUT) as FAILED on timeout, while task instances that never started stay SKIPPED and teardown tasks are left untouched so they can still run — mirroring the "mark run failed" rule. Fixes #37332.
   
   Test plan: new test_dag_timed_out_mapped_task_states covers a mapped task with running/pending TIs plus a teardown at timeout; updated test_do_schedule_max_active_runs_dag_timed_out asserts FAILED for the running TI. Could not run tests locally (no Python env); relying on CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Dag run timeout marking running mapped task instances as skipped [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed pull request #74567: Fix Dag run timeout marking running mapped task instances as skipped
URL: https://github.com/apache/airflow/pull/74567


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Dag run timeout marking running mapped task instances as skipped [airflow]

Posted by "yuseok89 (via GitHub)" <gi...@apache.org>.
yuseok89 commented on code in PR #74567:
URL: https://github.com/apache/airflow/pull/74567#discussion_r4244823188


##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -3266,7 +3266,20 @@ def _schedule_dag_run(
                 default=None,
             )
             for task_instance in unfinished_task_instances:
-                task_instance.set_state(TaskInstanceState.SKIPPED, session=session)
+                if task_instance.state in (
+                    TaskInstanceState.RUNNING,
+                    TaskInstanceState.DEFERRED,
+                    TaskInstanceState.UP_FOR_RESCHEDULE,
+                    TaskInstanceState.AWAITING_INPUT,
+                ):
+                    # The timeout interrupted a running task instance; mark it FAILED so a
+                    # mapped task with unfinished inner tasks reflects the failure instead
+                    # of appearing successful.
+                    task_instance.set_state(TaskInstanceState.FAILED, session=session)
+                elif not dag.task_dict[task_instance.task_id].is_teardown:
+                    # Task instances that never started stay SKIPPED. Teardowns are left
+                    # alone so they can still run.
+                    task_instance.set_state(TaskInstanceState.SKIPPED, session=session)

Review Comment:
   It looks like this teardown won't run. The run is already set to FAILED above, and the scheduler only looks at running runs, so a pending teardown would stay `None` or `scheduled` in a finished run. A running teardown would also be set to FAILED by the first branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Dag run timeout marking running mapped task instances as skipped [airflow]

Posted by "developer-rpai (via GitHub)" <gi...@apache.org>.
developer-rpai commented on code in PR #74567:
URL: https://github.com/apache/airflow/pull/74567#discussion_r4321908536


##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -3266,7 +3266,20 @@ def _schedule_dag_run(
                 default=None,
             )
             for task_instance in unfinished_task_instances:
-                task_instance.set_state(TaskInstanceState.SKIPPED, session=session)
+                if task_instance.state in (
+                    TaskInstanceState.RUNNING,
+                    TaskInstanceState.DEFERRED,
+                    TaskInstanceState.UP_FOR_RESCHEDULE,
+                    TaskInstanceState.AWAITING_INPUT,
+                ):
+                    # The timeout interrupted a running task instance; mark it FAILED so a
+                    # mapped task with unfinished inner tasks reflects the failure instead
+                    # of appearing successful.
+                    task_instance.set_state(TaskInstanceState.FAILED, session=session)
+                elif not dag.task_dict[task_instance.task_id].is_teardown:
+                    # Task instances that never started stay SKIPPED. Teardowns are left
+                    # alone so they can still run.
+                    task_instance.set_state(TaskInstanceState.SKIPPED, session=session)

Review Comment:
   Thanks for looking closely. You are right that the teardown never executes once the run has timed out; that is inherent to the timeout path and this change does not affect it either way. On the state it ends up in: the timeout branch in _schedule_dag_run collects all unfinished task instances of the run, and the loop that marks them does not filter by task type, so a pending teardown (state None) is included and is set to SKIPPED in the same pass that fails the run. The run only reaches FAILED on a timeout through this same branch, so there is no ordering where the run is already finished and the teardown is visited later and left dangling. That end state is what the test pins. If the test comment reads as claiming more than that, I am happy to reword it or split the teardown assertion out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Dag run timeout marking running mapped task instances as skipped [airflow]

Posted by "developer-rpai (via GitHub)" <gi...@apache.org>.
developer-rpai commented on PR #74567:
URL: https://github.com/apache/airflow/pull/74567#issuecomment-6366417710

   Thanks @shahar1 for the pointer — I hadn't seen that #37332 was resolved by #74883, or that #74809 documents timeout->skipped as intended. Understood that changing that rule needs a separate discussion first; if I pursue it, I'll open a discussion laying out the mapped-task-instance case before proposing any code change. Appreciate the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Dag run timeout marking running mapped task instances as skipped [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #74567:
URL: https://github.com/apache/airflow/pull/74567#issuecomment-6358833599

   Closing: #37332 was resolved by #74883 (Grid view secondary states), and #74809 documents timeout→skipped as intended behavior. Changing that rule needs a separate discussion first.
   
   ---
   Drafted-by: Claude Code (Opus 5.5); reviewed by @shahar1 before posting


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org