[PR] fix: allow in-flight DagRuns to continue when DAG is paused [airflow]

["waterWang (via GitHub)" <gi...@apache.org>]

waterWang opened a new pull request, #71384:
URL: https://github.com/apache/airflow/pull/71384

   ### Problem
   
   When a DAG is paused while it has an active (running) DagRun — e.g. one started via manual trigger before the pause, or a manual trigger on an already-paused DAG that got an initial burst of tasks dispatched — the scheduler stops scheduling further task instances for that run. The run does not fail and logs no error; it simply stalls indefinitely in `running` state with remaining tasks stuck in `scheduled`, `queued_when=None`, never sent to the executor.
   
   This happens because `is_paused` is checked as a single gate that governs both "should new DagRuns be created" and "should ready task instances in existing DagRuns be scheduled." The two are conflated.
   
   ### Root cause
   
   Two places filter by `is_paused` to block task scheduling, when only DagRun creation should be blocked:
   
   1. **`scheduler_job_runner.py`**: The task scheduling query filters out ALL task instances of paused DAGs via `.where(~DM.is_paused)`, including those belonging to already-RUNNING DagRuns.
   
   2. **`dagrun.py` `next_dagruns_to_examine()`**: The method that selects RUNNING DagRuns for scheduling examination filters out all DagRuns of paused DAGs via `DagModel.is_paused == false()`.
   
   ### Fix
   
   - Remove `.where(~DM.is_paused)` from the task scheduling query in `scheduler_job_runner.py`
   - Remove `DagModel.is_paused == false()` from `next_dagruns_to_examine()` in `dagrun.py`
   
   **Preserved**: `DagModel.dags_needing_dagruns()` still filters by `is_paused == false()`, so paused DAGs cannot create **new** DagRuns. The `_update_dag_run_state_for_paused_dags` timer still runs every 60s for edge cases. The QUEUED DagRun filter in `dagrun.py` (line 818) also retains its `is_paused` check, so queued runs for paused DAGs are not promoted.
   
   ### Related issue
   
   Closes #71381


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix: allow in-flight DagRuns to continue when DAG is paused [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil closed pull request #71384: fix: allow in-flight DagRuns to continue when DAG is paused
URL: https://github.com/apache/airflow/pull/71384


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix: allow in-flight DagRuns to continue when DAG is paused [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil commented on PR #71384:
URL: https://github.com/apache/airflow/pull/71384#issuecomment-5259031261

   Closing this as part of a cleanup of a large batch of PRs opened in quick succession from this account.
   
   18 PRs have been opened here in the past two weeks and none have merged. Several show signs of being generated and submitted without review: #71432 and #71433 are the same change across the same five files, opened two minutes apart, and several titles carry a leaked agent identifier that other contributors already flagged as garbled text on #70629 and #71322.
   
   Airflow is maintained by volunteers. Every PR costs reviewer time and CI capacity, so a high volume of unvetted submissions has a real cost to the project.
   
   You are welcome to keep contributing. Please open one change at a time, run it locally against the tests, and read the [contributors' guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst) before submitting. If you think a specific change here is correct, comment with the reasoning and a maintainer can reopen it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix: allow in-flight DagRuns to continue when DAG is paused [airflow]

["raphaelauv (via GitHub)" <gi...@apache.org>]

raphaelauv commented on PR #71384:
URL: https://github.com/apache/airflow/pull/71384#issuecomment-5250097617

   this is a big change
   
   what about adding this in a new feature and not changing the current behavior of the pause button ( status ) 
   
   maybe you could add a new toggle button ( stop starting new dag_run )
   
   so we would have 
   
   unpause / pause / pause_new_run
   
   wdyt ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org