You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "PNL0 (via GitHub)" <gi...@apache.org> on 2025/03/10 08:57:31 UTC

[PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

PNL0 opened a new pull request, #47557:
URL: https://github.com/apache/airflow/pull/47557

   The bug happened because the API function used to patch a DAG, only updated the is_paused field in the DagModel structure. This made the state of a running DAG to stay on running and consequently the duration will keep on increasing even though the DAG is paused.
   
   To fix this, I added some logic to the API function to also update the DAG state, depending on the value of is_paused field (paused makes the state change from running to queued, and vice versa).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2008091106


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +247,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.QUEUED)

Review Comment:
   > What about if you pause a running DAG? Maybe you can check the behaviour in Airflow 2 UI
   
   If you pause a running dag, then the dag run state will be stuck on running and as result the duration keeps on increasing permanently, which it's not the desired behavior for pause functionality. 
   
   This behavior happens in Airflow 2 and 3 UIs (also if I'm not mistaken, the issue was reported, when airflow 3 UI wasn't released).
   
   That's why I think the dag run state can't stay on running when the DAG is paused. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r1987876382


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -235,6 +236,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+    if is_paused is not None:
+        active_dag_runs = session.scalars(
+            select(DagRun).filter(
+                DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+            )
+        ).all()
+
+        for dag_run in active_dag_runs:
+            if is_paused:

Review Comment:
   I think there is a problem here. It cannot get into else here since the parent `if` is already ensuring it isn't `None`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2007550043


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +247,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.QUEUED)

Review Comment:
   > > I disagree with this. Why will a dag run go from running to queued state?
   > 
   > I think that if a DAG is paused, then the dag run shouldn't continue to run (which implies a change in its state from running to other state that keeps the dag run paused).
   > 
   > I chose queued state, because a similar behavior already happens in Airflow, when you trigger a paused dag (dag run state goes to queued).
   
   What about if you pause a running DAG? Maybe you can check the behaviour in Airflow 2 UI



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2014954702


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +247,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.QUEUED)
+        else:
+            if dag_run.state == DagRunState.QUEUED:
+                dag_run.set_state(DagRunState.RUNNING)
+

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to failed, when paused, to avoid the increment of their duration [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun closed pull request #47557: Fix #44443: DAGs state change from running to failed, when paused, to avoid the increment of their duration
URL: https://github.com/apache/airflow/pull/47557


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to failed, when paused, to avoid the increment of their duration [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #47557:
URL: https://github.com/apache/airflow/pull/47557#issuecomment-2820698011

   Closing for now, feel free to reopen if needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #47557:
URL: https://github.com/apache/airflow/pull/47557#issuecomment-2709853328

   Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
   - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
   - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
   - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
   Apache Airflow is a community-driven project and together we are making it better 🚀.
   In case of doubts contact the developers at:
   Mailing List: dev@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2014954702


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +247,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.QUEUED)
+        else:
+            if dag_run.state == DagRunState.QUEUED:
+                dag_run.set_state(DagRunState.RUNNING)
+

Review Comment:
   I agree. 
   In the update, I removed that code, since it wasn't necessary, as you said.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to failed, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2021788969


##########
airflow-core/src/airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +248,29 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.FAILED)
+            running_task_instances = session.scalars(
+                select(TaskInstance).filter(
+                    TaskInstance.dag_id == dag_run.dag_id,
+                    TaskInstance.run_id == dag_run.run_id,
+                    TaskInstance.state == TaskInstanceState.RUNNING,
+                )
+            ).all()
+            for task_instance in running_task_instances:
+                task_instance.set_state(TaskInstanceState.FAILED)

Review Comment:
   Hello @ephraimbuddy @pierrejeambrun,
   
   I created a post in the dev mailing list, as you suggested.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to failed, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on PR #47557:
URL: https://github.com/apache/airflow/pull/47557#issuecomment-2816102984

   Hello @ephraimbuddy @pierrejeambrun,
   
   From what I understood from the discussion in the dev mailing list, the current behavior of pause functionality is expected (staying on running and the duration incrementing).
   
   Right now, I don't really know what I should do in this bug fix, since the cluster activity is gone in the Airflow 3 UI and fixing top 5 running dags is not an option for this version.
   
   Should I keep going with this bug fix? I'm open to any ideas/suggestions you may have.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r1987875223


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -235,6 +236,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+    if is_paused is not None:

Review Comment:
   ```suggestion
       if is_paused:
   ```



##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -235,6 +236,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+    if is_paused is not None:
+        active_dag_runs = session.scalars(
+            select(DagRun).filter(
+                DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+            )
+        ).all()
+
+        for dag_run in active_dag_runs:
+            if is_paused:

Review Comment:
   I think there is a problem here. It cannot get into else here since the parent `if` is already ensuring it isn't None.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on PR #47557:
URL: https://github.com/apache/airflow/pull/47557#issuecomment-2713865837

   > I am not sure if this follows this comment in the issue stated in the MR title.
   > 
   > [#44443 (comment)](https://github.com/apache/airflow/issues/44443#issuecomment-2513833985)
   > 
   > > > I'm just into something, if we no need to rush, I will happily raise PR
   > 
   > > Raising a PR to fix the cluster activity (Top 5 running dags) is step forward. Maybe the issue in the title isn't matching with the PR. Could you please also update the description and include `closes:#<issue_number>` or `relates:#<issue_number>`?
   
   I updated the description with `relates: #<issue_number>`.  
   From my interpretation of the issue, the main problem was that the DAGs would be stuck on running even though they were paused. 
   I think fixing this problem also fixes (or at least contains) the other subsequently problems, such as the cluster activity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2007495901


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +247,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.QUEUED)

Review Comment:
   > I disagree with this. Why will a dag run go from running to queued state?
   
   I think that if a DAG is paused, then the dag run shouldn't continue to run (which implies a change in its state from running to other state that keeps the dag run paused). 
   
   I chose queued state, because a similar behavior already happens in Airflow, when you trigger a paused dag (dag run state goes to queued).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2020595110


##########
airflow-core/src/airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +248,29 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.FAILED)
+            running_task_instances = session.scalars(
+                select(TaskInstance).filter(
+                    TaskInstance.dag_id == dag_run.dag_id,
+                    TaskInstance.run_id == dag_run.run_id,
+                    TaskInstance.state == TaskInstanceState.RUNNING,
+                )
+            ).all()
+            for task_instance in running_task_instances:
+                task_instance.set_state(TaskInstanceState.FAILED)

Review Comment:
   Dev mailing list address is `dev@airflow.apache.org`. And you can take a look the archives there to see how things are discussed:
   https://lists.apache.org/list.html?dev@airflow.apache.org



##########
airflow-core/src/airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +248,29 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.FAILED)
+            running_task_instances = session.scalars(
+                select(TaskInstance).filter(
+                    TaskInstance.dag_id == dag_run.dag_id,
+                    TaskInstance.run_id == dag_run.run_id,
+                    TaskInstance.state == TaskInstanceState.RUNNING,
+                )
+            ).all()
+            for task_instance in running_task_instances:
+                task_instance.set_state(TaskInstanceState.FAILED)

Review Comment:
   Dev mailing list email address is `dev@airflow.apache.org`. And you can take a look the archives there to see how things are discussed:
   https://lists.apache.org/list.html?dev@airflow.apache.org



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r1989016938


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -235,6 +236,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+    if is_paused is not None:
+        active_dag_runs = session.scalars(
+            select(DagRun).filter(
+                DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+            )
+        ).all()
+
+        for dag_run in active_dag_runs:
+            if is_paused:

Review Comment:
   > I think there is a problem here. It cannot get into else here since the parent `if` is already ensuring it isn't `None`.
   
   My idea, with the if-else clause, is to cover the cases when the user pauses the dag (is_paused = true) and unpauses the dag (is_paused = false, so the else clause). 
   
   Maybe this became confusing because of the parent if-clause (to check the None), but has I said in the other comment, I think that I can safely remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2014969111


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +247,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.QUEUED)

Review Comment:
   > What do you think about failing all running dagruns once the dag is paused? Skipping those that are not running. Just the way we handle dagrun timeout
   
   I think it's a good idea. 
   I tried to implement what you described in my updated commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2013869649


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +247,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.QUEUED)
+        else:
+            if dag_run.state == DagRunState.QUEUED:
+                dag_run.set_state(DagRunState.RUNNING)
+

Review Comment:
   I don't think you need that. The scheduler will automatically Run this dagrun if it is unpaused.
   (`_start_queued_dagruns`, `get_queued_dag_runs_to_set_running`)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2010184605


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +247,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.QUEUED)

Review Comment:
   What do you think about failing all running dagruns once the dag is paused? Skipping those that are not running. Just the way we handle dagrun timeout



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on PR #47557:
URL: https://github.com/apache/airflow/pull/47557#issuecomment-2755676816

   Hello @pierrejeambrun @ephraimbuddy,
   
   After I saw your comments, I agree that changing the state from running to failed may be a better approach to the problem, since it reproduces a behavior that already exists in dagrun timeout.
   
   I just updated my commit (code and tests) to implement the new logic, and I would appreciate any feedback. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2015982948


##########
airflow-core/src/airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +248,29 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.FAILED)
+            running_task_instances = session.scalars(
+                select(TaskInstance).filter(
+                    TaskInstance.dag_id == dag_run.dag_id,
+                    TaskInstance.run_id == dag_run.run_id,
+                    TaskInstance.state == TaskInstanceState.RUNNING,
+                )
+            ).all()
+            for task_instance in running_task_instances:
+                task_instance.set_state(TaskInstanceState.FAILED)

Review Comment:
   As @pierrejeambrun suggested, we should discuss this change in behaviour in dev list. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r2006056724


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -246,6 +247,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+
+    active_dag_runs = session.scalars(
+        select(DagRun).filter(
+            DagRun.dag_id == dag_id, DagRun.state.in_([DagRunState.RUNNING, DagRunState.QUEUED])
+        )
+    ).all()
+
+    for dag_run in active_dag_runs:
+        if is_paused:
+            dag_run.set_state(DagRunState.QUEUED)

Review Comment:
   I disagree with this. Why will a dag run go from running to queued state?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on PR #47557:
URL: https://github.com/apache/airflow/pull/47557#issuecomment-2738272255

   Hello, @bugraoz93 @pierrejeambrun @ephraimbuddy.
   
   Do you have any new feedback for my PR?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix #44443: DAGs state change from running to queued, when paused, to avoid the increment of their duration [airflow]

Posted by "PNL0 (via GitHub)" <gi...@apache.org>.
PNL0 commented on code in PR #47557:
URL: https://github.com/apache/airflow/pull/47557#discussion_r1988998517


##########
airflow/api_fastapi/core_api/routes/public/dags.py:
##########
@@ -235,6 +236,23 @@ def patch_dag(
 
     data = patch_body.model_dump(include=fields_to_update, by_alias=True)
 
+    is_paused = data.get("is_paused")
+    if is_paused is not None:

Review Comment:
   I think I should remove the if-clause, because is_paused is a boolean that will always be true or false (since it represents the value of is_paused field from the DagModel, after the update, it's impossible to be None). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org