You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pankajastro (via GitHub)" <gi...@apache.org> on 2023/06/21 04:56:44 UTC
[GitHub] [airflow] pankajastro opened a new pull request, #32041: Raise error in for paused dag trigger
pankajastro opened a new pull request, #32041:
URL: https://github.com/apache/airflow/pull/32041
If dag is paused and a user trigger dag externally either with CLI or REST API.
It creates a dagrun with queued state and then stuck in the queued state which is confusing for the user.
Raising an error if dag is paused and if a user is trying to create a dagrun.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1642159455
Thank you for the feedback. closing this since current behaviour is right open PR https://github.com/apache/airflow/pull/32696 to update the docs string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1600779619
> When you trigger a paused DAG from the UI, the DAG is automatically unpaused, should we do the same for the REST API and CLI? I think it will be more consistent to do the same thing
The UI has an option to not unpause, we might as well add the option
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #32041: Raise error in dag trigger for paused dag
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on code in PR #32041:
URL: https://github.com/apache/airflow/pull/32041#discussion_r1237230442
##########
airflow/cli/commands/dag_command.py:
##########
@@ -145,6 +145,11 @@ def dag_backfill(args, dag: list[DAG] | DAG | None = None) -> None:
@cli_utils.action_cli
def dag_trigger(args) -> None:
"""Creates a dag run for the specified dag."""
+ dag = DagModel.get_dagmodel(args.dag_id)
+ if not dag:
+ raise SystemExit(f"DAG: {args.dag_id} does not exist in 'dag' table")
Review Comment:
Maybe we should a test for missing dag as well.
##########
airflow/cli/commands/dag_command.py:
##########
@@ -145,6 +145,11 @@ def dag_backfill(args, dag: list[DAG] | DAG | None = None) -> None:
@cli_utils.action_cli
def dag_trigger(args) -> None:
"""Creates a dag run for the specified dag."""
+ dag = DagModel.get_dagmodel(args.dag_id)
+ if not dag:
+ raise SystemExit(f"DAG: {args.dag_id} does not exist in 'dag' table")
Review Comment:
Maybe we should test for missing dag as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1601092377
> When you trigger a paused DAG from the UI, the DAG is automatically unpaused, should we do the same for the REST API and CLI? I think it will be more consistent to do the same thing
I agree on the consistency part. But changing DAG from pause to unpause silently maybe be surprising for some users. So If we decided to do it I'll prefer to go via some option like what @utkarsharma2 / @uranusjr has suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro closed pull request #32041: Raise error in dag trigger for paused dag
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro closed pull request #32041: Raise error in dag trigger for paused dag
URL: https://github.com/apache/airflow/pull/32041
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1601099759
> > When you trigger a paused DAG from the UI, the DAG is automatically unpaused, should we do the same for the REST API and CLI? I think it will be more consistent to do the same thing
>
> I agree on the consistency part. But changing DAG from pause to unpause silently maybe be surprising for some users. So If we decided to do it I'll prefer to go via some option like what @utkarsharma2 / @uranusjr has suggested.
Since we already have that flag in UI, there's no need to wait. WDYT @uranusjr ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1600311574
> @pankajastro Maybe we can also add an option to force-run both in CLI and API. This option can unpause dag and then run it. WDYT?
>
> Maybe in other PR.
We have another API as well as a CLI command to unpause a DAG. So I'm not sure if it makes sense to add another param in this API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1600357810
A `unpause` flag may make sense, but I’d suggest delay that until someone asks for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1600596656
When you trigger a paused DAG from the UI, the DAG is automatically unpaused, should we do the same for the REST API and CLI? I think it will be more consistent to do the same thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on a diff in pull request #32041: Raise error in dag trigger for paused dag
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on code in PR #32041:
URL: https://github.com/apache/airflow/pull/32041#discussion_r1237215659
##########
airflow/cli/commands/dag_command.py:
##########
@@ -145,6 +145,11 @@ def dag_backfill(args, dag: list[DAG] | DAG | None = None) -> None:
@cli_utils.action_cli
def dag_trigger(args) -> None:
"""Creates a dag run for the specified dag."""
+ dag = DagModel.get_dagmodel(args.dag_id)
+ if not dag:
+ raise SystemExit(f"DAG: {args.dag_id} does not exist in 'dag' table")
+ if dag.is_paused:
+ raise AirflowException(f"DAG with dag_id: {args.dag_id} is paused.")
Review Comment:
it shouldn't. thank you for pointing out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] utkarsharma2 commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1600276673
@pankajastro Maybe we can also add an option to `force-run` both in CLI and API. This option can unpause dag and then run it. WDYT?
Maybe in other PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #32041: Raise error in dag trigger for paused dag
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #32041:
URL: https://github.com/apache/airflow/pull/32041#discussion_r1236538601
##########
airflow/cli/commands/dag_command.py:
##########
@@ -145,6 +145,11 @@ def dag_backfill(args, dag: list[DAG] | DAG | None = None) -> None:
@cli_utils.action_cli
def dag_trigger(args) -> None:
"""Creates a dag run for the specified dag."""
+ dag = DagModel.get_dagmodel(args.dag_id)
+ if not dag:
+ raise SystemExit(f"DAG: {args.dag_id} does not exist in 'dag' table")
+ if dag.is_paused:
+ raise AirflowException(f"DAG with dag_id: {args.dag_id} is paused.")
Review Comment:
Why do these raise different exception classes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1603117597
> Since we already have that flag in UI, there's no need to wait. WDYT @uranusjr ?
The default behaviour in UI is `unpause=True` then raising the error does not make sense wdyt?.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #32041: Raise error in dag trigger for paused dag
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #32041:
URL: https://github.com/apache/airflow/pull/32041#issuecomment-1603137276
I think the existing API behaviour is right - UIs often combine multiple steps in to a single action.
We certainly shouldn't raise an error just because the dag is paused - they're are valid use cases where creating runs for a paused dag makes sense (and the UI unpausing had caused annoyance for some users)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org