You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/17 21:19:38 UTC
[GitHub] [airflow] RNHTTR opened a new issue, #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
RNHTTR opened a new issue, #24534:
URL: https://github.com/apache/airflow/issues/24534
### Apache Airflow version
2.3.2 (latest released)
### What happened
If a DAG with a `schedule_interval` set to `@daily` is paused, and the DAG is triggered by clicking the "play" button --> "Trigger DAG", two DAG runs will simultaneously be triggered: One for the current data interval and one for the manual run.
### What you think should happen instead
Maybe an alert with confirmation should pop up and inform the user that the DAG run will be for the current data interval rather than the current datetime?
### How to reproduce
1. Pause a DAG that hasn't run in the current data interval.
2. Trigger the DAG by clicking the "play" button then clicking "Trigger DAG"
### Operating System
Debian 11
### Versions of Apache Airflow Providers
n/a
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rohit-mobstac commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
rohit-mobstac commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1245228648
This is not a fix, but something I have noticed. Unpausing a scheduled DAG will automatically kickstart the dag run. Maybe you do not have to explicitly trigger the paused dag and just enable the dag. @RNHTTR
Seems like a good issue. Will look into this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rohit-mobstac commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
rohit-mobstac commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1245172181
Is this issue solved? I can see the same issue happening in AWS MWAA as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1245329614
Just did :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Triggering DAG from UI Creates 2 DAG runs if the DAG was paused [airflow]
Posted by "vaibhavnsingh (via GitHub)" <gi...@apache.org>.
vaibhavnsingh commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1839081587
probably I was not able to put my problem statement clearly. So basically When a DAG is paused the next run keeps on refreshing based on the interval. So when the DAG is enabled, it executes the last run which had expired. But in my case the next run doesn't refresh. It is the time when it was paused. So when enabled it executes this old run instance and new.
As you see below current time is 04-DEC-2023 10:30 but the next run still showing the time as
<img width="1708" alt="image" src="https://github.com/apache/airflow/assets/20038172/6967a38c-2cd5-400e-8299-ae5e4b53d141">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RNHTTR commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
RNHTTR commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1164729634
I will circle back in a couple weeks, but I don't have time at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1245208326
I think it would have been closed by now if it was. But if you want to take a stab and attempt to fix it @rohit-mobstac - feel free. @bispaul -> I have missed your comment before, but I think I have no specific guidance on it, this just need diagnosis why it happens and fixing, just regular contribution workflow :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RNHTTR commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1583329970
@bispaul sorry for the delayed response. No, a DAG must be unpaused in order to be run. You could prevent non-manual runs from being triggered by removing the schedule interval, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bispaul commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
bispaul commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1378657356
Hi @eladkal , I am working on it unfortunately I haven't been able to replicate the issue. Can anybody help me replicate the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1245211838
And just to explain - I have no idea why it happens, and 95% of solving this issue is tracking why it happened. Fixing it will be probably one-liner, so any guidance that I could spare would be equal to fixing the problem (but I have other pressing issues and this one is more suited for someone who might want to take first steps into contributing to Airflow, as this is at most annoying edge, case, nothign super important to fix immediately and it can wait for someone to pick it up (this is why it is marked as "good first issue").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Triggering DAG from UI Creates 2 DAG runs if the DAG was paused [airflow]
Posted by "vaibhavnsingh (via GitHub)" <gi...@apache.org>.
vaibhavnsingh commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1832149901
**Schedule Created time - 5:52 PM (frequency – 4 mins, starting 6:00 PM)**
1st trigger - 6:04 PM
2nd trigger - 6:08 PM
**Disabled - 6:09 PM**
**Enabled - 6:22 PM**
3rd trigger - 6:22 PM (had to be skipped) – which was supposed to be at 6:16 PM
4th trigger - 6:22 PM (had to be skipped) - – which was supposed to be at 6:20 PM
So when enabled at 6:22 it executed 2 runs, I have observed that sometime airflow is executing last 2 instances between the interval and finally it executes a schedule when interval expires at 6:24pm. Anyone with same observation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bispaul commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
bispaul commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1179732105
Hi @potiuk , Can I help out on this? I probably will need just a little guidance on getting started.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1378380250
@bispaul are you still working on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Triggering DAG from UI Creates 2 DAG runs if the DAG was paused [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1832613820
I no longer think this is actually a bug for DAG runs that are within a DAG's schedule interval. It makes sense that when a DAG is unpaused, the run for the current DAG interval is triggered. Triggering a DAG run via the "play" button unpauses the corresponding DAG, so this behavior is normal.
What could be improved is that this flow is not obvious. I think it'd be nice if there was some indication that the DAG will be unpaused and if one or more (if `catchup=True`) _scheduled_ DAG runs will be triggered upon the action, that should be made clear to the user. Either with a modal or maybe a toast or something.
@bispaul are you still working on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bispaul commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
bispaul commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1245265856
Hi @potiuk please assign it to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1159824046
Interesting edge-case. Would you like to take a stab at it @RNHTTR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RNHTTR commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
RNHTTR commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1378984059
I can still reproduce on 2.5.0:
If you have a paused DAG, and the DAG hasn't run for the current [data interval](https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html#data-interval), manually triggering the DAG using the UI will simultaneously trigger two DAG runs:
1. The manual run with a `run_id` similar to `manual__2023-01-11T15:27:32.643794+00:00`
2. The scheduled run, because the DAG needs to become unpaused to schedule tasks for the manual run
If you can't reproduce it, I recommend confirming that there are no dagruns within the current data interval.
I think the easiest change that would be generally good would be:
If a DAG is paused and a dagrun will be scheduled if the DAG is unpaused, _and_ the DAG is attempted to be triggered by clicking the play button from the UI, the UI should show a warning indicating that a scheduled run will also be triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Triggering DAG from UI Creates 2 DAG runs if the DAG was paused [airflow]
Posted by "vaibhavnsingh (via GitHub)" <gi...@apache.org>.
vaibhavnsingh commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1834423211
I do agree to your comment @RNHTTR . So once i resume the immediate interval which has expired would execute, that is what I have the understanding but in my case it is executing n and n-1 interval. That is kind of weird as I do not see that same behaviour on my different setup. In the other setup it executes only n.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1378700967
It could be that this was already fixed in main?
@RNHTTR can you please check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RNHTTR commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by GitBox <gi...@apache.org>.
RNHTTR commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1245384747
@rohit-mobstac If DAG is turned on within the DAG's data interval, it will trigger a DAG run when you unpause the DAG, which I think is to be expected. I think there should be some kind of logic that blocks that scheduled DAG run to run if you're triggering it from the UI. Or there should at least be the option to block it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bispaul commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by "bispaul (via GitHub)" <gi...@apache.org>.
bispaul commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1465172614
Hi @RNHTTR wanted to know isn't there a way to run the job manually without the job getting unpaused?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ajithshetty commented on issue #24534: Triggering DAG from UI Creates 2 DAG runs if the DAG was paused
Posted by "ajithshetty (via GitHub)" <gi...@apache.org>.
ajithshetty commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1583295934
I could this happening in the MWAA 2.5.1.
1. Set the schedule to @once
2. Make sure the DAG is Paused
3. Deleted all the previous Runs(if any)
4. Trigger the DAG from UI
But in case of the Schedule "None", by triggering the job from UI will only trigger 1 dag run.
As its already discussed above,
When you run the PAUSED DAG, it will:
1. Turn on the DAG
2. default schedule will kick in and execute accordingly
3. Execute another run as per User's request(from the UI)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Triggering DAG from UI Creates 2 DAG runs if the DAG was paused [airflow]
Posted by "bispaul (via GitHub)" <gi...@apache.org>.
bispaul commented on issue #24534:
URL: https://github.com/apache/airflow/issues/24534#issuecomment-1834453438
Hi @RNHTTR , I can start working on it now that it is clear a message needs to be displayed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org