You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jonathanrainer (via GitHub)" <gi...@apache.org> on 2023/04/14 13:18:51 UTC
[GitHub] [airflow] jonathanrainer opened a new issue, #30642: DAG Run Started Time Appears To Move As the DAG executes, which leads to misleading durations
jonathanrainer opened a new issue, #30642:
URL: https://github.com/apache/airflow/issues/30642
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
This is happening in Airflow 2.3.4, and thus may have been fixed in later versions
We've been running DAGs for a while in our production Airflow setup, however we have a problem which is that as DAGs execute the value that displays next to `Started:` when you click on the DAG Run in Grid View seems to change as the DAG executes. This gets to the point where for some DAGs their start and end times are completely different to the actual execution of that DAG run.
### What you think should happen instead
I would expect that the start time would reflect the start of the execution of the first task in the DAG and then the end would be the end time of the latest executing task in the DAG.
### How to reproduce
Run a DAG that runs for a particularly long period of time and watch as the `Started:` time changes as the DAG executes.
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==4.3.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-docker~=3.1.0
### Deployment
Other
### Deployment details
We're using a modified version of the official Helm Chart deployed in GKE, to run tasks we're using the CeleryExecutor atop Kubernetes in GKE.
### Anything else
Tried to dig through the code for the UI but I didn't get very far so wanted to raise this as a bug to see if there was any more help to be had here.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #30642: DAG Run Started Time Appears To Move As the DAG executes, which leads to misleading durations
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #30642:
URL: https://github.com/apache/airflow/issues/30642#issuecomment-1509429640
I wonder if you cleared a task, distinct from the one shown in the second screenshot as well as any of its upstream tasks, at 15:57:15. This action would have resulted in resetting the state and start date of the DagRun. (necessary to reset the timeout)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jonathanrainer commented on issue #30642: DAG Run Started Time Appears To Move As the DAG executes, which leads to misleading durations
Posted by "jonathanrainer (via GitHub)" <gi...@apache.org>.
jonathanrainer commented on issue #30642:
URL: https://github.com/apache/airflow/issues/30642#issuecomment-1509514013
Ah now that's interesting, there certainly was a task retried at that time, but it would have been cleared with its downstreams not upstreams. That being said the behaviour seems somewhat unexpected. I would have thought that once the first task had started executing the start time of the run would be fixed, and then any retries would simply extend out the finished time of the DAG Run rather than the start as well, because retrying a task doesn't change the past even if you completely restarted the run. Is this behaviour documented anywhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala closed issue #30642: DAG Run Started Time Appears To Move As the DAG executes, which leads to misleading durations
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala closed issue #30642: DAG Run Started Time Appears To Move As the DAG executes, which leads to misleading durations
URL: https://github.com/apache/airflow/issues/30642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #30642: DAG Run Started Time Appears To Move As the DAG executes, which leads to misleading durations
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #30642:
URL: https://github.com/apache/airflow/issues/30642#issuecomment-1508494084
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #30642: DAG Run Started Time Appears To Move As the DAG executes, which leads to misleading durations
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #30642:
URL: https://github.com/apache/airflow/issues/30642#issuecomment-1509618345
Let me explain what happens when you clear a task:
If you have two tasks in your dag: `t1 --> t2`
When the first task succeeds but the second fails, the run state will be failed and you will have something like:
```
t1 -> (instant 1, instant 2), state success
t2 -> (instant 3, instant 4), state failed
dag run -> (start < instant 1, end > instant 4), state failed
```
When you clear the task t2, we reset the state and the dates of t2 and the dag run, then when they finishes, will have something like:
```
t1 -> (instant 1, instant 2), state success
t2 -> (None, None), state None
dag run -> ( start > run first end, None), queued
```
Then when it finishes:
```
t1 -> (instant 1, instant 2), state success
t2 -> (t5 > run first end, t6), state success
dag run -> ( run first end < start < t5, end > t6), success
```
The run duration is the duration between the start and the end of the DagRun, if we don't reset the start date, the run duration will not be correct, and that affects a lot of things like the DagRun timeout.
We can discuss the possibility to add a new column to store the first start_date if there is a real use case, but as a new feature and not a bug fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #30642: DAG Run Started Time Appears To Move As the DAG executes, which leads to misleading durations
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #30642:
URL: https://github.com/apache/airflow/issues/30642#issuecomment-1508946003
Can you please show some screenshots of what you observe - Ideally showing the tasks in progress ? I just want to make absolutely sure that the start date /end date and "running DAGs" are well understood here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jonathanrainer commented on issue #30642: DAG Run Started Time Appears To Move As the DAG executes, which leads to misleading durations
Posted by "jonathanrainer (via GitHub)" <gi...@apache.org>.
jonathanrainer commented on issue #30642:
URL: https://github.com/apache/airflow/issues/30642#issuecomment-1509071042
<img width="761" alt="Screenshot 2023-04-14 at 19 34 15" src="https://user-images.githubusercontent.com/1529660/232128367-3e188e7e-95cf-4d33-83c8-ff960938d7d3.png">
So here's an example of the kind of thing we see, this DAG is set up to run every three hours, and as we can see from the data interval it should start running at some point after 2pm BST. Now if we look into the actual tasks we can see in the screenshot below that the first task, which has one downstream, which further has everything else in the DAG as a downstream starts at 14:00:14 BST and ran for around 17 minutes which is in-line with what we'd expect.
However if we look again at the first screenshot it reports that the DAG Run started at 15:57:15 BST, which is clearly wrong. Now curiously, the times reported by the DAG Run are the same as the start and end times of the last task in the DAG to run chronologically. Could it possibly be that the start and end time are both getting updated when it should only be the end time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org