You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "tirkarthi (via GitHub)" <gi...@apache.org> on 2024/03/16 17:40:23 UTC
[PR] Add average duration markline in task and dagrun duration charts. [airflow]
tirkarthi opened a new pull request, #38214:
URL: https://github.com/apache/airflow/pull/38214
Add average duration for run and queued time in task duration and dag run duration. This can help in visualizing potential delays in runs like below dag for one month where there is more sleep time in the last 5 days compared to previous days.
Related: #22132
Ref comment in PR review : https://github.com/apache/airflow/pull/35863#issuecomment-1894981925


```python
from datetime import datetime, timedelta
from airflow import DAG
from airflow.decorators import task
from airflow.operators.empty import EmptyOperator
from datetime import timedelta
with DAG(
dag_id="task_duration_average",
start_date=datetime(2024, 1, 1),
end_date=datetime(2024, 1, 31),
catchup=True,
schedule_interval="@daily",
) as dag:
@task
def sleeper(ds=None):
import random, time, pendulum
end_term = pendulum.datetime(2024, 1, 25)
execution_date = pendulum.parse(ds)
if execution_date < end_term:
duration = random.randint(1, 5)
else:
duration = random.randint(10, 20)
time.sleep(duration)
sleeper()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add average duration markline in task and dagrun duration charts. [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #38214:
URL: https://github.com/apache/airflow/pull/38214#issuecomment-2002060137
That is very nice!
Is there a way the user can set if they want to see avg, p50, p75 ?
avg may be less useful in case there is 1 very long run and alot of short runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add average duration markline in task and dagrun duration charts. [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on PR #38214:
URL: https://github.com/apache/airflow/pull/38214#issuecomment-2002287981
> That is very nice!
> Is there a way the user can set if they want to see avg, p50, p75 ?
> avg may be less useful in case there is 1 very long run and alot of short runs.
Nice. Percentile calculations are going to be very useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add average duration markline in task and dagrun duration charts. [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on PR #38214:
URL: https://github.com/apache/airflow/pull/38214#issuecomment-2002067064
The value is calculated by echarts using type option. It only supports average, min, max and median values. But since runduration and queued duration is calculated in javascript the p50, p70 values can also be calculated to be supplied to markline and they can have different colors. As per my understanding p50 and median are equivalent which could be more useful than average.
https://echarts.apache.org/en/option.html#series-bar.markLine.data.0.type
https://math.stackexchange.com/questions/2048470/is-50th-percentile-equal-to-median
Here median is red and average is blue for the same task as above.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add average duration markline in task and dagrun duration charts. [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #38214:
URL: https://github.com/apache/airflow/pull/38214#issuecomment-2002074774
> Here median is red and average is blue for the same task as above.
That is cool and useful!
We will need a legend for the distinction.
I am not a UX expert but lets see if we can choose color pallet that will be color blind friendly. We got several requests about this in the past.
> As per my understanding p50 and median are equivalent
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add average duration markline in task and dagrun duration charts. [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on PR #38214:
URL: https://github.com/apache/airflow/pull/38214#issuecomment-2004607556
Nice work!
If we do median and average, let's change the tick style so we rely on more than just color. Also, we should make sure the colors are separate from any state colors like red and green.
I also agree that we need a legend.
Later on, we can move more of this logic to the backend and we can do more complex metrics. I would still like to do a historic gantt chart with a box-and-whisker plot of task durations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add average duration markline in task and dagrun duration charts. [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on PR #38214:
URL: https://github.com/apache/airflow/pull/38214#issuecomment-2004610358
But if we do just average, this looks good. Does this also work for Run Duration when "Show Landing Times" is checked?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add average duration markline in task and dagrun duration charts. [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on PR #38214:
URL: https://github.com/apache/airflow/pull/38214#issuecomment-2012403028
> But if we do just average, this looks good. Does this also work for Run Duration when "Show Landing Times" is checked?
After using this in several charts of different variations I agree with @eladkal that median looks more useful since one of the outlier task that takes more time to run skews the average to be higher when other tasks executed in few minutes and median looks more useful in the chart compared to average value. I am updating the PR to show median value only to skip average.
Please feel free to suggest color that is accessible since I just used the default one provided by echarts.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add average duration markline in task and dagrun duration charts. [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi merged PR #38214:
URL: https://github.com/apache/airflow/pull/38214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org