You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/13 16:32:57 UTC
[GitHub] [airflow] yxiao1996 opened a new issue, #28341: Showing Task State in Gantt Chart?
yxiao1996 opened a new issue, #28341:
URL: https://github.com/apache/airflow/issues/28341
### Description
Hi team,
Recently I've been working on using Airflow to orchestrate some of my teams ETL jobs. Lately I run into some performance issue and noticed that the Gantt chart feature is a pretty handy tool for understanding long running DAG runs. I observed large gaps between tasks in the Gantt chart, showing there's a resource bottleneck. However, it seems like the Gantt chart only tracks "running" state of the tasks, so I was not able to fully pin-point whether the bottleneck is the worker(tasks being "queued" for a long time) or the scheduler(tasks being in "no_status" for a long time). So I'm thinking about if airflow could track different task state in the Gantt chart, it would be easier for users to troubleshoot similar issues.
Please educate me if my understanding is wrong.
Thanks,
Yu
### Use case/motivation
By having task state tracking in Gantt chart, it would be easier for airflow users to troubleshoot performance issue in DAG's.
### Related issues
n/a
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1349004821
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi closed issue #28341: Showing Task State in Gantt Chart?
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi closed issue #28341: Showing Task State in Gantt Chart?
URL: https://github.com/apache/airflow/issues/28341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1353249776
Yes, that sounds about right. And the class will just be `queued` instead of the task state. You can see how the current boxes are created [here](https://github.com/apache/airflow/blob/753b9eced5938a4216ddaf0c4afcf89bd7a39a4e/airflow/www/static/js/gantt.js#L243). Reach out if you have any other questions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
pierrejeambrun commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1350590051
Hello @yxiao1996,
Good idea!
I see you are willing to contribute, assigning you :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1369209008
Looks great! I'd say open a PR so we can can a closer look. If it's not fully ready, open it up as a draft.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1369224598
Looks nice!
BTW would be intresting as a followup to see what we can do about https://github.com/apache/airflow/issues/17487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #28341: Showing Task State in Gantt Chart?
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1416398974
Closed by https://github.com/apache/airflow/pull/28686
But a future improvement could be to use Task Reschedules in Gantt as explained in https://github.com/apache/airflow/issues/17487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yxiao1996 commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
yxiao1996 commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1351886749
thanks for the comments. wip....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yxiao1996 commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
yxiao1996 commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1352241000
Hi team, I looked into the code a little bit, want to check with you about the general approach before making the change.
Seems like I should be able to retrieved the `queued_dttm` field from the `ti` objects [here](https://github.com/apache/airflow/blob/753b9eced5938a4216ddaf0c4afcf89bd7a39a4e/airflow/www/views.py#L3441)? If that is the case, looks like I can add queued_dttm field into `task_dict`, which would be passed to the front-end. Then I can make drop new boxes in gantt.js around [here](https://github.com/apache/airflow/blob/753b9eced5938a4216ddaf0c4afcf89bd7a39a4e/airflow/www/static/js/gantt.js#L186) with box width defined by queued_dttm and task start_date.
Am I making sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yxiao1996 commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
yxiao1996 commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1368054390

Here's a snapshot of this feature in my dev environment. Basically I'm drawing these gray boxes for tasks instances with non-null queued_dttm, also add a tooltip on the queued state box to provide related info. Curious do you guys have any comments on how this should look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yxiao1996 commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
yxiao1996 commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1367676299
Hi,
I think I'm getting close to put up a PR. I noticed that the queued_dttm is already passed to the UI through [this line](https://github.com/apache/airflow/blob/753b9eced5938a4216ddaf0c4afcf89bd7a39a4e/airflow/www/views.py#L3448) so I'm only changing the javascript. I'm curious are there any tests I need to created for this new feature? I don't see on the surface there's any tests for these javascripts. Also, is there a template I need to follow for PR?
Thanks,
Yu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #28341: Showing Task State in Gantt Chart?
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #28341:
URL: https://github.com/apache/airflow/issues/28341#issuecomment-1349058502
We do have `queued_dttm` for task instances, so that shouldn't be too hard to add. Otherwise, I think we'll need a new table to record task events like state changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org