You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/14 15:15:50 UTC
[GitHub] [airflow] scauglog opened a new issue #13066: Lazy load Task in TaskGroup
scauglog opened a new issue #13066:
URL: https://github.com/apache/airflow/issues/13066
**Description**
Lazy load task in a task group
**Use case / motivation**
I have a DAG with 4 * 3 000 task that can run in parallel I have grouped them under 4 subdags and it work fine but when I use TaskGroup instead of Subdag the DAG is really slow to display. I would like the UI to only load html element for the "main dag" for graph and tree view.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #13066: Lazy load Task in TaskGroup
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13066:
URL: https://github.com/apache/airflow/issues/13066#issuecomment-744765416
Yeah, that would be a nice improvement!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yuqian90 edited a comment on issue #13066: Lazy load Task in TaskGroup
Posted by GitBox <gi...@apache.org>.
yuqian90 edited a comment on issue #13066:
URL: https://github.com/apache/airflow/issues/13066#issuecomment-745804077
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13066: Lazy load Task in TaskGroup
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13066:
URL: https://github.com/apache/airflow/issues/13066#issuecomment-745549875
cc @yuqian90 @ryanahamilton Any interest in doing it :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #13066: Lazy load Task in TaskGroup
Posted by GitBox <gi...@apache.org>.
bbovenzi commented on issue #13066:
URL: https://github.com/apache/airflow/issues/13066#issuecomment-923943642
We'll try to tackle this when updating the tree view for #18009
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yuqian90 commented on issue #13066: Lazy load Task in TaskGroup
Posted by GitBox <gi...@apache.org>.
yuqian90 commented on issue #13066:
URL: https://github.com/apache/airflow/issues/13066#issuecomment-745804077
I'd love to. But i'm busy with other things at the moment. Won't be able to get this at least till next month. If someone likes to take it please go for it.
For the time being, @scauglog , did you consider simply adding a few layers of TaskGroup to nest the 300 tasks in each TaskGroup into a more levels? E.g. first level has 4 TaskGroups. Inside each TaskGroup, there are 10 sub-TaskGroups. In each sub-TaskGroup, there are 300 tasks. I guess it might help because the bottleneck is usually when Graph View tries to render all the nodes. As long as it doesn't have to render all 3000 tasks at once, it's usually not too bad. Let me know if that improves the speed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org