You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "codecae (via GitHub)" <gi...@apache.org> on 2023/10/31 17:32:37 UTC
[I] New Graph UI seems to struggle with task-to-task dependencies in nested task groups [airflow]
codecae opened a new issue, #35301:
URL: https://github.com/apache/airflow/issues/35301
### Apache Airflow version
2.7.2
### What happened
When rendering task-to-task dependencies between tasks that are buried into nested task groups, the UI tends to draw the relationships and node spacing in unusual ways.
<img width="723" alt="image" src="https://github.com/apache/airflow/assets/8164265/ad86c0cc-7bbb-496c-a80a-a1dadb6a8c4e">
### What you think should happen instead
The UI should render the graph consistently, regardless of what level you have drilled into the task groups.
### How to reproduce
Run this test code to reproduce the issue:
```
from airflow.decorators import dag, task, task_group
import pendulum
from typing import List, Dict
from airflow.models.baseoperator import BaseOperator
_all_tasks: Dict[str, List[BaseOperator]]={}
@dag(
schedule=None,
start_date=pendulum.datetime(2023,1,1),
)
def test_dag():
def _generate_task_group_callable(group_id, tasks):
@task_group(group_id=group_id)
def _task_group():
[t() for t in tasks]
return _task_group
def _generate_task_callable(task_id, task_set):
def _generate():
@task(task_id=task_id)
def _task():
pass
if _all_tasks.get(task_set, None) == None:
_all_tasks[task_set]=list()
_all_tasks[task_set].append(_task())
return _generate
_set_callables=list()
for _set in range(0,2):
_set_id=f"set_{_set}"
_group_callables_a=list()
for _nga in range(0,3):
_nga_id=f"group_{_nga}"
_group_callables_b=list()
for _ngb in range(0,5):
_ngb_id=f"group_{_ngb}"
_task_callables=list()
for _task in range(0,10):
_task_id=f"task_{_task}"
_callable=_generate_task_callable(_task_id, _set_id)
_task_callables.append(_callable)
_group_callable_inner=_generate_task_group_callable(_ngb_id,_task_callables)
_group_callables_b.append(_group_callable_inner)
_group_callable_outer=_generate_task_group_callable(_nga_id, _group_callables_b)
_group_callables_a.append(_group_callable_outer)
_set_callable=_generate_task_group_callable(_set_id, _group_callables_a)
_set_callables.append(_set_callable)
[_s() for _s in _set_callables]
for i in range(0,len(_all_tasks['set_0'])):
_all_tasks['set_0'][i] >> _all_tasks['set_1'][i]
test_dag()
```
### Operating System
Debian GNU/Linux 11 (bullseye) -- apache/airflow:slim-2.7.2-python3.10 container
### Versions of Apache Airflow Providers
apache-airflow-providers-common-sql==1.7.2
apache-airflow-providers-ftp==3.5.2
apache-airflow-providers-hashicorp==3.5.0
apache-airflow-providers-http==4.5.2
apache-airflow-providers-imap==3.3.2
apache-airflow-providers-sqlite==3.4.3
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
if the grid is is filtered, it has been observed that the graph renders properly in some scenarios. Doing task-group-to-task-group relationships sometimes renders properly, as well.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] New Graph UI seems to struggle with task-to-task dependencies in nested task groups [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #35301:
URL: https://github.com/apache/airflow/issues/35301#issuecomment-2022085824
cc @bbovenzi is these a quick solution to this bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] New Graph UI seems to struggle with task-to-task dependencies in nested task groups [airflow]
Posted by "codecae (via GitHub)" <gi...@apache.org>.
codecae commented on issue #35301:
URL: https://github.com/apache/airflow/issues/35301#issuecomment-1975565209
Seems that this may have become buried... Anything I can do to increase awareness? The graph view is generally unusable on larger, more complex dags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] New Graph UI seems to struggle with task-to-task dependencies in nested task groups [airflow]
Posted by "codecae (via GitHub)" <gi...@apache.org>.
codecae commented on issue #35301:
URL: https://github.com/apache/airflow/issues/35301#issuecomment-2022646461
Due to not being versed enough react/ts development and debugging, I can't say for sure, but it seemed likely after a cursory review of the logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] New Graph UI seems to struggle with task-to-task dependencies in nested task groups [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #35301:
URL: https://github.com/apache/airflow/issues/35301#issuecomment-1787672283
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] New Graph UI seems to struggle with task-to-task dependencies in nested task groups [airflow]
Posted by "codecae (via GitHub)" <gi...@apache.org>.
codecae commented on issue #35301:
URL: https://github.com/apache/airflow/issues/35301#issuecomment-2022638641
While attempting to root out the big, the only thing I could find that might have any influence would be when the bend points are set explicitly.
https://github.com/apache/airflow/blob/bf9bb72bb631c098bfee34a372aefb332c3ffd64/airflow/www/static/js/dag/details/graph/utils.ts#L209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org