You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/04 22:11:24 UTC
[GitHub] [airflow] ashb opened a new pull request, #22741: Show tasks in grid view based on topological sort.
ashb opened a new pull request, #22741:
URL: https://github.com/apache/airflow/pull/22741
This takes the existing topological sort that existed on a DAG and moves
it down to TaskGroup.
In order to do this (and not have duplicated sort) the existing sort on
DAG is re-implemented on top of the new method.
This also surfaced a tiny bug in deserialize_task_group where the
SerializedTaskGroup did not have `dag` set -- it didn't cause any
problems until now but was needed to call `upstream_list` on a
SerializedTaskGroup object.
Fixes https://github.com/apache/airflow/issues/22731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22741: Show tasks in grid view based on topological sort.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22741:
URL: https://github.com/apache/airflow/pull/22741#issuecomment-1088661139
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22741: Show tasks in grid view based on topological sort.
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #22741:
URL: https://github.com/apache/airflow/pull/22741#discussion_r842588752
##########
airflow/utils/task_group.py:
##########
@@ -401,6 +401,69 @@ def expand(self, arg: Iterable) -> "MappedTaskGroup":
self.task_group._remove(self)
return MappedTaskGroup(group_id=self._group_id, dag=self.dag, mapped_arg=arg)
+ def topological_sort(self, _include_subdag_tasks: bool = False):
+ """
+ Sorts children in topographical order, such that a task comes after any of its
+ upstream dependencies.
+
+ Heavily inspired by:
+ http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/
Review Comment:
Oh yeah :( I copied the doc over without checking it. I'll see if I can find a suitable replacement URL (and remove it from the doc string)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #22741: Show tasks in grid view based on topological sort.
Posted by GitBox <gi...@apache.org>.
ashb merged PR #22741:
URL: https://github.com/apache/airflow/pull/22741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22741: Show tasks in grid view based on topological sort.
Posted by GitBox <gi...@apache.org>.
ashb commented on code in PR #22741:
URL: https://github.com/apache/airflow/pull/22741#discussion_r842601547
##########
airflow/utils/task_group.py:
##########
@@ -401,6 +401,69 @@ def expand(self, arg: Iterable) -> "MappedTaskGroup":
self.task_group._remove(self)
return MappedTaskGroup(group_id=self._group_id, dag=self.dag, mapped_arg=arg)
+ def topological_sort(self, _include_subdag_tasks: bool = False):
+ """
+ Sorts children in topographical order, such that a task comes after any of its
+ upstream dependencies.
+
+ Heavily inspired by:
+ http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/
Review Comment:
Can't find what was on that URL, but I've worked out that this is an implementation of Kahn's Topo Sort algorithm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Show tasks in grid view based on topological sort. [airflow]
Posted by "prithvi-git (via GitHub)" <gi...@apache.org>.
prithvi-git commented on PR #22741:
URL: https://github.com/apache/airflow/pull/22741#issuecomment-1953646652
We're using airflow version 2.6.3 in GCP composer v2.6.0. We're still facing this issue -
Task dependency is set as per the task name order, but **3-3** and **5-1** appear incorrectly in Grid view (PFA 1). Shows corectly in Graph view (PFA 2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] norm commented on a diff in pull request #22741: Show tasks in grid view based on topological sort.
Posted by GitBox <gi...@apache.org>.
norm commented on code in PR #22741:
URL: https://github.com/apache/airflow/pull/22741#discussion_r842581975
##########
airflow/utils/task_group.py:
##########
@@ -401,6 +401,69 @@ def expand(self, arg: Iterable) -> "MappedTaskGroup":
self.task_group._remove(self)
return MappedTaskGroup(group_id=self._group_id, dag=self.dag, mapped_arg=arg)
+ def topological_sort(self, _include_subdag_tasks: bool = False):
+ """
+ Sorts children in topographical order, such that a task comes after any of its
+ upstream dependencies.
+
+ Heavily inspired by:
+ http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/
Review Comment:
Alas, this URL no longer works and is not in the Wayback Machine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Show tasks in grid view based on topological sort. [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on PR #22741:
URL: https://github.com/apache/airflow/pull/22741#issuecomment-1955160229
@prithvi-git Mind opening a new github issue for this please? I wonder if this is because of the task groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org