You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dheerajturaga (via GitHub)" <gi...@apache.org> on 2026/09/26 04:42:20 UTC
[PR] Deprecate cyclic TaskGroup dependencies [airflow]
dheerajturaga opened a new pull request, #73746:
URL: https://github.com/apache/airflow/pull/73746
Dags whose TaskGroups depend on each other in a cycle, when each group is treated as a single unit, parse and run today even though planned TaskGroup features need an unambiguous order between groups. Following the dev list discussion, this deprecates them: they keep working, users are warned now, and Airflow 3.5 is planned to reject them at parse time. This replaces the immediate parse-time rejection proposed in #73087.
A task with no upstream inside its own TaskGroup counts as a root of that group, even when a task outside the group is its upstream. So these cycles show up in Dags that look ordinary, for example `a >> bridge >> b` with `a` and `b` in the same group and `bridge` outside it.
Parsing a Dag with such a cycle now:
- issues a `TaskGroupCycleDeprecationWarning` from `DAG.check_cycle()`, so CI can catch it, for example with `pytest -W error::airflow.sdk.exceptions.TaskGroupCycleDeprecationWarning`;
- records a new `task group cycle` Dag warning, shown in the UI's Dag warnings and returned by `GET /api/v2/dagWarnings`.
The message names only the tasks and TaskGroups on the cycle:
> Dag 'etl': group1 and group2 depend on each other in a cycle. Cyclic TaskGroup dependencies are deprecated and will fail Dag parsing in Airflow 3.5. See "Cyclic TaskGroup dependencies" in the docs.
A new "Cyclic TaskGroup dependencies" section in the Dags docs covers the Dag shapes that trigger this, how to fix them, and how to catch them in CI.
Python ignores `DeprecationWarning` by default outside `__main__`, so the Dag processor enables this category while it records the warning for the UI, and still re-issues it to the caller's own warning filters.
The Grid/Graph HTTP 500 for these Dags is fixed separately in #73724. The parse-time rejection in 3.5 is tracked in #73678.
related: #73678
Discussion: https://lists.apache.org/thread/sossl7b2w2ftyk4028qrhps2tcdxj2px
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5.5)
Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga merged PR #73746:
URL: https://github.com/apache/airflow/pull/73746
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on PR #73746:
URL: https://github.com/apache/airflow/pull/73746#issuecomment-5994703657
@bbovenzi would like your thoughts on the UI, would like to get this in 3.4 as agreed in Lazy consensus https://lists.apache.org/thread/5fpwlxg4y6jhyrw7n2m3jz3o7t6xvhlk
cc: @ashb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4234686138
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -46,6 +46,7 @@
from airflow.executors.executor_loader import ExecutorLoader
from airflow.listeners.listener import get_listener_manager
from airflow.models.pool import Pool
+from airflow.sdk.exceptions import TaskGroupCycleDeprecationWarning # noqa: SDK001
Review Comment:
oops, fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4210751260
##########
task-sdk/src/airflow/sdk/definitions/taskgroup.py:
##########
@@ -702,6 +710,80 @@ def _sort_via_pass_numbering(
sorted_indices = sorted(range(n), key=lambda i: (pass_of[i], i))
return [nodes[i] for i in sorted_indices]
+ def _find_dependency_cycles(self, *, group_dict: dict[str, TaskGroup]) -> list[list[str]]:
+ """
+ Find children that depend on each other in a cycle when each child TaskGroup is one unit.
+
+ An edge into any task of a child group counts as an edge into the group, so a path that
+ leaves a group and comes back into it is a cycle even though the task-level graph is
+ acyclic. This is stricter than the ordering ``topological_sort`` needs, which only looks
+ at edges into a group's roots.
+
+ :return: one list of child node ids per cycle, each in insertion order
+ """
+ nodes = list(self.children.values())
+ id_to_idx = {nid: i for i, nid in enumerate(self.children)}
+ projected = [
+ self._project_upstream_ids(i, self._get_unit_upstream_ids(child), id_to_idx, group_dict)
+ for i, child in enumerate(nodes)
+ ]
+ members: dict[int, list[str]] = {}
+ for i, component in enumerate(self._find_projection_components(projected)):
+ members.setdefault(component, []).append(nodes[i].node_id)
+ return [node_ids for node_ids in members.values() if len(node_ids) > 1]
+
+ @staticmethod
+ def _get_unit_upstream_ids(child: DAGNode) -> Collection[str]:
+ if not isinstance(child, TaskGroup):
+ return child._topological_upstream_ids
+ upstream_ids = set(child._topological_upstream_ids)
+ upstream_ids.update(edge_id for task in child for edge_id in task.upstream_task_ids)
Review Comment:
Good catch, reproduced: `tg.expand(param=[1, 2, 3])` with a child on `trigger_rule="always"` raised the `ValueError` from `MappedTaskGroup.__iter__`. Fixed in dc8bb11e32: the group's unit edges now come from its `upstream_task_ids`, its non-None `upstream_group_ids` and the `upstream_task_ids` of each task in `iter_tasks()`, as you suggested. Added `test_acyclic_task_groups_do_not_warn` in the SDK tests, which turns the warning into an error and covers that shape, the bridge moved into the group, and a setup and teardown pair with the work inside the group.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @dheerajturaga before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4234612486
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -46,6 +46,7 @@
from airflow.executors.executor_loader import ExecutorLoader
from airflow.listeners.listener import get_listener_manager
from airflow.models.pool import Pool
+from airflow.sdk.exceptions import TaskGroupCycleDeprecationWarning # noqa: SDK001
Review Comment:
Why noqa?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4217947774
##########
task-sdk/src/airflow/sdk/definitions/dag.py:
##########
@@ -1150,6 +1154,30 @@ def check_cycle(self) -> None:
f"Cycle detected in Dag: {self.dag_id}. Faulty task: {faulty_task_id}"
)
+ self._warn_task_group_cycles()
+
+ def _warn_task_group_cycles(self) -> None:
+ task_group_dict = self.task_group.get_task_group_dict()
+ # With only the root group, the projection is the task graph, which is acyclic here.
+ if len(task_group_dict) == 1:
+ return
+ cycles = [
+ f"{', '.join(cycle[:-1])} and {cycle[-1]}"
+ for task_group in task_group_dict.values()
+ for cycle in task_group._find_dependency_cycles(group_dict=task_group_dict)
+ ]
+ if not cycles:
+ return
+ # Airflow 3.5 raises AirflowDagCycleException here instead; tracked at
+ # https://github.com/apache/airflow/issues/73678
+ warnings.warn(
+ f"Dag '{self.dag_id}': {'; '.join(cycles)} depend on each other in a cycle. Cyclic TaskGroup "
Review Comment:
This lists every member of every cycle, and it ends up in `DagWarning.message`, a plain `Text` column (64 KB on MySQL; `DagCode.source_code` uses `Text().with_variant(MEDIUMTEXT(), "mysql")` for this reason). The setup/teardown shape from the new docs with about 2,000 tasks of ~30-character ids between `create` and `delete` gives a 68 KB message, and under MySQL strict mode that insert fails at the `session.flush()` closing `update_dag_parsing_results_in_db` (outside the warnings `try`, same transaction as the serialized Dag writes), so the file's parse results wouldn't save for a Dag this PR says keeps working. Could each cycle be capped at the first N members plus "and K more", with a test on a large cycle?
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -491,7 +496,23 @@ def bag_dag(self, dag: DAG):
:raises: AirflowDagCycleException if a cycle is detected.
:raises: AirflowDagDuplicatedIdException if this dag already exists in the bag.
"""
- dag.check_cycle()
+ from airflow.sdk.exceptions import TaskGroupCycleDeprecationWarning # noqa: SDK001
Review Comment:
Could this move to the module-level imports? Importing `airflow.sdk.exceptions` doesn't load this module, so there's no cycle to avoid, and `check_sdk_imports_in_core.py` honours `# noqa: SDK001` per line wherever the import sits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on PR #73746:
URL: https://github.com/apache/airflow/pull/73746#issuecomment-6023506616
Makes sense to me UI-wise. I am working on improving our warning banner UX but having another category doesn't get in the way of that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4210751260
##########
task-sdk/src/airflow/sdk/definitions/taskgroup.py:
##########
@@ -702,6 +710,80 @@ def _sort_via_pass_numbering(
sorted_indices = sorted(range(n), key=lambda i: (pass_of[i], i))
return [nodes[i] for i in sorted_indices]
+ def _find_dependency_cycles(self, *, group_dict: dict[str, TaskGroup]) -> list[list[str]]:
+ """
+ Find children that depend on each other in a cycle when each child TaskGroup is one unit.
+
+ An edge into any task of a child group counts as an edge into the group, so a path that
+ leaves a group and comes back into it is a cycle even though the task-level graph is
+ acyclic. This is stricter than the ordering ``topological_sort`` needs, which only looks
+ at edges into a group's roots.
+
+ :return: one list of child node ids per cycle, each in insertion order
+ """
+ nodes = list(self.children.values())
+ id_to_idx = {nid: i for i, nid in enumerate(self.children)}
+ projected = [
+ self._project_upstream_ids(i, self._get_unit_upstream_ids(child), id_to_idx, group_dict)
+ for i, child in enumerate(nodes)
+ ]
+ members: dict[int, list[str]] = {}
+ for i, component in enumerate(self._find_projection_components(projected)):
+ members.setdefault(component, []).append(nodes[i].node_id)
+ return [node_ids for node_ids in members.values() if len(node_ids) > 1]
+
+ @staticmethod
+ def _get_unit_upstream_ids(child: DAGNode) -> Collection[str]:
+ if not isinstance(child, TaskGroup):
+ return child._topological_upstream_ids
+ upstream_ids = set(child._topological_upstream_ids)
+ upstream_ids.update(edge_id for task in child for edge_id in task.upstream_task_ids)
Review Comment:
Good catch, reproduced: `tg.expand(param=[1, 2, 3])` with a child on `trigger_rule="always"` raised the `ValueError` from `MappedTaskGroup.__iter__`. Fixed: the group's unit edges now come from its `upstream_task_ids`, its non-None `upstream_group_ids` and the `upstream_task_ids` of each task in `iter_tasks()`, as you suggested. Added `test_acyclic_task_groups_do_not_warn` in the SDK tests, which turns the warning into an error and covers that shape, the bridge moved into the group, and a setup and teardown pair with the work inside the group.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @dheerajturaga before posting
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -491,7 +496,23 @@ def bag_dag(self, dag: DAG):
:raises: AirflowDagCycleException if a cycle is detected.
:raises: AirflowDagDuplicatedIdException if this dag already exists in the bag.
"""
- dag.check_cycle()
+ from airflow.sdk.exceptions import TaskGroupCycleDeprecationWarning # noqa: SDK001
+
+ self.task_group_cycle_warnings.pop(dag.dag_id, None)
+ with warnings.catch_warnings(record=True) as captured_warnings:
+ # DeprecationWarning is ignored by default outside __main__, which would hide it here too.
+ warnings.simplefilter("always", TaskGroupCycleDeprecationWarning)
+ dag.check_cycle()
+ for captured in captured_warnings:
+ if issubclass(captured.category, TaskGroupCycleDeprecationWarning):
+ self.task_group_cycle_warnings[dag.dag_id] = str(captured.message)
Review Comment:
Reproduced both orders with two files sharing a Dag id. Fixed: `bag_dag` now keeps the message in a local and sets or pops `task_group_cycle_warnings` only after `_add_to_bag` succeeds, so a rejected duplicate (or a Dag a cluster policy skips) never touches it. `test_dag_warnings_task_group_cycle_ignores_rejected_duplicate` covers both orders.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @dheerajturaga before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #73746:
URL: https://github.com/apache/airflow/pull/73746#issuecomment-6109297600
### Backport failed to create: airflow-ctl/v0-1-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/38141474050'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>airflow-ctl/v0-1-test</td>
<td><a href="https://github.com/apache/airflow/commit/4ca1f6693bb380ae1f148a6187001adbcadbfa22"><img src='https://img.shields.io/badge/Commit-4ca1f66-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 4ca1f66 airflow-ctl/v0-1-test
```
This should apply the commit to the airflow-ctl/v0-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4210897199
##########
airflow-core/docs/core-concepts/dags.rst:
##########
@@ -624,6 +624,59 @@ If you want to see a more advanced use of TaskGroup, you can look at the ``examp
When using the ``@task_group`` decorator, the decorated-function's docstring will be used as the TaskGroups tooltip in the UI except when a ``tooltip`` value is explicitly supplied.
+.. _concepts:taskgroup-cycles:
+
+Cyclic TaskGroup dependencies
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. deprecated:: 3.4.0
+ Dags with cyclic TaskGroup dependencies are planned to fail Dag parsing from Airflow 3.5.
+
+When each TaskGroup is treated as a single unit, a TaskGroup and its siblings must not depend on each other
+in a cycle. A dependency into or out of any task in a group counts as a dependency of the whole group. This
+can make a group both upstream and downstream of a sibling, although no task-level dependency forms a cycle:
+
+.. code-block:: python
+
+ with TaskGroup("group1"):
+ a1 = EmptyOperator(task_id="a1")
+ a2 = EmptyOperator(task_id="a2")
+
+ with TaskGroup("group2"):
+ b1 = EmptyOperator(task_id="b1")
+ b2 = EmptyOperator(task_id="b2")
+
+ a1 >> b1 # group2 depends on group1
+ b2 >> a2 # group1 depends on group2
+
+A path that leaves a TaskGroup and comes back into it also forms a cycle, even when the tasks inside the
Review Comment:
Agreed, it is intended. dags.rst now has it as its own example, says a TaskGroup that holds a setup and its teardown must also hold the tasks between them, and gives the fix (move `work` into `cluster`). The SDK tests pin both sides: `setup-teardown-around-outside-task` warns with "cluster and work", and `setup-teardown-in-group` does not warn.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @dheerajturaga before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4210943106
##########
task-sdk/src/airflow/sdk/definitions/dag.py:
##########
@@ -1150,6 +1154,30 @@ def check_cycle(self) -> None:
f"Cycle detected in Dag: {self.dag_id}. Faulty task: {faulty_task_id}"
)
+ self._warn_task_group_cycles()
+
+ def _warn_task_group_cycles(self) -> None:
Review Comment:
Lang SDKs are out of scope for this PR. It keeps the deprecation to the Python SDK for 3.4, and dags.rst now says Go, Java and TypeScript SDK Dags are not checked yet. I've noted on #73678 that Lang SDK Dags need the check on `SerializedTaskGroup` in core before the 3.5 rejection lands, so the rejection doesn't end up Python-only.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @dheerajturaga before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4210405066
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -491,7 +496,23 @@ def bag_dag(self, dag: DAG):
:raises: AirflowDagCycleException if a cycle is detected.
:raises: AirflowDagDuplicatedIdException if this dag already exists in the bag.
"""
- dag.check_cycle()
+ from airflow.sdk.exceptions import TaskGroupCycleDeprecationWarning # noqa: SDK001
+
+ self.task_group_cycle_warnings.pop(dag.dag_id, None)
+ with warnings.catch_warnings(record=True) as captured_warnings:
+ # DeprecationWarning is ignored by default outside __main__, which would hide it here too.
+ warnings.simplefilter("always", TaskGroupCycleDeprecationWarning)
+ dag.check_cycle()
+ for captured in captured_warnings:
+ if issubclass(captured.category, TaskGroupCycleDeprecationWarning):
+ self.task_group_cycle_warnings[dag.dag_id] = str(captured.message)
Review Comment:
This records the warning before the cluster policies and `_add_to_bag` run, and `dag_warnings` only checks that the id is in `self.dags`. In a folder-wide bag (`airflow dags reserialize` passes `dagbag.dag_warnings` to the DB), an acyclic Dag bagged first followed by a cyclic duplicate with the same id leaves the acyclic Dag with a cycle warning, and the reverse order pops the real one. Keeping the message in a local and writing the dict after `_add_to_bag` succeeds would avoid both.
##########
airflow-core/docs/core-concepts/dags.rst:
##########
@@ -624,6 +624,59 @@ If you want to see a more advanced use of TaskGroup, you can look at the ``examp
When using the ``@task_group`` decorator, the decorated-function's docstring will be used as the TaskGroups tooltip in the UI except when a ``tooltip`` value is explicitly supplied.
+.. _concepts:taskgroup-cycles:
+
+Cyclic TaskGroup dependencies
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. deprecated:: 3.4.0
+ Dags with cyclic TaskGroup dependencies are planned to fail Dag parsing from Airflow 3.5.
+
+When each TaskGroup is treated as a single unit, a TaskGroup and its siblings must not depend on each other
+in a cycle. A dependency into or out of any task in a group counts as a dependency of the whole group. This
+can make a group both upstream and downstream of a sibling, although no task-level dependency forms a cycle:
+
+.. code-block:: python
+
+ with TaskGroup("group1"):
+ a1 = EmptyOperator(task_id="a1")
+ a2 = EmptyOperator(task_id="a2")
+
+ with TaskGroup("group2"):
+ b1 = EmptyOperator(task_id="b1")
+ b2 = EmptyOperator(task_id="b2")
+
+ a1 >> b1 # group2 depends on group1
+ b2 >> a2 # group1 depends on group2
+
+A path that leaves a TaskGroup and comes back into it also forms a cycle, even when the tasks inside the
Review Comment:
This rule also flags a common setup/teardown shape: `create` and `delete` in `TaskGroup("cluster")`, `work` outside it, and `create >> work >> delete.as_teardown(setups=create)` gives cluster -> work -> cluster. It matches the rule from the lazy-consensus thread, but since the planned rejection would make it a parse error, could the docs call out the setup/teardown case explicitly (and a test pin it), so users with that layout know it's intended?
##########
airflow-core/newsfragments/73746.significant.rst:
##########
@@ -0,0 +1,20 @@
+Deprecate cyclic TaskGroup dependencies
+
+A Dag whose TaskGroups depend on each other in a cycle, when each TaskGroup is treated as a single unit,
+now issues a ``TaskGroupCycleDeprecationWarning`` at parse time and shows a Dag warning in the UI that
+names the TaskGroups and tasks involved. These Dags still parse and run, because their task dependencies
+are acyclic, but Airflow 3.5 is planned to reject them at parse time.
+
+A dependency into or out of any task in a TaskGroup counts as a dependency of the whole group, so a path
+that leaves a TaskGroup and comes back into it is a cycle. For example, ``a >> bridge >> b``, with ``a``
+and ``b`` in the same TaskGroup and ``bridge`` outside it, makes the group depend on ``bridge`` and
+``bridge`` depend on the group, even if ``a >> b`` is also set.
+
+**Migration:**
+
+- Move tasks between TaskGroups, or out of them, so that each group depends on the others in one direction
+ only. In the example above, move ``bridge`` into the group, or move ``b`` out of it.
+- To catch these Dags in CI, run your Dag tests with
+ ``-W error::airflow.sdk.exceptions.TaskGroupCycleDeprecationWarning``.
Review Comment:
`check_cycle()` only runs from `bag_dag`, so `-W error` catches these only in tests that load the Dags through a `DagBag` and assert no import errors; a test that imports the Dag file directly still passes. dags.rst states that condition, could the newsfragment carry it too?
##########
task-sdk/src/airflow/sdk/definitions/dag.py:
##########
@@ -1150,6 +1154,30 @@ def check_cycle(self) -> None:
f"Cycle detected in Dag: {self.dag_id}. Faulty task: {faulty_task_id}"
)
+ self._warn_task_group_cycles()
+
+ def _warn_task_group_cycles(self) -> None:
Review Comment:
The check only lives in the Python SDK's `check_cycle()`. Go SDK Dags serialize task groups too (`go-sdk/airflow/serialize.go`), but `lang_sdk_processor.py` only validates the serialized Dag, so they get neither this warning nor, as it stands, the planned parse-time rejection. Is that intended? If not, running the check on `SerializedTaskGroup` in core would cover every SDK; if Lang SDKs are out of scope, worth saying so in the docs and in #73678.
##########
task-sdk/src/airflow/sdk/definitions/taskgroup.py:
##########
@@ -702,6 +710,80 @@ def _sort_via_pass_numbering(
sorted_indices = sorted(range(n), key=lambda i: (pass_of[i], i))
return [nodes[i] for i in sorted_indices]
+ def _find_dependency_cycles(self, *, group_dict: dict[str, TaskGroup]) -> list[list[str]]:
+ """
+ Find children that depend on each other in a cycle when each child TaskGroup is one unit.
+
+ An edge into any task of a child group counts as an edge into the group, so a path that
+ leaves a group and comes back into it is a cycle even though the task-level graph is
+ acyclic. This is stricter than the ordering ``topological_sort`` needs, which only looks
+ at edges into a group's roots.
+
+ :return: one list of child node ids per cycle, each in insertion order
+ """
+ nodes = list(self.children.values())
+ id_to_idx = {nid: i for i, nid in enumerate(self.children)}
+ projected = [
+ self._project_upstream_ids(i, self._get_unit_upstream_ids(child), id_to_idx, group_dict)
+ for i, child in enumerate(nodes)
+ ]
+ members: dict[int, list[str]] = {}
+ for i, component in enumerate(self._find_projection_components(projected)):
+ members.setdefault(component, []).append(nodes[i].node_id)
+ return [node_ids for node_ids in members.values() if len(node_ids) > 1]
+
+ @staticmethod
+ def _get_unit_upstream_ids(child: DAGNode) -> Collection[str]:
+ if not isinstance(child, TaskGroup):
+ return child._topological_upstream_ids
+ upstream_ids = set(child._topological_upstream_ids)
+ upstream_ids.update(edge_id for task in child for edge_id in task.upstream_task_ids)
Review Comment:
This walk goes through `MappedTaskGroup.__iter__` (here, and in `get_roots()` via `_topological_upstream_ids` on the line above), which raises `ValueError("Task-generated mapping within a mapped task group is not allowed with trigger rule 'always'")` for any direct child with `trigger_rule=ALWAYS`, even when the group is expanded over a literal list. So `@task_group def tg(p): t1(p)` with `t1` on `ALWAYS` and `tg.expand(param=[1, 2, 3])` parses on main but becomes an import error with this PR, which breaks the "they keep working" promise for Dags that have no cycle at all.
Building the set from `child.upstream_task_ids`, the non-None `child.upstream_group_ids`, and the `upstream_task_ids` of each task in `child.iter_tasks()` avoids the validating iterator (roots are a subset of all member tasks, so nothing is lost). Could you add that shape to a test that asserts acyclic Dags don't warn? There's no negative test on the SDK side yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4210834603
##########
airflow-core/newsfragments/73746.significant.rst:
##########
@@ -0,0 +1,20 @@
+Deprecate cyclic TaskGroup dependencies
+
+A Dag whose TaskGroups depend on each other in a cycle, when each TaskGroup is treated as a single unit,
+now issues a ``TaskGroupCycleDeprecationWarning`` at parse time and shows a Dag warning in the UI that
+names the TaskGroups and tasks involved. These Dags still parse and run, because their task dependencies
+are acyclic, but Airflow 3.5 is planned to reject them at parse time.
+
+A dependency into or out of any task in a TaskGroup counts as a dependency of the whole group, so a path
+that leaves a TaskGroup and comes back into it is a cycle. For example, ``a >> bridge >> b``, with ``a``
+and ``b`` in the same TaskGroup and ``bridge`` outside it, makes the group depend on ``bridge`` and
+``bridge`` depend on the group, even if ``a >> b`` is also set.
+
+**Migration:**
+
+- Move tasks between TaskGroups, or out of them, so that each group depends on the others in one direction
+ only. In the example above, move ``bridge`` into the group, or move ``b`` out of it.
+- To catch these Dags in CI, run your Dag tests with
+ ``-W error::airflow.sdk.exceptions.TaskGroupCycleDeprecationWarning``.
Review Comment:
Done: the migration bullet now says the check runs when a Dag is added to a `DagBag`, so only tests that load Dags into a `DagBag` and assert no import errors catch it, and a test that only imports a Dag file does not. I tightened the same sentence in dags.rst.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @dheerajturaga before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deprecate cyclic TaskGroup dependencies [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #73746:
URL: https://github.com/apache/airflow/pull/73746#discussion_r4224577380
##########
task-sdk/src/airflow/sdk/definitions/dag.py:
##########
@@ -1150,6 +1154,30 @@ def check_cycle(self) -> None:
f"Cycle detected in Dag: {self.dag_id}. Faulty task: {faulty_task_id}"
)
+ self._warn_task_group_cycles()
+
+ def _warn_task_group_cycles(self) -> None:
+ task_group_dict = self.task_group.get_task_group_dict()
+ # With only the root group, the projection is the task graph, which is acyclic here.
+ if len(task_group_dict) == 1:
+ return
+ cycles = [
+ f"{', '.join(cycle[:-1])} and {cycle[-1]}"
+ for task_group in task_group_dict.values()
+ for cycle in task_group._find_dependency_cycles(group_dict=task_group_dict)
+ ]
+ if not cycles:
+ return
+ # Airflow 3.5 raises AirflowDagCycleException here instead; tracked at
+ # https://github.com/apache/airflow/issues/73678
+ warnings.warn(
+ f"Dag '{self.dag_id}': {'; '.join(cycles)} depend on each other in a cycle. Cyclic TaskGroup "
Review Comment:
Capped. Each cycle now lists its first five members and then "and K more", and the message lists at most ten cycles, then "(K more cycles not listed)". I capped the cycle count too because a Dag with many TaskGroups, each in its own small cycle, runs into the same limit. With both caps, the message length stays bounded whatever the Dag's size. `test_task_group_cycle_warning_caps_listed_ids` covers the setup/teardown shape from the docs with 100 tasks between `create` and `delete`, and Dags with 11 and 12 bridged groups. dags.rst now mentions the caps.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @dheerajturaga before posting
##########
airflow-core/src/airflow/dag_processing/dagbag.py:
##########
@@ -491,7 +496,23 @@ def bag_dag(self, dag: DAG):
:raises: AirflowDagCycleException if a cycle is detected.
:raises: AirflowDagDuplicatedIdException if this dag already exists in the bag.
"""
- dag.check_cycle()
+ from airflow.sdk.exceptions import TaskGroupCycleDeprecationWarning # noqa: SDK001
Review Comment:
Done, it's with the module-level imports now and keeps `# noqa: SDK001` on that line.
---
Drafted-by: Claude Code (Opus 5.5); reviewed by @dheerajturaga before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org