You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "JH0917 (via GitHub)" <gi...@apache.org> on 2026/09/30 03:57:57 UTC

[PR] Fix Grid and Graph 500 errors for TaskGroup-level cycles [airflow]

JH0917 opened a new pull request, #73933:
URL: https://github.com/apache/airflow/pull/73933

   Since 3.3.1 (#69933), Grid and Graph return HTTP 500 when TaskGroups wait on each other as
   units, even though the task graph is acyclic. A path that leaves a group and re-enters it
   (`g.a >> ext >> g.b`) is enough, and `partial_subset` filtering can create the same shape
   (found by @shubhamraj-git in #73678), so the UI sort must tolerate it.
   
   **Changes:**
   
   1. When all remaining children are blocked, pass numbering releases one child on the cycle
      and continues. The sweep path falls back to pass numbering instead of raising.
   2. Dags without a group-level cycle are sorted exactly as before. The task-sdk sort still raises.
   
   **vs #73724:** same bug, different approach. #73724 collapses each SCC into one unit; this
   breaks the cycle at a single child inside the existing loop, with no new helpers, and puts
   fewer dependencies out of order (ring of three groups: 1 vs 2).
   
   <details>
   <summary>Benchmark and randomized check</summary>
   
   Root-level `topological_sort`, best of 3:
   
   | Dag shape | children | this PR | #73724 |
   |---|---|---|---|
   | acyclic chain | 10,000 | 5.6 ms | 5.6 ms |
   | 5,000 `g.a >> bridge >> g.b` cycles | 10,000 | 22.2 ms | 24.2 ms |
   | reverse-declared chain behind one cycle | 10,002 | 13.2 ms | 20.8 ms |
   | 5,000-task chain behind 2,000 cycles | 9,000 | 37.3 ms | 20.8 ms |
   
   3,000 random Dags: groups without a group-level cycle (11,504) sort identically to main;
   in groups with one (2,061), out-of-order dependencies total 3,882 vs 4,618.
   
   </details>
   
   related: #73678
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Opus 5.5)
   
   Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Render Grid and Graph when a path leaves and re-enters a TaskGroup [airflow]

Posted by "fat-catTW (via GitHub)" <gi...@apache.org>.
fat-catTW commented on PR #73933:
URL: https://github.com/apache/airflow/pull/73933#issuecomment-5933345574

   Thanks for working on this. 
   But I’m concerned that the current fallback may produce a misleading order beyond just breaking one edge in the group-level cycle.
   
   In the new test case:
   
   ```python
   extract >> staging >> spark_job >> mart >> publish
   ```
   
   the real task path is:
   
   ```text
   extract -> models.staging -> spark_job -> models.mart -> publish
   ```
   
   But the expected root-level order is:
   
   ```text
   extract, models, publish, spark_job
   ```
   
   That puts `publish` before `spark_job`, even though every real task path to `publish` goes through `spark_job`.
   
   I understand that once the collapsed/group-level graph has a cycle, no perfect topological order exists. But this seems potentially confusing for users because the UI may suggest that downstream work outside the cycle can happen before a task it actually depends on. More generally, since a collapsed TaskGroup can only appear once, a path like `models.staging -> spark_job -> models.mart` cannot be faithfully represented as `models -> spark_job -> models`, so the fallback can make the group look like a single execution unit before or after `spark_job`, even though only part of the group is before it.
   
   Is this tradeoff intentional? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Render Grid and Graph when a path leaves and re-enters a TaskGroup [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed pull request #73933: Render Grid and Graph when a path leaves and re-enters a TaskGroup
URL: https://github.com/apache/airflow/pull/73933


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Render Grid and Graph when a path leaves and re-enters a TaskGroup [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #73933:
URL: https://github.com/apache/airflow/pull/73933#issuecomment-5940121506

   Closing this. #73724 is already open for #73678, from the maintainer who filed the issue and said they would handle it, five days before this one. This PR also encodes an order that contradicts the Dag: the new test builds `extract >> staging >> spark_job >> mart >> publish` and expects `["extract", "models", "publish", "spark_job"]`, which puts `publish` before `spark_job` even though every path to `publish` runs through `spark_job`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org