You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ColtenOuO (via GitHub)" <gi...@apache.org> on 2026/08/29 21:37:47 UTC

[PR] Fix asset dependency graph dropping outlets for tasks that both consume and produce [airflow]

ColtenOuO opened a new pull request, #72272:
URL: https://github.com/apache/airflow/pull/72272

   ### Summary
   
   `get_data_dependencies()` (the Assets dependency-graph API) shares a single `processed_tasks` set between its producer-side and consumer-side BFS branches. A task that both consumes one asset and produces another gets marked "processed" by whichever branch the BFS reaches it from first, which silently skips the batched lookup the *other* branch would have run — dropping any asset only reachable through that skipped direction, with no error or warning that the graph is incomplete.
   
   ### Example
   
   Task `transform` reads asset `X` and writes assets `Y` and `Z`:
   
   ```mermaid
   flowchart LR
       X([asset X]) --> transform[[transform]]
       transform --> Y([asset Y])
       transform --> Z([asset Z])
   ```
   
   Querying the dependency graph rooted at `Y`:
   
   ```mermaid
   sequenceDiagram
       participant Y as asset Y (query root)
       participant BFS
       participant X as asset X
   
       Note over BFS: Round 1 — visiting Y
       Y->>BFS: transform produces Y
       BFS->>BFS: mark transform processed<br/>query its INLETS → finds X
       Note over BFS: Round 2 — visiting X
       X->>BFS: transform consumes X
       BFS->>BFS: transform already processed<br/>→ OUTLET query skipped
       Note over BFS: Z is never queried — silently missing from the graph
   ```
   
   `transform` is discovered twice — once as a producer of `Y` (which queries its *inlets*, finding `X`), and once as a consumer of `X` (which should query its *outlets* to keep tracing downstream). Because both branches gate their lookup on the same `processed_tasks` set, the second visit sees `transform` already marked and skips the outlet query entirely — so `Z`, an outlet unrelated to `X` or `Y`, never appears in the graph.
   
   ### Fix
   
   Split the shared `processed_tasks` set into two independently-tracked sets, `processed_inlet_tasks` and `processed_outlet_tasks`, so a task reached from one direction still has the other direction's lookup run exactly once. The final team-annotation step, which previously derived `all_dag_ids` from `processed_tasks`, now derives it directly from the task nodes already collected in the graph — the more direct source of truth, and unaffected by the split.
   
   ### Testing
   
   Added `test_data_dependencies_finds_all_outlets_for_task_that_also_consumes_an_asset`, reproducing the exact scenario above (task reads X, writes Y and Z; query rooted at Y). Verified the test fails on the pre-fix code (`Z` missing from the response) and passes after the fix.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Sonnet 5) for writing test
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix asset dependency graph dropping outlets for tasks that both consume and produce [airflow]

Posted by "fat-catTW (via GitHub)" <gi...@apache.org>.
fat-catTW commented on PR #72272:
URL: https://github.com/apache/airflow/pull/72272#issuecomment-5559041771

   Look nice!
   Thanks for contribution :D


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix asset dependency graph dropping outlets for tasks that both consume and produce [airflow]

Posted by "ColtenOuO (via GitHub)" <gi...@apache.org>.
ColtenOuO commented on PR #72272:
URL: https://github.com/apache/airflow/pull/72272#issuecomment-5807041421

   Resolved the conflict :D


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix asset dependency graph dropping outlets for tasks that both consume and produce [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #72272: Fix asset dependency graph dropping outlets for tasks that both consume and produce
URL: https://github.com/apache/airflow/pull/72272


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix asset dependency graph dropping outlets for tasks that both consume and produce [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #72272:
URL: https://github.com/apache/airflow/pull/72272#issuecomment-5839222624

   Hello @ColtenOuO - thank you for your contributions to Apache Airflow!
   
   The Airflow community has introduced a limit of **5 open pull requests at a time** for contributors without write access to the repository. You currently have 19 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
   
   * #70472 - [AIP-94] Add airflowctl Tasks List Command
   * #71853 - Let HookToolset retry a failed tool call instead of failing the task
   * #71962 - Fix scheduler crash when the API server response is a redirect
   * #72003 - Fix manually triggered runs blocking a scheduled run at the same logical_date
   * #72007 - Fix mismatched XCom values when a dynamically mapped task has a gap
   * #72070 - Refactor schema version checks to share target-branch and worktree logic
   * #72073 - Document custom OTel metrics endpoint path support
   * #72107 - Batch XCom lookups when resolving a mapped task's expand() arguments
   * #72270 - Remove redundant full-Dag TaskInstance query from the execution API task-states endpoint
   * #72272 - Fix asset dependency graph dropping outlets for tasks that both consume and produce
   * #72585 - Make supervisor split-line stdout test actually cover the split in Task SDK
   * #72599 - Fix GET /assets/{asset_id} returning 500 when asset events share a timestamp
   * #72831 - Show detailed API error messages in UI toasters
   
   These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
   
   * #70132 - Fix output type loss in AgentOperator's human-in-the-loop review
   * #70433 - Decode base64 for bytes-typed HookToolset parameters
   * #70541 - Allow providers to ship testing Dags separately from example Dags
   * #71003 - Reject unknown update_mask fields instead of silently ignoring them
   * #71203 - Speed up clearing downstream tasks on large Dags
   * #72296 - Add deadline alert coverage to Dag endpoint tests
   
   This is **not** a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.
   
   What we ask you to do is to make your **first prioritization decision**: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or `gh pr reopen <PR_NUMBER> --repo apache/airflow`. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.
   
   While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.
   
   Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.
   
   ---
   Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org