You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "anzinmhd (via GitHub)" <gi...@apache.org> on 2026/10/04 11:30:43 UTC
[PR] Fix mapped task group short circuit skipping [airflow]
anzinmhd opened a new pull request, #74188:
URL: https://github.com/apache/airflow/pull/74188
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
## What is the issue?
When a `ShortCircuitOperator` is used inside a mapped task group, transitive downstream tasks can incorrectly execute even when the short-circuit condition is `False`.
The `ShortCircuitOperator` records all downstream tasks that should be skipped in its `SkipMixin` XCom. However, `NotPreviouslySkippedDep` only checks the direct upstream relatives of a task.
As a result, a transitive downstream task with a permissive trigger rule such as `all_done` can run even though it was explicitly recorded as skipped by the upstream `ShortCircuitOperator`.
For example:
```text
group.gate
|
group.a
|
group.b (all_done)
```
For a short-circuited map index, `group.a` is skipped correctly, but `group.b` can incorrectly run.
## What does this PR change?
Update `NotPreviouslySkippedDep` to inspect all upstream relatives instead of only direct upstream relatives.
This allows the dependency to find the upstream `SkipMixin` task and honor its recorded skip decision for transitive downstream tasks.
## Tests
Added a regression test covering:
* A `ShortCircuitOperator` inside a mapped task group.
* A transitive downstream task using `TriggerRule.ALL_DONE`.
* Skip decisions for a specific map index.
* Ensuring the affected map index is skipped.
* Ensuring the unaffected map index is not skipped.
## Validation
The following checks were run successfully:
* Focused `NotPreviouslySkippedDep` unit tests.
* Ruff checks for the modified files.
* Manual reproduction of the reported issue.
The manual reproduction now results in:
```text
Map 0:
gate -> SUCCESS
a -> SUCCESS
b -> SUCCESS
Map 1:
gate -> SUCCESS
a -> SKIPPED
b -> SKIPPED
```
closes: #74175
---
##### Was generative AI tooling used to co-author this PR?
* [x] Yes (ChatGPT)
<!--
Generated-by: ChatGPT following the guidelines:
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions
-->
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix mapped task group short circuit skipping [airflow]
Posted by "anzinmhd (via GitHub)" <gi...@apache.org>.
anzinmhd commented on PR #74188:
URL: https://github.com/apache/airflow/pull/74188#issuecomment-5996087992
@shahar1, I’ve addressed the requested changes from your review and pushed the updates.
The fix now walks transitive upstream relatives while applying the followed rule only to direct parents, and I’ve also added the Branch → Join regression test and updated the NotPreviouslySkippedDep docstring. The requested formatting and validation checks have also been completed.
Could you please take another look when you have a chance? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix mapped task group short circuit skipping [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #74188:
URL: https://github.com/apache/airflow/pull/74188#issuecomment-5979474485
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix mapped task group short circuit skipping [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #74188:
URL: https://github.com/apache/airflow/pull/74188#issuecomment-5999849036
Closing in favour of #74283.
The cause you found is the right one. The problem is `get_flat_relatives` runs for every task in every Dag, so each task below any `SkipMixin` task does its own XCom query on each scheduling pass. I measured a plain unmapped Dag (a short-circuit and 1999 tasks after it): 1999 queries per pass instead of 1, 0.64 s instead of 0.029 s on SQLite.
#74283 only reads skip decisions for tasks inside a mapped task group, with one query per pass, and your branch + join case is one of its tests. If it doesn't land we can reopen this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix mapped task group short circuit skipping [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed pull request #74188: Fix mapped task group short circuit skipping
URL: https://github.com/apache/airflow/pull/74188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix mapped task group short circuit skipping [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #74188:
URL: https://github.com/apache/airflow/pull/74188#discussion_r4178451878
##########
airflow-core/src/airflow/ti_deps/deps/not_previously_skipped_dep.py:
##########
@@ -46,7 +46,7 @@ class NotPreviouslySkippedDep(BaseTIDep):
def _get_dep_statuses(self, ti, dep_context, *, session):
from airflow.utils.state import TaskInstanceState
- upstream = ti.task.get_direct_relatives(upstream=True)
+ upstream = ti.task.get_flat_relatives(upstream=True)
Review Comment:
**Blocking:** this regresses every branch + join Dag.
`SkipMixin.skip_all_except` stores `{"followed": [...]}` with only the **direct** children it followed. The `followed` rule below ("skip anything not in `followed`") now fires for every transitive descendant of a branch operator, so a `join` downstream of `task1` gets `SKIPPED`:
```text
branch --> task1 --> join (none_failed_min_one_success)
\-----> task2 --/
```
Verified locally on this branch: `get_dep_statuses(join)` returns `passed=False, reason='Skipping because of previous XCom result from parent task branch'`; on `main` it returns `[]`.
Keep the flat walk, but only apply the `followed` rule to direct parents (the explicit `skipped` list is safe from any ancestor):
```suggestion
direct_upstream_ids = {t.task_id for t in ti.task.get_direct_relatives(upstream=True)}
upstream = ti.task.get_flat_relatives(upstream=True)
```
and guard the `followed` branch with `parent.task_id in direct_upstream_ids and ...`. With that, your new test, the existing suite and a branch + join probe all pass (10/10). Full probe test in the review body; please add it to this file.
Also worth noting in the description: `get_flat_relatives` walks all ancestors on every dep evaluation, which is more scheduler work than the previous direct lookup on large Dags.
##########
airflow-core/tests/unit/ti_deps/deps/test_not_previously_skipped_dep.py:
##########
@@ -262,6 +263,56 @@ def group(value):
assert dep.is_met(tis[("group.child", 0)], session=session)
assert tis[("group.child", 0)].state != State.SKIPPED
+def test_parent_in_mapped_task_group_skips_transitive_downstream(session, dag_maker):
Review Comment:
nit: `ruff format` wants two blank lines before this `def` (and reflows the `>>` chain below). The static-checks job will fail as-is; `prek run ruff-format --from-ref main` fixes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix mapped task group short circuit skipping [airflow]
Posted by "anzinmhd (via GitHub)" <gi...@apache.org>.
anzinmhd commented on PR #74188:
URL: https://github.com/apache/airflow/pull/74188#issuecomment-5996000906
@kaxil, the maintainer suggested getting a second look from you since this PR touches the skip-dependency changes from #73959. I've updated `NotPreviouslySkippedDep` to handle transitive skip decisions while keeping the `followed` behavior limited to direct parents, and added a Branch → Join regression test. Would appreciate your review when you have a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org