You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dv-matthew-crown (via GitHub)" <gi...@apache.org> on 2025/09/03 18:18:42 UTC
[I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
dv-matthew-crown opened a new issue, #55225:
URL: https://github.com/apache/airflow/issues/55225
### Apache Airflow version
3.0.6
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
The short circuit operator does not skip mapped tasks when trigger rule is set to none_failed and ignore_downstream_trigger_rules is set to True
### What you think should happen instead?
I would expect that mapped task instances would be skipped if the short circuit task returns a False value, but instead they are executed
### How to reproduce
```from airflow.sdk import Asset, dag, task
a = Asset("test_tmp_dag")
b = Asset("test_tmp_dag2")
@dag(dag_id="test_tmp_dag", schedule=None, default_args={"trigger_rule": "none_failed"})
def test_tmp_dag():
multi = [1,2,3]
@task.short_circuit(task_id="dummy1", ignore_downstream_trigger_rules=True)
def dummy1():
if 2 == 2:
return False
else:
return "dummy1"
@task(task_id="dummy2")
def dummy2():
return "dummy2"
@task(task_id="dummy3")
def dummy3(x: int):
return f"dummy3-{x}"
@task(task_id="publish", outlets=[a,b])
def publish():
return "published"
d1 = dummy1()
d2 = dummy2()
d3 = dummy3.expand(x=multi)
p = publish()
# The result of the task function call is a Task object
d1 >> d2 >> d3 >> p
test_tmp_dag()
```
This dag demonstrates the problem - dummy2 and publish are correctly skipped, but the mapped tasks in dummy3 are executed.
### Operating System
Ubuntu 24.04.2 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.12.0
apache-airflow-providers-celery==3.12.2
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.5
apache-airflow-providers-http==5.3.3
apache-airflow-providers-postgres==6.2.3
apache-airflow-providers-smtp==2.2.0
apache-airflow-providers-standard==1.6.0
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "dv-matthew-crown (via GitHub)" <gi...@apache.org>.
dv-matthew-crown commented on issue #55225:
URL: https://github.com/apache/airflow/issues/55225#issuecomment-5910741417
yep, i have been seeing this consistently since i reported and your minimal example shows the same state - @vatsrahul1001 can we reopen ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "suhail-zemoso (via GitHub)" <gi...@apache.org>.
suhail-zemoso commented on issue #55225:
URL: https://github.com/apache/airflow/issues/55225#issuecomment-3251998157
I would like to work on this issue--Let me know if it's being picked by someone else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "dv-matthew-crown (via GitHub)" <gi...@apache.org>.
dv-matthew-crown commented on issue #55225:
URL: https://github.com/apache/airflow/issues/55225#issuecomment-3259436147
@suhail-zemoso what is also interesting, is that despite the mapped tasks failing the overall DAG is marked as successful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #55225:
URL: https://github.com/apache/airflow/issues/55225#issuecomment-3250284506
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #55225:
URL: https://github.com/apache/airflow/issues/55225#issuecomment-3460049144
Not able to reproduce this with version 3.1.1. Closing the issue. Please feel free to reopen if you’re still facing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #55225:
URL: https://github.com/apache/airflow/issues/55225#issuecomment-5914092644
Reopening: both reports reproduce on main, and they are two separate bugs.
1. **Unmapped short-circuit with a mapped task further downstream** (the original report). The skip request only matched the `map_index=-1` row of a mapped task. A task mapped over a literal list already has task instances `0..n-1` when the run is created, so none of them were skipped, and `none_failed` let them run. This has been the case since 3.0. Fix: #73960.
2. **Short-circuit inside a mapped task group** (the example above). Since #62287 (3.2.0), `NotPreviouslySkippedDep` reads the skip decision at `map_index=-1` unless the parent is a mapped operator. A task inside a mapped task group writes that decision per map index, so it was never found. Fix: #73959.
In the playground example, `group_2.printstr` will be skipped for every map index once any `group_1.printstr` is skipped, not about half the time. `group_2` is a separate mapped task group, so each of its instances depends on all `group_1.printstr` instances, and the default `all_success` rule skips it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "dv-matthew-crown (via GitHub)" <gi...@apache.org>.
dv-matthew-crown commented on issue #55225:
URL: https://github.com/apache/airflow/issues/55225#issuecomment-3259434821
My mapped tasks require trigger rule none_failed when running normally, so this would not be a solution in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "Chais (via GitHub)" <gi...@apache.org>.
Chais commented on issue #55225:
URL: https://github.com/apache/airflow/issues/55225#issuecomment-5907674742
Commenting because I see this misbehaviour in 3.3.0. In fact it looks like short circuiting doesn't work as soon as mapped tasks are involved.
Minimal example:
```py
from datetime import datetime
from random import random, sample
from string import ascii_letters
from typing import List
from zoneinfo import ZoneInfo
from airflow.sdk import dag, task, task_group
@task.short_circuit()
def no() -> bool:
return False
@task.short_circuit()
def fifty_fifty() -> bool:
return random() > 0.5
@task.python()
def printstr(value: str) -> None:
print(value)
@task.python()
def get_rand_strs(s: int, k: int) -> List[str]:
return ["".join(sample(ascii_letters, k)) for _ in range(s)]
@dag(
"playground",
description="Try random things",
schedule=None,
start_date=datetime(2024, 11, 1, 0, 0, 0, 0, ZoneInfo("Europe/Berlin")),
catchup=False,
dag_display_name="Playground",
)
def playground():
@task_group()
def group_1(somestr: str):
(
fifty_fifty()
>> printstr(somestr) # Should be skipped about half the time, but isn't
)
@task_group()
def group_2(somestr: str):
printstr(f"{somestr} again") # Should also be skipped about half the time, isn't either
strings = get_rand_strs(20, 10)
group_1.expand(somestr=strings) >> group_2.expand(somestr=strings)
no() >> printstr("Should be skipped and is.")
playground()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 closed issue #55225: short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks
URL: https://github.com/apache/airflow/issues/55225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "suhail-zemoso (via GitHub)" <gi...@apache.org>.
suhail-zemoso commented on issue #55225:
URL: https://github.com/apache/airflow/issues/55225#issuecomment-3259272637
@dv-matthew-crown
If you add trigger_rule="all_success" to the mapped task (dummy3 and published), we can achieve the expected behavior, which is that dummy2, dummy3, and published will be skipped.
<img width="1846" height="886" alt="Image" src="https://github.com/user-attachments/assets/7d1f14fc-c353-41da-9709-9cdb18d9c10c" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #55225: short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks
URL: https://github.com/apache/airflow/issues/55225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org