You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hoiekim (via GitHub)" <gi...@apache.org> on 2025/12/10 22:47:03 UTC
[I] Dynamic Task Mapping Skips Before Upstream Has Started [airflow]
hoiekim opened a new issue, #59289:
URL: https://github.com/apache/airflow/issues/59289
### Apache Airflow version
Other Airflow 2/3 version (please specify below)
### If "Other Airflow 2/3 version" selected, which one?
3.0.6
### What happened?
"Dynamic Task Mapping Skips Before Upstream Has Started"
This is a similar bug as https://github.com/apache/airflow/issues/28973
Difference is that adding `"trigger_rule": TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS` makes the bug happens.
### What you think should happen instead?
Should not skip tasks
### How to reproduce
```
from datetime import datetime
from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.decorators import dag, task, task_group
from airflow.utils.trigger_rule import TriggerRule
@task
def get_the_list():
return ["one", "two", "three"]
@task
def process_something(x: str, y: str, **kwargs):
output = f"{x}_{y}"
print(output)
return output
@task_group
def the_task_group(x: str, y: str, **kwargs):
start = EmptyOperator(task_id="start")
something = process_something(x, y)
another = process_something(something)
end = EmptyOperator(task_id="end")
start >> something >> another >> end
@dag(
default_args={
"start_date": datetime(year=2025, month=12, day=1),
"trigger_rule": TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS,
}
)
def dtm_bug_reproduce_dag():
start = EmptyOperator(task_id="start")
end = EmptyOperator(task_id="end")
the_list = get_the_list()
mapped_group = (
the_task_group
.partial(x="Unlimited")
.expand(y=the_list)
)
start >> the_list >> mapped_group >> end
dag = dtm_bug_reproduce_dag()
```
### Operating System
MWAA
### Versions of Apache Airflow Providers
_No response_
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else?
I was able to reproduce the bug with the attached snippet in 3.0.6. I haven't tried it in the latest version but I did search through the issues and haven't found it reported so I think it's likely that it's not yet fixed.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic Task Mapping Skips Before Upstream Has Started [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on issue #59289:
URL: https://github.com/apache/airflow/issues/59289#issuecomment-3765217741
I have checked the `main` branch using a variety of DAG shapes following the same theme as the one in the bug report and the issue is still present. So it appears any recent work in this specific area (that is separate from PR #59691) which has been merged into `main`, has not resolved the bug.
As of this comment, I cannot see any open PRs that could resolve the issue (anyone reading this comment is free to correct me).
PR #59691 has been rebased and is ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic Task Mapping Skips Before Upstream Has Started [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on issue #59289:
URL: https://github.com/apache/airflow/issues/59289#issuecomment-4060773268
@eladkal
I can confirm it is present in 3.1.x as well. So adding a label for that would be helpful for triage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic Task Mapping Skips Before Upstream Has Started [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #59289:
URL: https://github.com/apache/airflow/issues/59289#issuecomment-3639263836
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic Task Mapping Skips Before Upstream Has Started [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on issue #59289:
URL: https://github.com/apache/airflow/issues/59289#issuecomment-4373069544
@potiuk
This issue is still present in the main branch i.e. 3.2.x. I checked using the exact same DAG shapes as before with matching failure modes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic Task Mapping Skips Before Upstream Has Started [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on issue #59289:
URL: https://github.com/apache/airflow/issues/59289#issuecomment-3678904446
@hoiekim
I was able to reproduce this bug in 3.0.6 as well 3.1.5 and the main branch (which might contain fixes that have not been released). It is caused by the placeholder for a task (in this case `another = process_something`) in the task group being replaced before the placeholder for the task in the same task group immediately downsteram of it (`end = EmptyOperator`) is able to resolve it's upstream counterpart. The bug you reported is a symptom of this problem as it silently breaks trigger rule evaluation for task instances. It readily surfaces when the `"trigger_rule"` is set to `TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS` and when there is a terminal task outside the task group on which the mapped tasks converge but I believe this could affect any DAG with mapped task groups structured like the one in this bug report.
Anyway, I have opened a PR (#59691) that should fix this bug. Please ignore the previous PR (#59561) as it was incorrect. It has been closed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic Task Mapping Skips Before Upstream Has Started [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla closed issue #59289: Dynamic Task Mapping Skips Before Upstream Has Started
URL: https://github.com/apache/airflow/issues/59289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org