You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "salimmoulouel (via GitHub)" <gi...@apache.org> on 2023/10/24 10:06:38 UTC
[I] make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish [airflow]
salimmoulouel opened a new issue, #35149:
URL: https://github.com/apache/airflow/issues/35149
### Description
currently the one_failed, launch the tasks just after one of the upstream tasks failed, it can be good to be able to force the task wait the whole upstream tasks finish to be launched
### Use case/motivation
move file to error after been used by multiple upstream tasks, currently it can move the file before the whole upstream tasks finish their job
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #35149:
URL: https://github.com/apache/airflow/issues/35149#issuecomment-1845582073
This use case doesnt make sense to me.
I assume your real case is:
A >> [C, D]
B >> [ C, E]
Where A and B are branch operator.
When A say to follow to D (thus skip C) and B say follow to C (Thus skip E) you end up in a wierd case as its not so clear if C should be wxecuted or not. C upstream tasks (A and B) are both success either way.
Please explain the logic of how you decide if C should be execute or not. I think that this case should not happen to begin with.. having to branch operators pointing to the exact same task is very odd. If you must do so I would use EmptyOperato to decouple the two branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish [airflow]
Posted by "salimmoulouel (via GitHub)" <gi...@apache.org>.
salimmoulouel commented on issue #35149:
URL: https://github.com/apache/airflow/issues/35149#issuecomment-1845672576
attached a picture of the dag
If any of the four black-surrounded tasks completes and sends a skip signal to the green-surrounded task, the green-surrounded task will not wait for the others to finish. However, my concern lies with the red task, as it requires all four black-surrounded tasks to finish before launching, as it involves moving a file utilized by other black tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish [airflow]
Posted by "salimmoulouel (via GitHub)" <gi...@apache.org>.
salimmoulouel commented on issue #35149:
URL: https://github.com/apache/airflow/issues/35149#issuecomment-1845776165
https://github.com/apache/airflow/issues/7858
i found an issue which look like mine.
@eladkal thank you for time spent to understand my problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish [airflow]
Posted by "salimmoulouel (via GitHub)" <gi...@apache.org>.
salimmoulouel commented on issue #35149:
URL: https://github.com/apache/airflow/issues/35149#issuecomment-1845567233
Sorry i didn't describe well my problem bacause i was beginner when i wrote the issue
A >> D
B >> D
A and B are BranchOperator, when one of them send skip signal to D, D is skipped without waiting all the previous task to finish even if it's "all_done" trigger rule, i hope i'm clear now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #35149:
URL: https://github.com/apache/airflow/issues/35149#issuecomment-1846790127
Great.
Closing then as no issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #35149: make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish
URL: https://github.com/apache/airflow/issues/35149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #35149:
URL: https://github.com/apache/airflow/issues/35149#issuecomment-1843394403
> wait the whole upstream tasks finish to be launched
How is that different from `all_done` trigger rule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make a trigger_rule for one upstream failed but wait the whole upstream tasks to finish [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #35149:
URL: https://github.com/apache/airflow/issues/35149#issuecomment-1776910378
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org