You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/07 09:58:25 UTC
[GitHub] [airflow] sivieri opened a new issue, #24279: Rules for upstream_failed transmission not clear
sivieri opened a new issue, #24279:
URL: https://github.com/apache/airflow/issues/24279
### What do you see as an issue?
Looking at the [documentation](https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#concepts-trigger-rules) about trigger rules, it is not clear how the `upstream_failed` status is transmitted downstream.
The documentation contains several details about how the rules behave with skipped tasks, but no information (that I could find) is dedicated to `upstream_failed`, which is equally important in my opinion.
For instance, from past experience I understand that:
- a task with multiple upstream dependencies and `all_success` is immediately executed as soon as one of the upstream tasks fails (or upstream-fails due to its own upstream dependencies), and it goes into a failed state
- a task with multiple upstream dependencies and `all_done` does wait for all upstream dependencies, regardless of the fact that one of them has `upstream_failed`, and it is fully executed, it does not go automatically into a failed state.
The use case that prompted my question is the fact that I have a task with multiple upstream dependencies, and if one of those fails or receives an upstream failure (and goes into `upstream_failed`), it should wait for all the other dependencies to arrive at some state before being executed (and failing as well, so `all_done` would not be enough).
### Solving the problem
If possible, it would be nice to have a section, similar to the skipped tasks one, that explains how the failures (and upstream failures) are transmitted down the graph.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sivieri commented on issue #24279: Rules for upstream_failed transmission not clear
Posted by GitBox <gi...@apache.org>.
sivieri commented on issue #24279:
URL: https://github.com/apache/airflow/issues/24279#issuecomment-1148597364
@potiuk I could open a PR, but the main reason I opened this issue is that I would hope that someone already knows how the `upstream_failed` status trickles downstream, while I only know the two examples I made above (and there are a larger number of trigger rules).
I would also like to understand if there is a solution for the use case I have written above (for which I can put together an example if needed), and if not I would think that there is something missing in the functionality itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24279: Rules for upstream_failed transmission not clear
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24279:
URL: https://github.com/apache/airflow/issues/24279#issuecomment-1148643248
It's realy just about setting expectations right :)
* if you help with that and open PR, you might achieve the goal fast (and you can control the progress)
* if you don't, the docs might or might not get updated, depending if somoene picks an interest about
Just this - no more, no less.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sivieri commented on issue #24279: Rules for upstream_failed transmission not clear
Posted by GitBox <gi...@apache.org>.
sivieri commented on issue #24279:
URL: https://github.com/apache/airflow/issues/24279#issuecomment-1148655952
I think it might be better to open a discussion then, and then it could be turned into a PR. I did not see the discussion page, I will do that for now. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24279: Rules for upstream_failed transmission not clear
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24279:
URL: https://github.com/apache/airflow/issues/24279#issuecomment-1148628174
> @potiuk I could open a PR, but the main reason I opened this issue is that I would hope that someone already knows how the `upstream_failed` status trickles downstream, while I only know the two examples I made above (and there are a larger number of trigger rules). I would also like to understand if there is a solution for the use case I have written above (for which I can put together an example if needed), and if not I would think that there is something missing in the functionality itself.
I think if it is not documented, then the only way to check is to look it up in the code. And then person, who has already made a considerable effort to look up is llikely the best to document it. Trying to document it, is likely best way to learn it (especially that you will get maintainer's help with that).
There is no "must" here of course - but if you will not do it then it will have to wait for someone who would like to. That's why I marked it as "good first issue" - sounds a good issue for someone who would like to spend their time on digging in.
Also you might not understand how open-source works. Issues will get implemented because someone would like to take care about them, not because they are opened. So if your goal was to "create the documentation for it", the most certain way is to implement it by you. Otherwise it will have to wait for someone to pick it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sivieri closed issue #24279: Rules for upstream_failed transmission not clear
Posted by GitBox <gi...@apache.org>.
sivieri closed issue #24279: Rules for upstream_failed transmission not clear
URL: https://github.com/apache/airflow/issues/24279
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #24279: Rules for upstream_failed transmission not clear
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24279:
URL: https://github.com/apache/airflow/issues/24279#issuecomment-1148456636
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24279: Rules for upstream_failed transmission not clear
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24279:
URL: https://github.com/apache/airflow/issues/24279#issuecomment-1148489530
@sivieri - I think you are one of the best people who could make a PR to describe it. You feel the need, You know where it shoudl be, you've already done research on it. And once you open a PR, maintainers will be able to verify your proposed descrption and point you in the direction of explaining it better and more precise if you make mistake - and you can learn a lot during this task. Happy to help with that.
Airlflow (including the airflow documentation) is created by more than 2000 contributors - like you. Adding helpful documentaiton like that is really good and rather simple way to become a contrbutor as well. And good way to help other people like you to make good use of Airflow.
And tt's super easy to create a Pulll Request (even easier than opening an issue like this one). Simply go to the page you want to edit, click "Suggest a change on this page" and you will get a PR opened for you in Github UI (so basically you do not need more skills or development environment than those you already shown by opening the issue).
Shall I assign you this issue @sivieri
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24279: Rules for upstream_failed transmission not clear
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24279:
URL: https://github.com/apache/airflow/issues/24279#issuecomment-1148646696
BTW, If what you really wanted to achieve is to get a better understanding on how it works - then better open a GitHub Discussion about it - https://github.com/apache/airflow/discussions - then it can be turned into a PR without necessarily even opening an issue about it :D. I only asked about your willingness because you opened and issue about it with "bug" label which indicated something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org