You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/28 11:42:19 UTC
[GitHub] [airflow] potiuk opened a new issue #21867: Retry for TaskGroup
potiuk opened a new issue #21867:
URL: https://github.com/apache/airflow/issues/21867
### Discussed in https://github.com/apache/airflow/discussions/21333
<div type='discussions-op-text'>
<sup>Originally posted by **sartyukhov** February 4, 2022</sup>
### Description
Hello!
Previously, a SubDag was used to organize tasks into groups. Now you've introduced a TaskGroups to the world .
It's nice and very clever. But it has a one big disadvantage over the SubDag - it cant be repeated.
### Use case/motivation
For example:
In a project I have two task (A >> B):
A - collect data (PythonOperator)
B - update material view in postgres (PostgresOperator)
'A' could collect only part of data and mark itself as failed (there is no "half-failed" status as I know). But task 'B' should run regardless of A`s result (trigger_rule="all_done" for example) to update matview with part of data.
In an ~ hour I would like to repeat that process (A >> B).
With SubDag I could do that:
- initiate SubDag with parameter _retries=10_
- add DummyTask 'C' with trigger_rule="all_success"
- change flow to A >> B >> C and A >> C
and that's it, C marks dag as failed and trigger it to retry.
But TaskGroup does not have retry parameter.
I also can't retry whole DAG, because it's big.
I also don't want to update material view inside task 'A' because in that way I can't do [A0, A1..An] >> B (update material view just once for several collects).
I hope it's possible. Or maybe it could be done some other way.
Thanks in advance.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
</div>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #21867: Retry for TaskGroup
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #21867:
URL: https://github.com/apache/airflow/issues/21867#issuecomment-1054191418
What if TakGroup contains of 20 tasks and one of them is HTTP that just retired on some trasist error. Do we really want to retry the whole task group because of it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] victorfuzaro commented on issue #21867: Retry for TaskGroup
Posted by GitBox <gi...@apache.org>.
victorfuzaro commented on issue #21867:
URL: https://github.com/apache/airflow/issues/21867#issuecomment-1054198574
> What if TakGroup contains of 20 tasks and one of them is HTTP that just retired on some trasist error. Do we really want to retry the whole task group because of it?
It'll depends on the use case. Of course if you have a lot of tasks in a group, maybe you won't need the entire group to be executed again. In my case for example (which is described in the "additional explanation" area, I have only 2 tasks in the same task group, definitely would be useful for small groups. This is just an additional option that users of task groups could have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sartyukhov commented on issue #21867: Retry for TaskGroup
Posted by GitBox <gi...@apache.org>.
sartyukhov commented on issue #21867:
URL: https://github.com/apache/airflow/issues/21867#issuecomment-1054224492
SubDagOperator which one is deprecated now inherits from BaseSensorOperator -> BaseOperator. And it has retries functional.
But TaskGroup inherits from TaskMixin only, and in short, it just adds group name prefix to tasks and draws pretty visual group on web interface.
I don't know how the inventor of TaskGroup would react, for example, to adding the BaseOperator inheritance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21867: Retry for TaskGroup
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21867:
URL: https://github.com/apache/airflow/issues/21867#issuecomment-1054173390
cc: @sartyukhov @victorfuzaro
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sartyukhov commented on issue #21867: Retry for TaskGroup
Posted by GitBox <gi...@apache.org>.
sartyukhov commented on issue #21867:
URL: https://github.com/apache/airflow/issues/21867#issuecomment-1054198766
> What if TakGroup contains of 20 tasks and one of them is HTTP that just retired on some trasist error. Do we really want to retry the whole task group because of it?
I think every developer can answer on this question himself. It depends on architecture of each dag...
Main question is how to decide when TaskGroup is failed?
- **last** task in a group is failed
- **any** task in a group is failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org