[GitHub] [airflow] sartyukhov opened a new issue #21331: Retry for TaskGroup

[GitBox <gi...@apache.org>]

sartyukhov opened a new issue #21331:
URL: https://github.com/apache/airflow/issues/21331


   ### Description
   
   Hello!
   
   Previously, a SubDag was used to organize tasks into groups. Now you've introduced a TaskGroups to the world .
   It's nice and very clever. But it has a one big disadvantage over the SubDag - it cant be repeated.
   
   ### Use case/motivation
   
   For example:
   
   In a project I have two task (A >> B):
   A - collect data (PythonOperator)
   B - update material view in postgres (PostgresOperator)
   
   'A' could collect only part of data and mark itself as failed (there is no "half-failed" status as I know). But task 'B' should run regardless of A`s result (trigger_rule="all_done" for example) to update matview with part of data.
   
   With SubDag I could just initiate SubDag (with this tasks inside) with parameter _retries=1_ and that's it. 
   But TaskGroup does not have retry parameter.
   I also can't retry whole DAG, because it's big. 
   I also don't want to update material view inside task 'A' because in that way I can't do [A0, A1..An] >> B (update material view just once for several collects).
   
   I hope it's possible. Or maybe it could be done some other way.
   Thanks in advance.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org