Re: Add retry option for TaskGroup

[Rafal Biegacz <ra...@google.com.INVALID>]

Hi,

Up to now I considered TaskGroup mostly as a "visual feature". If we start
adding add'l "subdag" features to it we are going to gravitate to a
"subdag" concept, which is about to be dropped.

If I had a need to have a more complex logic in a group of tasks then I
would use the approach of DAGs triggering other DAGs via
TriggerDagRunOperator.

Left my comment in:
https://github.com/apache/airflow/issues/21867#issuecomment-1094065646

Regards, Rafal.

On Sun, Mar 13, 2022 at 9:28 PM Elad Kalif <el...@apache.org> wrote:

> The relationship between task group retry and task retry may not be that
> simple.
>
> I commented on the issue:
>
> https://github.com/apache/airflow/issues/21867#issuecomment-1054191418
>
> בתאריך יום א׳, 13 במרץ 2022, 21:33, מאת Tomasz Urbaszek ‏<
> turbaszek@apache.org>:
>
>> Hey folks,
>>
>> There's an interesting feature request for retry option on TaskGroup
>> level:
>> https://github.com/apache/airflow/discussions/21333
>>
>> I'm not sure if we can handle this nicely. Any opinions would be welcome!
>>
>> Cheers,
>> Tomek
>>
>

Re: Add retry option for TaskGroup

[Jarek Potiuk <ja...@potiuk.com>]

I think I share Elad's and Rafał's (and I feel Tomek's since he raised it)
sentiment,

Up till now it was a "visual feature" and yeah "subdag" left a bad taste,
so if this is going to be implemented, I'd say AIP with deep discussion and
considering all edge cases is needed first.
I agree with Elad this is not something that should be taken lightly and
"settled" after PR discussion.

The Dynamic Task Mapping implementation has shown that things that look
simple on the surface, can have multiple edge cases (and in this case I see
potential for things like deadlocks, lifecycle problems). Very similarly as
in a recent discussion we had about Composite Sensors (where also Task
Groups were mentioned as a possible approach BTW)
https://github.com/apache/airflow/issues/22816#issuecomment-1092584037).

All the mentioned issues have one thing in common - instead of acting on
one object, you want to act on multiple "object" - and provide at least
some level of atomic operations. This ain't easy when it comes to
distributed operations at scale.

It's really an order of magnitude more complex problem when you act and
mutate a single entity vs. when you act and mutate multiple of those in a
distributed system.

J.



On Sat, Apr 9, 2022 at 5:21 PM Rafal Biegacz
<ra...@google.com.invalid> wrote:

> Hi,
>
> Up to now I considered TaskGroup mostly as a "visual feature". If we start
> adding add'l "subdag" features to it we are going to gravitate to a
> "subdag" concept, which is about to be dropped.
>
> If I had a need to have a more complex logic in a group of tasks then I
> would use the approach of DAGs triggering other DAGs via
> TriggerDagRunOperator.
>
> Left my comment in:
> https://github.com/apache/airflow/issues/21867#issuecomment-1094065646
>
> Regards, Rafal.
>
> On Sun, Mar 13, 2022 at 9:28 PM Elad Kalif <el...@apache.org> wrote:
>
>> The relationship between task group retry and task retry may not be that
>> simple.
>>
>> I commented on the issue:
>>
>> https://github.com/apache/airflow/issues/21867#issuecomment-1054191418
>>
>> בתאריך יום א׳, 13 במרץ 2022, 21:33, מאת Tomasz Urbaszek ‏<
>> turbaszek@apache.org>:
>>
>>> Hey folks,
>>>
>>> There's an interesting feature request for retry option on TaskGroup
>>> level:
>>> https://github.com/apache/airflow/discussions/21333
>>>
>>> I'm not sure if we can handle this nicely. Any opinions would be welcome!
>>>
>>> Cheers,
>>> Tomek
>>>
>>