You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@apache.org> on 2020/09/18 12:33:52 UTC
[DISCUSS] Deprecate SubDags in favor of TaskGroups ?
Hi all,
One of the things we discussed on Monday's Airflow 2.0 Dev call was around
*TaskGroups* - a new concept introduced by
https://github.com/apache/airflow/pull/10153 (AIP-34
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator>)
from
Qian Yu.
There was a point raised in the meeting that we should deprecate SubDags in
favour of TaskGroups since TaskGroups are UI-only concept and do not impact
scheduling decisions or have known limitations around Concurrency / Pools
limits like SubDags.
Having thought. a little bit more after the meeting, I think we should
hold-off on Deprecating Subdags in 2.0. We can wait until 2.2-2.3 and see
how users feel about TaskGroup and we, the developers would also know more
about limitations of it (if any).
We can then issue a Deprecation Warning and remove SubDags eventually in
Airflow 3.0. Until then both can live in the codebase and we document the
TaskGroups better.
What do you all think?
Meeting Notes from that call are at:
https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#4:14Sep2020
<https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020>
Regards,
Kaxil
Re: [DISCUSS] Deprecate SubDags in favor of TaskGroups ?
Posted by Jarek Potiuk <Ja...@polidea.com>.
One thing only here. While I am all for removing Sub-dags eventually, We
either remove them in 2.0 or 3.0. We can't remove any functionality or
change Airflow in a backwards-incompatible way in 2.X (
https://semver.org/#spec-item-8)
We already agreed I believe, that we are using Semver <https://semver.org/>
and that basically means that there are no intentional
backwards-incompatible changes when Minor version is increased.
I think we ourselves experienced that quite a number of times when our
dependencies did not keep to their semver promise, so let's not do that to
others :).
Just a thought - I wonder if deprecating SubDags now and removing it in
2.0 is a possibility?
I think the more we can remove from Airflow when getting to 2.0 - the
better. We still have a chance to ADD stuff in 2.1 and beyond. It can then
be better thought, more integrated with the future of Airflow as we see it.
If we will get information from the users that they miss it after we
removed it in 2.0 and they cannot live without it and cannot migrate to 2.0
without it, we can always add it. We have a viable TaskGroup replacement
now. And maybe we can even get some semi-automated conversions SubDags ->
TaskGroups.
J.
On Fri, Sep 18, 2020 at 11:30 PM Daniel Imberman <da...@gmail.com>
wrote:
> I agree with Gerard on all fronts.
>
> SubDags are difficult, slow, and can cause a lot of strange edge cases.
> The difficulties in SubDags is a sticking point for Airflow competitors
> that I want to remove as quickly as possible. I think that 2.0 is a perfect
> time to introduce TaskGroups as people are already prepared mentally for
> breaking changes/new functionality.
>
> I don’t think we need to give a specific “this will be around until 3.0”
> message either. We can just say it WILL be taken out in a future version
> and we can decide later whether that means 2.1 or 3.0. I really don’t like
> the idea of setting a feature for deprecation in a minor release. I also
> think that even if TaskGroups ends up not being the answer, it’s very
> likely that it will be a different feature, rather than a reversion to
> SubDags.
>
> via Newton Mail [
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2
> ]
> On Fri, Sep 18, 2020 at 2:16 PM, Gerard Casas Saez <gc...@twitter.com.invalid>
> wrote:
> Internally, we have blocked people from using SubDags due to its slowness
> and sometimes instability. I would vote for adding TaskGroups earliest as
> possible + remove documentation for SubDags (users can go to old versions
> of docs to find that) + add a notice of deprecation on new TaskGroup
> documentation page + Add deprecation warning on import/usage SubDag.
>
> I would vote for not having to wait until 3.X for deprecating it as it's
> difficult to use and adds complexity that may not be needed. Instead
> schedule removal at 2.1 or so.
>
> This is the take from our side that we don't have SubDags, would be curious
> on people usages of SubDags and to check if there's any use case that does
> not fit TaskGroups.
>
> Gerard Casas Saez
> Twitter | Cortex | @casassaez <http://twitter.com/casassaez>
>
>
> On Fri, Sep 18, 2020 at 6:34 AM Kaxil Naik <ka...@apache.org> wrote:
>
> > Hi all,
> >
> > One of the things we discussed on Monday's Airflow 2.0 Dev call was
> around
> > *TaskGroups* - a new concept introduced by
> > https://github.com/apache/airflow/pull/10153 (AIP-34
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator
> > >)
> > from
> > Qian Yu.
> >
> > There was a point raised in the meeting that we should deprecate SubDags
> in
> > favour of TaskGroups since TaskGroups are UI-only concept and do not
> impact
> > scheduling decisions or have known limitations around Concurrency / Pools
> > limits like SubDags.
> >
> > Having thought. a little bit more after the meeting, I think we should
> > hold-off on Deprecating Subdags in 2.0. We can wait until 2.2-2.3 and see
> > how users feel about TaskGroup and we, the developers would also know
> more
> > about limitations of it (if any).
> >
> > We can then issue a Deprecation Warning and remove SubDags eventually in
> > Airflow 3.0. Until then both can live in the codebase and we document the
> > TaskGroups better.
> >
> > What do you all think?
> >
> > Meeting Notes from that call are at:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#4:14Sep2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020
> >
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020
> > >
> >
> > Regards,
> > Kaxil
> >
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Deprecate SubDags in favor of TaskGroups ?
Posted by Daniel Imberman <da...@gmail.com>.
I agree with Gerard on all fronts.
SubDags are difficult, slow, and can cause a lot of strange edge cases. The difficulties in SubDags is a sticking point for Airflow competitors that I want to remove as quickly as possible. I think that 2.0 is a perfect time to introduce TaskGroups as people are already prepared mentally for breaking changes/new functionality.
I don’t think we need to give a specific “this will be around until 3.0” message either. We can just say it WILL be taken out in a future version and we can decide later whether that means 2.1 or 3.0. I really don’t like the idea of setting a feature for deprecation in a minor release. I also think that even if TaskGroups ends up not being the answer, it’s very likely that it will be a different feature, rather than a reversion to SubDags.
via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2]
On Fri, Sep 18, 2020 at 2:16 PM, Gerard Casas Saez <gc...@twitter.com.invalid> wrote:
Internally, we have blocked people from using SubDags due to its slowness
and sometimes instability. I would vote for adding TaskGroups earliest as
possible + remove documentation for SubDags (users can go to old versions
of docs to find that) + add a notice of deprecation on new TaskGroup
documentation page + Add deprecation warning on import/usage SubDag.
I would vote for not having to wait until 3.X for deprecating it as it's
difficult to use and adds complexity that may not be needed. Instead
schedule removal at 2.1 or so.
This is the take from our side that we don't have SubDags, would be curious
on people usages of SubDags and to check if there's any use case that does
not fit TaskGroups.
Gerard Casas Saez
Twitter | Cortex | @casassaez <http://twitter.com/casassaez>
On Fri, Sep 18, 2020 at 6:34 AM Kaxil Naik <ka...@apache.org> wrote:
> Hi all,
>
> One of the things we discussed on Monday's Airflow 2.0 Dev call was around
> *TaskGroups* - a new concept introduced by
> https://github.com/apache/airflow/pull/10153 (AIP-34
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator
> >)
> from
> Qian Yu.
>
> There was a point raised in the meeting that we should deprecate SubDags in
> favour of TaskGroups since TaskGroups are UI-only concept and do not impact
> scheduling decisions or have known limitations around Concurrency / Pools
> limits like SubDags.
>
> Having thought. a little bit more after the meeting, I think we should
> hold-off on Deprecating Subdags in 2.0. We can wait until 2.2-2.3 and see
> how users feel about TaskGroup and we, the developers would also know more
> about limitations of it (if any).
>
> We can then issue a Deprecation Warning and remove SubDags eventually in
> Airflow 3.0. Until then both can live in the codebase and we document the
> TaskGroups better.
>
> What do you all think?
>
> Meeting Notes from that call are at:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#4:14Sep2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020
> >
>
> Regards,
> Kaxil
>
Re: [DISCUSS] Deprecate SubDags in favor of TaskGroups ?
Posted by Gerard Casas Saez <gc...@twitter.com.INVALID>.
Internally, we have blocked people from using SubDags due to its slowness
and sometimes instability. I would vote for adding TaskGroups earliest as
possible + remove documentation for SubDags (users can go to old versions
of docs to find that) + add a notice of deprecation on new TaskGroup
documentation page + Add deprecation warning on import/usage SubDag.
I would vote for not having to wait until 3.X for deprecating it as it's
difficult to use and adds complexity that may not be needed. Instead
schedule removal at 2.1 or so.
This is the take from our side that we don't have SubDags, would be curious
on people usages of SubDags and to check if there's any use case that does
not fit TaskGroups.
Gerard Casas Saez
Twitter | Cortex | @casassaez <http://twitter.com/casassaez>
On Fri, Sep 18, 2020 at 6:34 AM Kaxil Naik <ka...@apache.org> wrote:
> Hi all,
>
> One of the things we discussed on Monday's Airflow 2.0 Dev call was around
> *TaskGroups* - a new concept introduced by
> https://github.com/apache/airflow/pull/10153 (AIP-34
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator
> >)
> from
> Qian Yu.
>
> There was a point raised in the meeting that we should deprecate SubDags in
> favour of TaskGroups since TaskGroups are UI-only concept and do not impact
> scheduling decisions or have known limitations around Concurrency / Pools
> limits like SubDags.
>
> Having thought. a little bit more after the meeting, I think we should
> hold-off on Deprecating Subdags in 2.0. We can wait until 2.2-2.3 and see
> how users feel about TaskGroup and we, the developers would also know more
> about limitations of it (if any).
>
> We can then issue a Deprecation Warning and remove SubDags eventually in
> Airflow 3.0. Until then both can live in the codebase and we document the
> TaskGroups better.
>
> What do you all think?
>
> Meeting Notes from that call are at:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#4:14Sep2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%234:14Sep2020
> >
>
> Regards,
> Kaxil
>