You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2024/07/11 17:27:51 UTC
[DISCUSS] To AIP-44 or not to AIP-44
Following up after today's Airflow 3 call, I wanted to start
discussion on whether we should continue implementing AIP-44 and
release it in Airflow 2 or whether we should focus on AIP-72 and
Airflow 3.
We could go both ways.
The implementation is somewhat 80% done - what we really miss there is
automation of testing it in CI to make sure that all DB access is
going through RPC (and complete remaining things that aren't). Daniel
particularly put a lot of effort into implementing some of the
remaining pieces and manually got the "task" working. also Jens had
done some manual testing with "remote executor" (AIP-69) and got it to
"work". But thorough testing - especially "forward-looking-testing" is
needed. Also wiring the authentication is needed (following the
webserver/log authentication we already have).
On one hand - I know Google Team is particularly interested in having
it completed, also Jens would like to use it to run the "AIP-69"
prototype in Airflow 2. if we don't implement AIP-44, AIP-69 will only
land in Airflow 3.
On the other hand, it will drag out some efforts from Airflow 3 and
will keep people out of implementing some of the changes we need. Also
if we implement and release it now - we will have to support and solve
issues Also in Airflow 2. Users who will deploy it, will have to redo
their deployment for Airflow 3. And without multi-team AIP-67 it will
only partially separate out the DB access (there will be no team
isolation, DAG authors will still be able to access "all other" DAGs,
tasks, connections, variables etc., but they won't be able to run
arbitrary DB operations).
We could go both ways, and yes - there is a lot of "Sunken cost".
I'd love to hear what others think about it.
J.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Jarek Potiuk <ja...@potiuk.com>.
All right - waiting for the last review on
https://github.com/apache/airflow/pull/40916 but I think we are
ready before the cut-off.
What nicely work there (at least manually checks):
a) airflow in breeze works out-of-the-box with any combination of:
* --db-isolation-mode
* --standalone-dag-processor
* Local and Celery Executor (Local Executor mostly because I wanted to test
if it works when run K8S Executor that intenrnally uses LocalExecutor
b) Authentication works nicely using separate internal_api_secret_key and
JWT signer.
c) Performance does not look bad actually at least on local machine - but I
am sure it can be optimized further.
d) we do not use https:// - but I think this should be described in the
documentation that SSL terminating proxies should be used in front of
internal-api to add SSL (usual practice). We might want to give a big
WARNING if someone uses HTTP:// in the client
What we will need more is:
a) special tests for isolation mode
b) documentation (stressing experimental status)
But those can be even added after the cut-off for 2.10 branch and
cherry-picked potentially.
I think particularly the Dag File Processor was an interesting one in light
of AIP-72 - I think it can be much more easily mapped almost 1-1 to
whatever AIP-72 brings. And nice thing about it that we can tell
adventurous users to use the internal API and report any issues they
encounter with lack of DB access on the side of worker and parser long
before AIP-72 will be out)
J.
On Fri, Jul 19, 2024 at 8:58 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Also one thing after spending some time on it - after discussion with Jens
> I already made the "DB Isolation" mode works end-to-end in breeze (
> https://github.com/apache/airflow/pull/40894) - which was fairly easy
> (and it seems to work pretty well/stable with some limitations (no
> mini-scheduler for now). Bt I am also working through making the standalone
> DAGFileProcessor working (it had missing bits and pieces) and I strongly
> believe that work will be extremely useful for AIP-72 as well because it
> turns out that there is also quite a some "spaggetish" DB / access at
> places so whatever refactoring and changes are implemented now, they should
> basically be a base for AIP-72 DAGFileProcessing API. It's not "com[plex"
> but a bit tedious - and it will show immediately where API calls will be
> needed as well.
>
> So that work is not **entirely* useless.
>
> J.
>
> On Fri, Jul 19, 2024 at 6:59 PM Vikram Koka <vi...@astronomer.io.invalid>
> wrote:
>
>> I am very relieved with the "experimental, don't try this at home" label
>> on
>> this body of work, since this will be replaced shortly.
>>
>> If there is belief that this will help accelerate learning towards AIP-69
>> (Remote Executor), I understand the rationale for proceeding forward with
>> this given the caveats above.
>>
>> Given that, I don't have strong feelings either way on this. I am ok with
>> the conclusion above of releasing it largely "as is" to be used as a test
>> bed before AIP-72 is released as part of 3.0.
>>
>>
>>
>> On Thu, Jul 18, 2024 at 11:54 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > Remote Executor not worker.
>> >
>> > On Thu, Jul 18, 2024 at 8:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > > My conclusion is (I spoke to Jens about it today) that we should
>> attempt
>> > > to get a "workable, not necessarily optimized for performance, and
>> > > potentially having a few things not working version of it" and label
>> it
>> > "
>> > > highly experimental, don't do it at home" - rather similar to what
>> Jens'
>> > > approach is for the "pre" release of remote worker. Jens already has a
>> > > working Remote vorker on it, so having the base implemented in Airflow
>> > 2.10
>> > > might be for him (and few other brave souls) who would want to test
>> > remote
>> > > worker on "2.10" version. And maybe we will also find a thing or two
>> that
>> > > might be useful for AIP-72 implementation while doing it.
>> > >
>> > > J.
>> > >
>> > > On Thu, Jul 18, 2024 at 8:44 PM Daniel Standish
>> > > <da...@astronomer.io.invalid> wrote:
>> > >
>> > >> I don't have strong feelings here either way, but we should probably
>> try
>> > >> to
>> > >> arrive at a conclusion here sooner than later because meanwhile work
>> on
>> > >> this continues.
>> > >>
>> > >> I think the question is sort of, is anyone going to essentially veto
>> it,
>> > >> or
>> > >> perhaps veto it unless certain conditions are met. E.g. it has been
>> > >> proposed that the feature must be marked as experimental so as to not
>> > >> increase support burden on what is perhaps a dead end feature.
>> > >>
>> > >> Any other sort of proposals or questions outstanding on this one?
>> > >>
>> > >>
>> > >>
>> > >> On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai <
>> amoghdesai.oss@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > I'd prefer to move it completely to Airflow 3, because I don't
>> really
>> > >> see
>> > >> > spending cycles
>> > >> > on developing something for the short term here.
>> > >> >
>> > >> > With Airflow 3, it would come out more complete and something that
>> > >> could be
>> > >> > usable by users without
>> > >> > having to put a halt to it and starting it again.
>> > >> >
>> > >> > I understand that it would affect the AIP 69 but as I whole i don't
>> > see
>> > >> > doing AIP 44 for Airflow 2 as beneficial.
>> > >> >
>> > >> >
>> > >> > Thanks & Regards,
>> > >> > Amogh Desai
>> > >> >
>> > >> >
>> > >> > On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <ash@apache.org
>> >
>> > >> wrote:
>> > >> >
>> > >> > > -0.25 to me continuing - I don't see the value long term in
>> putting
>> > >> more
>> > >> > > effort into this, and any lessons from this have already been
>> > learnt,
>> > >> as
>> > >> > > development on Airflow 3 has already started, so this would
>> > literally
>> > >> be
>> > >> > > happening in parallel.
>> > >> > >
>> > >> > > Given we are planning, at most, 1 or two more releases of
>> Airflow 2
>> > >> this
>> > >> > > feature will have to stay marked experimental so that it doesn't
>> > >> increase
>> > >> > > our support burden.
>> > >> > >
>> > >> > > My TL;Dr: I see this feature as a dead end now, but if you want
>> to
>> > >> work
>> > >> > on
>> > >> > > it by my guest, as long as it doesn't cause us more (support)
>> work
>> > in
>> > >> the
>> > >> > > long term.
>> > >> > >
>> > >> > > -ash
>> > >> > >
>> > >> > > On 12 July 2024 14:20:14 BST, "Michał Modras" <
>> > >> michalmodras@google.com
>> > >> > .INVALID>
>> > >> > > wrote:
>> > >> > > >I think we should finish the AIP within Airflow 2 - it will take
>> > time
>> > >> > > until
>> > >> > > >Airflow 3 is out, and I believe some learnings from finishing
>> and
>> > >> > running
>> > >> > > >this AIP might be useful for Airflow 3. We plan to contribute to
>> > >> > finishing
>> > >> > > >this AIP.
>> > >> > > >
>> > >> > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T)
>> > >> > > ><Je...@de.bosch.com.invalid> wrote:
>> > >> > > >
>> > >> > > >> Hi,
>> > >> > > >>
>> > >> > > >> I'd favor to make it usable - especially as we are at 80%.
>> > >> > > >>
>> > >> > > >> Main motivation is that with our environment we see stability
>> > >> problems
>> > >> > > >> with the distributed setup and using Celery, which was the
>> main
>> > >> > > motivation
>> > >> > > >> to spin the discussion about AIP-69. AIP-69 is depending on
>> the
>> > >> > feature.
>> > >> > > >> Waiting another 12-18 months to be able to host a stable
>> > >> distributed
>> > >> > > setup
>> > >> > > >> based on Airflow 3 is something hard to argue. And I can
>> confirm
>> > >> it is
>> > >> > > >> working already in my AIP-69 PoC.
>> > >> > > >>
>> > >> > > >> In this light I could offer to move it to at least the level
>> that
>> > >> it
>> > >> > can
>> > >> > > >> be used and is properly CI tested as using it for AIP-69 as
>> first
>> > >> > > consumer
>> > >> > > >> (which could reduce the scope to task execution, DAG parsing
>> and
>> > >> > > triggered
>> > >> > > >> could be taken out-of-scope for AIP-69 dependency for
>> example). I
>> > >> > could
>> > >> > > >> offer supporting to close the gaps to completion.
>> > >> > > >>
>> > >> > > >> In regards of workload the completion should be a target
>> before
>> > the
>> > >> > > >> cut-off to Airflow 3, so I would assume only "keeping the
>> lights
>> > >> on"
>> > >> > > would
>> > >> > > >> be a distraction while developing Airflow 3.
>> > >> > > >>
>> > >> > > >> Mit freundlichen Grüßen / Best regards
>> > >> > > >>
>> > >> > > >> Jens Scheffler
>> > >> > > >>
>> > >> > > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
>> > >> > > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565
>> > >> Stuttgart-Vaihingen |
>> > >> > > >> GERMANY | www.bosch.com
>> > >> > > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160
>> > >> 90417410
>> > >> > > >> <+49%20160%2090417410> | Jens.Scheffler@de.bosch.com
>> > >> > > >>
>> > >> > > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB
>> > 14000;
>> > >> > > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
>> > >> > > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer,
>> Dr.
>> > >> > Markus
>> > >> > > >> Forschner,
>> > >> > > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja
>> > Rückert
>> > >> > > >>
>> > >> > > >> -----Original Message-----
>> > >> > > >> From: Jed Cunningham <je...@apache.org>
>> > >> > > >> Sent: Thursday, July 11, 2024 10:41 PM
>> > >> > > >> To: dev@airflow.apache.org
>> > >> > > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
>> > >> > > >>
>> > >> > > >> It feels a little weird to add a new "forever" experimental
>> > >> feature in
>> > >> > > >> Airflow 2 that we already know won't be there in Airflow 3.
>> Not
>> > >> > > something
>> > >> > > >> I'd want to be really user facing at this point in time
>> either.
>> > >> > > >>
>> > >> > > >> Given the short timeline for Airflow 3, I imagine we'd be
>> better
>> > >> off
>> > >> > > >> spending those cycles elsewhere. My 2c - not my cycles :)
>> > >> > > >>
>> > >> > >
>> > >> > >
>> > ---------------------------------------------------------------------
>> > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > >> > > For additional commands, e-mail: dev-help@airflow.apache.org
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Jarek Potiuk <ja...@potiuk.com>.
Also one thing after spending some time on it - after discussion with Jens
I already made the "DB Isolation" mode works end-to-end in breeze (
https://github.com/apache/airflow/pull/40894) - which was fairly easy (and
it seems to work pretty well/stable with some limitations (no
mini-scheduler for now). Bt I am also working through making the standalone
DAGFileProcessor working (it had missing bits and pieces) and I strongly
believe that work will be extremely useful for AIP-72 as well because it
turns out that there is also quite a some "spaggetish" DB / access at
places so whatever refactoring and changes are implemented now, they should
basically be a base for AIP-72 DAGFileProcessing API. It's not "com[plex"
but a bit tedious - and it will show immediately where API calls will be
needed as well.
So that work is not **entirely* useless.
J.
On Fri, Jul 19, 2024 at 6:59 PM Vikram Koka <vi...@astronomer.io.invalid>
wrote:
> I am very relieved with the "experimental, don't try this at home" label on
> this body of work, since this will be replaced shortly.
>
> If there is belief that this will help accelerate learning towards AIP-69
> (Remote Executor), I understand the rationale for proceeding forward with
> this given the caveats above.
>
> Given that, I don't have strong feelings either way on this. I am ok with
> the conclusion above of releasing it largely "as is" to be used as a test
> bed before AIP-72 is released as part of 3.0.
>
>
>
> On Thu, Jul 18, 2024 at 11:54 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Remote Executor not worker.
> >
> > On Thu, Jul 18, 2024 at 8:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > My conclusion is (I spoke to Jens about it today) that we should
> attempt
> > > to get a "workable, not necessarily optimized for performance, and
> > > potentially having a few things not working version of it" and label it
> > "
> > > highly experimental, don't do it at home" - rather similar to what
> Jens'
> > > approach is for the "pre" release of remote worker. Jens already has a
> > > working Remote vorker on it, so having the base implemented in Airflow
> > 2.10
> > > might be for him (and few other brave souls) who would want to test
> > remote
> > > worker on "2.10" version. And maybe we will also find a thing or two
> that
> > > might be useful for AIP-72 implementation while doing it.
> > >
> > > J.
> > >
> > > On Thu, Jul 18, 2024 at 8:44 PM Daniel Standish
> > > <da...@astronomer.io.invalid> wrote:
> > >
> > >> I don't have strong feelings here either way, but we should probably
> try
> > >> to
> > >> arrive at a conclusion here sooner than later because meanwhile work
> on
> > >> this continues.
> > >>
> > >> I think the question is sort of, is anyone going to essentially veto
> it,
> > >> or
> > >> perhaps veto it unless certain conditions are met. E.g. it has been
> > >> proposed that the feature must be marked as experimental so as to not
> > >> increase support burden on what is perhaps a dead end feature.
> > >>
> > >> Any other sort of proposals or questions outstanding on this one?
> > >>
> > >>
> > >>
> > >> On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai <
> amoghdesai.oss@gmail.com>
> > >> wrote:
> > >>
> > >> > I'd prefer to move it completely to Airflow 3, because I don't
> really
> > >> see
> > >> > spending cycles
> > >> > on developing something for the short term here.
> > >> >
> > >> > With Airflow 3, it would come out more complete and something that
> > >> could be
> > >> > usable by users without
> > >> > having to put a halt to it and starting it again.
> > >> >
> > >> > I understand that it would affect the AIP 69 but as I whole i don't
> > see
> > >> > doing AIP 44 for Airflow 2 as beneficial.
> > >> >
> > >> >
> > >> > Thanks & Regards,
> > >> > Amogh Desai
> > >> >
> > >> >
> > >> > On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <as...@apache.org>
> > >> wrote:
> > >> >
> > >> > > -0.25 to me continuing - I don't see the value long term in
> putting
> > >> more
> > >> > > effort into this, and any lessons from this have already been
> > learnt,
> > >> as
> > >> > > development on Airflow 3 has already started, so this would
> > literally
> > >> be
> > >> > > happening in parallel.
> > >> > >
> > >> > > Given we are planning, at most, 1 or two more releases of Airflow
> 2
> > >> this
> > >> > > feature will have to stay marked experimental so that it doesn't
> > >> increase
> > >> > > our support burden.
> > >> > >
> > >> > > My TL;Dr: I see this feature as a dead end now, but if you want to
> > >> work
> > >> > on
> > >> > > it by my guest, as long as it doesn't cause us more (support) work
> > in
> > >> the
> > >> > > long term.
> > >> > >
> > >> > > -ash
> > >> > >
> > >> > > On 12 July 2024 14:20:14 BST, "Michał Modras" <
> > >> michalmodras@google.com
> > >> > .INVALID>
> > >> > > wrote:
> > >> > > >I think we should finish the AIP within Airflow 2 - it will take
> > time
> > >> > > until
> > >> > > >Airflow 3 is out, and I believe some learnings from finishing and
> > >> > running
> > >> > > >this AIP might be useful for Airflow 3. We plan to contribute to
> > >> > finishing
> > >> > > >this AIP.
> > >> > > >
> > >> > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> > >> > > ><Je...@de.bosch.com.invalid> wrote:
> > >> > > >
> > >> > > >> Hi,
> > >> > > >>
> > >> > > >> I'd favor to make it usable - especially as we are at 80%.
> > >> > > >>
> > >> > > >> Main motivation is that with our environment we see stability
> > >> problems
> > >> > > >> with the distributed setup and using Celery, which was the main
> > >> > > motivation
> > >> > > >> to spin the discussion about AIP-69. AIP-69 is depending on the
> > >> > feature.
> > >> > > >> Waiting another 12-18 months to be able to host a stable
> > >> distributed
> > >> > > setup
> > >> > > >> based on Airflow 3 is something hard to argue. And I can
> confirm
> > >> it is
> > >> > > >> working already in my AIP-69 PoC.
> > >> > > >>
> > >> > > >> In this light I could offer to move it to at least the level
> that
> > >> it
> > >> > can
> > >> > > >> be used and is properly CI tested as using it for AIP-69 as
> first
> > >> > > consumer
> > >> > > >> (which could reduce the scope to task execution, DAG parsing
> and
> > >> > > triggered
> > >> > > >> could be taken out-of-scope for AIP-69 dependency for
> example). I
> > >> > could
> > >> > > >> offer supporting to close the gaps to completion.
> > >> > > >>
> > >> > > >> In regards of workload the completion should be a target before
> > the
> > >> > > >> cut-off to Airflow 3, so I would assume only "keeping the
> lights
> > >> on"
> > >> > > would
> > >> > > >> be a distraction while developing Airflow 3.
> > >> > > >>
> > >> > > >> Mit freundlichen Grüßen / Best regards
> > >> > > >>
> > >> > > >> Jens Scheffler
> > >> > > >>
> > >> > > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> > >> > > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565
> > >> Stuttgart-Vaihingen |
> > >> > > >> GERMANY | www.bosch.com
> > >> > > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160
> > >> 90417410
> > >> > > >> <+49%20160%2090417410> | Jens.Scheffler@de.bosch.com
> > >> > > >>
> > >> > > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB
> > 14000;
> > >> > > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> > >> > > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer,
> Dr.
> > >> > Markus
> > >> > > >> Forschner,
> > >> > > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja
> > Rückert
> > >> > > >>
> > >> > > >> -----Original Message-----
> > >> > > >> From: Jed Cunningham <je...@apache.org>
> > >> > > >> Sent: Thursday, July 11, 2024 10:41 PM
> > >> > > >> To: dev@airflow.apache.org
> > >> > > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
> > >> > > >>
> > >> > > >> It feels a little weird to add a new "forever" experimental
> > >> feature in
> > >> > > >> Airflow 2 that we already know won't be there in Airflow 3. Not
> > >> > > something
> > >> > > >> I'd want to be really user facing at this point in time either.
> > >> > > >>
> > >> > > >> Given the short timeline for Airflow 3, I imagine we'd be
> better
> > >> off
> > >> > > >> spending those cycles elsewhere. My 2c - not my cycles :)
> > >> > > >>
> > >> > >
> > >> > >
> > ---------------------------------------------------------------------
> > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
I am very relieved with the "experimental, don't try this at home" label on
this body of work, since this will be replaced shortly.
If there is belief that this will help accelerate learning towards AIP-69
(Remote Executor), I understand the rationale for proceeding forward with
this given the caveats above.
Given that, I don't have strong feelings either way on this. I am ok with
the conclusion above of releasing it largely "as is" to be used as a test
bed before AIP-72 is released as part of 3.0.
On Thu, Jul 18, 2024 at 11:54 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Remote Executor not worker.
>
> On Thu, Jul 18, 2024 at 8:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > My conclusion is (I spoke to Jens about it today) that we should attempt
> > to get a "workable, not necessarily optimized for performance, and
> > potentially having a few things not working version of it" and label it
> "
> > highly experimental, don't do it at home" - rather similar to what Jens'
> > approach is for the "pre" release of remote worker. Jens already has a
> > working Remote vorker on it, so having the base implemented in Airflow
> 2.10
> > might be for him (and few other brave souls) who would want to test
> remote
> > worker on "2.10" version. And maybe we will also find a thing or two that
> > might be useful for AIP-72 implementation while doing it.
> >
> > J.
> >
> > On Thu, Jul 18, 2024 at 8:44 PM Daniel Standish
> > <da...@astronomer.io.invalid> wrote:
> >
> >> I don't have strong feelings here either way, but we should probably try
> >> to
> >> arrive at a conclusion here sooner than later because meanwhile work on
> >> this continues.
> >>
> >> I think the question is sort of, is anyone going to essentially veto it,
> >> or
> >> perhaps veto it unless certain conditions are met. E.g. it has been
> >> proposed that the feature must be marked as experimental so as to not
> >> increase support burden on what is perhaps a dead end feature.
> >>
> >> Any other sort of proposals or questions outstanding on this one?
> >>
> >>
> >>
> >> On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai <am...@gmail.com>
> >> wrote:
> >>
> >> > I'd prefer to move it completely to Airflow 3, because I don't really
> >> see
> >> > spending cycles
> >> > on developing something for the short term here.
> >> >
> >> > With Airflow 3, it would come out more complete and something that
> >> could be
> >> > usable by users without
> >> > having to put a halt to it and starting it again.
> >> >
> >> > I understand that it would affect the AIP 69 but as I whole i don't
> see
> >> > doing AIP 44 for Airflow 2 as beneficial.
> >> >
> >> >
> >> > Thanks & Regards,
> >> > Amogh Desai
> >> >
> >> >
> >> > On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <as...@apache.org>
> >> wrote:
> >> >
> >> > > -0.25 to me continuing - I don't see the value long term in putting
> >> more
> >> > > effort into this, and any lessons from this have already been
> learnt,
> >> as
> >> > > development on Airflow 3 has already started, so this would
> literally
> >> be
> >> > > happening in parallel.
> >> > >
> >> > > Given we are planning, at most, 1 or two more releases of Airflow 2
> >> this
> >> > > feature will have to stay marked experimental so that it doesn't
> >> increase
> >> > > our support burden.
> >> > >
> >> > > My TL;Dr: I see this feature as a dead end now, but if you want to
> >> work
> >> > on
> >> > > it by my guest, as long as it doesn't cause us more (support) work
> in
> >> the
> >> > > long term.
> >> > >
> >> > > -ash
> >> > >
> >> > > On 12 July 2024 14:20:14 BST, "Michał Modras" <
> >> michalmodras@google.com
> >> > .INVALID>
> >> > > wrote:
> >> > > >I think we should finish the AIP within Airflow 2 - it will take
> time
> >> > > until
> >> > > >Airflow 3 is out, and I believe some learnings from finishing and
> >> > running
> >> > > >this AIP might be useful for Airflow 3. We plan to contribute to
> >> > finishing
> >> > > >this AIP.
> >> > > >
> >> > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> >> > > ><Je...@de.bosch.com.invalid> wrote:
> >> > > >
> >> > > >> Hi,
> >> > > >>
> >> > > >> I'd favor to make it usable - especially as we are at 80%.
> >> > > >>
> >> > > >> Main motivation is that with our environment we see stability
> >> problems
> >> > > >> with the distributed setup and using Celery, which was the main
> >> > > motivation
> >> > > >> to spin the discussion about AIP-69. AIP-69 is depending on the
> >> > feature.
> >> > > >> Waiting another 12-18 months to be able to host a stable
> >> distributed
> >> > > setup
> >> > > >> based on Airflow 3 is something hard to argue. And I can confirm
> >> it is
> >> > > >> working already in my AIP-69 PoC.
> >> > > >>
> >> > > >> In this light I could offer to move it to at least the level that
> >> it
> >> > can
> >> > > >> be used and is properly CI tested as using it for AIP-69 as first
> >> > > consumer
> >> > > >> (which could reduce the scope to task execution, DAG parsing and
> >> > > triggered
> >> > > >> could be taken out-of-scope for AIP-69 dependency for example). I
> >> > could
> >> > > >> offer supporting to close the gaps to completion.
> >> > > >>
> >> > > >> In regards of workload the completion should be a target before
> the
> >> > > >> cut-off to Airflow 3, so I would assume only "keeping the lights
> >> on"
> >> > > would
> >> > > >> be a distraction while developing Airflow 3.
> >> > > >>
> >> > > >> Mit freundlichen Grüßen / Best regards
> >> > > >>
> >> > > >> Jens Scheffler
> >> > > >>
> >> > > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> >> > > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565
> >> Stuttgart-Vaihingen |
> >> > > >> GERMANY | www.bosch.com
> >> > > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160
> >> 90417410
> >> > > >> <+49%20160%2090417410> | Jens.Scheffler@de.bosch.com
> >> > > >>
> >> > > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB
> 14000;
> >> > > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> >> > > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr.
> >> > Markus
> >> > > >> Forschner,
> >> > > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja
> Rückert
> >> > > >>
> >> > > >> -----Original Message-----
> >> > > >> From: Jed Cunningham <je...@apache.org>
> >> > > >> Sent: Thursday, July 11, 2024 10:41 PM
> >> > > >> To: dev@airflow.apache.org
> >> > > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
> >> > > >>
> >> > > >> It feels a little weird to add a new "forever" experimental
> >> feature in
> >> > > >> Airflow 2 that we already know won't be there in Airflow 3. Not
> >> > > something
> >> > > >> I'd want to be really user facing at this point in time either.
> >> > > >>
> >> > > >> Given the short timeline for Airflow 3, I imagine we'd be better
> >> off
> >> > > >> spending those cycles elsewhere. My 2c - not my cycles :)
> >> > > >>
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> >> > >
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Jarek Potiuk <ja...@potiuk.com>.
Remote Executor not worker.
On Thu, Jul 18, 2024 at 8:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> My conclusion is (I spoke to Jens about it today) that we should attempt
> to get a "workable, not necessarily optimized for performance, and
> potentially having a few things not working version of it" and label it "
> highly experimental, don't do it at home" - rather similar to what Jens'
> approach is for the "pre" release of remote worker. Jens already has a
> working Remote vorker on it, so having the base implemented in Airflow 2.10
> might be for him (and few other brave souls) who would want to test remote
> worker on "2.10" version. And maybe we will also find a thing or two that
> might be useful for AIP-72 implementation while doing it.
>
> J.
>
> On Thu, Jul 18, 2024 at 8:44 PM Daniel Standish
> <da...@astronomer.io.invalid> wrote:
>
>> I don't have strong feelings here either way, but we should probably try
>> to
>> arrive at a conclusion here sooner than later because meanwhile work on
>> this continues.
>>
>> I think the question is sort of, is anyone going to essentially veto it,
>> or
>> perhaps veto it unless certain conditions are met. E.g. it has been
>> proposed that the feature must be marked as experimental so as to not
>> increase support burden on what is perhaps a dead end feature.
>>
>> Any other sort of proposals or questions outstanding on this one?
>>
>>
>>
>> On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai <am...@gmail.com>
>> wrote:
>>
>> > I'd prefer to move it completely to Airflow 3, because I don't really
>> see
>> > spending cycles
>> > on developing something for the short term here.
>> >
>> > With Airflow 3, it would come out more complete and something that
>> could be
>> > usable by users without
>> > having to put a halt to it and starting it again.
>> >
>> > I understand that it would affect the AIP 69 but as I whole i don't see
>> > doing AIP 44 for Airflow 2 as beneficial.
>> >
>> >
>> > Thanks & Regards,
>> > Amogh Desai
>> >
>> >
>> > On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <as...@apache.org>
>> wrote:
>> >
>> > > -0.25 to me continuing - I don't see the value long term in putting
>> more
>> > > effort into this, and any lessons from this have already been learnt,
>> as
>> > > development on Airflow 3 has already started, so this would literally
>> be
>> > > happening in parallel.
>> > >
>> > > Given we are planning, at most, 1 or two more releases of Airflow 2
>> this
>> > > feature will have to stay marked experimental so that it doesn't
>> increase
>> > > our support burden.
>> > >
>> > > My TL;Dr: I see this feature as a dead end now, but if you want to
>> work
>> > on
>> > > it by my guest, as long as it doesn't cause us more (support) work in
>> the
>> > > long term.
>> > >
>> > > -ash
>> > >
>> > > On 12 July 2024 14:20:14 BST, "Michał Modras" <
>> michalmodras@google.com
>> > .INVALID>
>> > > wrote:
>> > > >I think we should finish the AIP within Airflow 2 - it will take time
>> > > until
>> > > >Airflow 3 is out, and I believe some learnings from finishing and
>> > running
>> > > >this AIP might be useful for Airflow 3. We plan to contribute to
>> > finishing
>> > > >this AIP.
>> > > >
>> > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T)
>> > > ><Je...@de.bosch.com.invalid> wrote:
>> > > >
>> > > >> Hi,
>> > > >>
>> > > >> I'd favor to make it usable - especially as we are at 80%.
>> > > >>
>> > > >> Main motivation is that with our environment we see stability
>> problems
>> > > >> with the distributed setup and using Celery, which was the main
>> > > motivation
>> > > >> to spin the discussion about AIP-69. AIP-69 is depending on the
>> > feature.
>> > > >> Waiting another 12-18 months to be able to host a stable
>> distributed
>> > > setup
>> > > >> based on Airflow 3 is something hard to argue. And I can confirm
>> it is
>> > > >> working already in my AIP-69 PoC.
>> > > >>
>> > > >> In this light I could offer to move it to at least the level that
>> it
>> > can
>> > > >> be used and is properly CI tested as using it for AIP-69 as first
>> > > consumer
>> > > >> (which could reduce the scope to task execution, DAG parsing and
>> > > triggered
>> > > >> could be taken out-of-scope for AIP-69 dependency for example). I
>> > could
>> > > >> offer supporting to close the gaps to completion.
>> > > >>
>> > > >> In regards of workload the completion should be a target before the
>> > > >> cut-off to Airflow 3, so I would assume only "keeping the lights
>> on"
>> > > would
>> > > >> be a distraction while developing Airflow 3.
>> > > >>
>> > > >> Mit freundlichen Grüßen / Best regards
>> > > >>
>> > > >> Jens Scheffler
>> > > >>
>> > > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
>> > > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565
>> Stuttgart-Vaihingen |
>> > > >> GERMANY | www.bosch.com
>> > > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160
>> 90417410
>> > > >> <+49%20160%2090417410> | Jens.Scheffler@de.bosch.com
>> > > >>
>> > > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
>> > > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
>> > > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr.
>> > Markus
>> > > >> Forschner,
>> > > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
>> > > >>
>> > > >> -----Original Message-----
>> > > >> From: Jed Cunningham <je...@apache.org>
>> > > >> Sent: Thursday, July 11, 2024 10:41 PM
>> > > >> To: dev@airflow.apache.org
>> > > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
>> > > >>
>> > > >> It feels a little weird to add a new "forever" experimental
>> feature in
>> > > >> Airflow 2 that we already know won't be there in Airflow 3. Not
>> > > something
>> > > >> I'd want to be really user facing at this point in time either.
>> > > >>
>> > > >> Given the short timeline for Airflow 3, I imagine we'd be better
>> off
>> > > >> spending those cycles elsewhere. My 2c - not my cycles :)
>> > > >>
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>> > >
>> > >
>> >
>>
>
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Jarek Potiuk <ja...@potiuk.com>.
My conclusion is (I spoke to Jens about it today) that we should attempt to
get a "workable, not necessarily optimized for performance, and potentially
having a few things not working version of it" and label it " highly
experimental, don't do it at home" - rather similar to what Jens' approach
is for the "pre" release of remote worker. Jens already has a working
Remote vorker on it, so having the base implemented in Airflow 2.10 might
be for him (and few other brave souls) who would want to test remote worker
on "2.10" version. And maybe we will also find a thing or two that might be
useful for AIP-72 implementation while doing it.
J.
On Thu, Jul 18, 2024 at 8:44 PM Daniel Standish
<da...@astronomer.io.invalid> wrote:
> I don't have strong feelings here either way, but we should probably try to
> arrive at a conclusion here sooner than later because meanwhile work on
> this continues.
>
> I think the question is sort of, is anyone going to essentially veto it, or
> perhaps veto it unless certain conditions are met. E.g. it has been
> proposed that the feature must be marked as experimental so as to not
> increase support burden on what is perhaps a dead end feature.
>
> Any other sort of proposals or questions outstanding on this one?
>
>
>
> On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai <am...@gmail.com>
> wrote:
>
> > I'd prefer to move it completely to Airflow 3, because I don't really see
> > spending cycles
> > on developing something for the short term here.
> >
> > With Airflow 3, it would come out more complete and something that could
> be
> > usable by users without
> > having to put a halt to it and starting it again.
> >
> > I understand that it would affect the AIP 69 but as I whole i don't see
> > doing AIP 44 for Airflow 2 as beneficial.
> >
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > -0.25 to me continuing - I don't see the value long term in putting
> more
> > > effort into this, and any lessons from this have already been learnt,
> as
> > > development on Airflow 3 has already started, so this would literally
> be
> > > happening in parallel.
> > >
> > > Given we are planning, at most, 1 or two more releases of Airflow 2
> this
> > > feature will have to stay marked experimental so that it doesn't
> increase
> > > our support burden.
> > >
> > > My TL;Dr: I see this feature as a dead end now, but if you want to work
> > on
> > > it by my guest, as long as it doesn't cause us more (support) work in
> the
> > > long term.
> > >
> > > -ash
> > >
> > > On 12 July 2024 14:20:14 BST, "Michał Modras" <michalmodras@google.com
> > .INVALID>
> > > wrote:
> > > >I think we should finish the AIP within Airflow 2 - it will take time
> > > until
> > > >Airflow 3 is out, and I believe some learnings from finishing and
> > running
> > > >this AIP might be useful for Airflow 3. We plan to contribute to
> > finishing
> > > >this AIP.
> > > >
> > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> > > ><Je...@de.bosch.com.invalid> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I'd favor to make it usable - especially as we are at 80%.
> > > >>
> > > >> Main motivation is that with our environment we see stability
> problems
> > > >> with the distributed setup and using Celery, which was the main
> > > motivation
> > > >> to spin the discussion about AIP-69. AIP-69 is depending on the
> > feature.
> > > >> Waiting another 12-18 months to be able to host a stable distributed
> > > setup
> > > >> based on Airflow 3 is something hard to argue. And I can confirm it
> is
> > > >> working already in my AIP-69 PoC.
> > > >>
> > > >> In this light I could offer to move it to at least the level that it
> > can
> > > >> be used and is properly CI tested as using it for AIP-69 as first
> > > consumer
> > > >> (which could reduce the scope to task execution, DAG parsing and
> > > triggered
> > > >> could be taken out-of-scope for AIP-69 dependency for example). I
> > could
> > > >> offer supporting to close the gaps to completion.
> > > >>
> > > >> In regards of workload the completion should be a target before the
> > > >> cut-off to Airflow 3, so I would assume only "keeping the lights on"
> > > would
> > > >> be a distraction while developing Airflow 3.
> > > >>
> > > >> Mit freundlichen Grüßen / Best regards
> > > >>
> > > >> Jens Scheffler
> > > >>
> > > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> > > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen
> |
> > > >> GERMANY | www.bosch.com
> > > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160
> 90417410
> > > >> <+49%20160%2090417410> | Jens.Scheffler@de.bosch.com
> > > >>
> > > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> > > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> > > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr.
> > Markus
> > > >> Forschner,
> > > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
> > > >>
> > > >> -----Original Message-----
> > > >> From: Jed Cunningham <je...@apache.org>
> > > >> Sent: Thursday, July 11, 2024 10:41 PM
> > > >> To: dev@airflow.apache.org
> > > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
> > > >>
> > > >> It feels a little weird to add a new "forever" experimental feature
> in
> > > >> Airflow 2 that we already know won't be there in Airflow 3. Not
> > > something
> > > >> I'd want to be really user facing at this point in time either.
> > > >>
> > > >> Given the short timeline for Airflow 3, I imagine we'd be better off
> > > >> spending those cycles elsewhere. My 2c - not my cycles :)
> > > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
I don't have strong feelings here either way, but we should probably try to
arrive at a conclusion here sooner than later because meanwhile work on
this continues.
I think the question is sort of, is anyone going to essentially veto it, or
perhaps veto it unless certain conditions are met. E.g. it has been
proposed that the feature must be marked as experimental so as to not
increase support burden on what is perhaps a dead end feature.
Any other sort of proposals or questions outstanding on this one?
On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai <am...@gmail.com>
wrote:
> I'd prefer to move it completely to Airflow 3, because I don't really see
> spending cycles
> on developing something for the short term here.
>
> With Airflow 3, it would come out more complete and something that could be
> usable by users without
> having to put a halt to it and starting it again.
>
> I understand that it would affect the AIP 69 but as I whole i don't see
> doing AIP 44 for Airflow 2 as beneficial.
>
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > -0.25 to me continuing - I don't see the value long term in putting more
> > effort into this, and any lessons from this have already been learnt, as
> > development on Airflow 3 has already started, so this would literally be
> > happening in parallel.
> >
> > Given we are planning, at most, 1 or two more releases of Airflow 2 this
> > feature will have to stay marked experimental so that it doesn't increase
> > our support burden.
> >
> > My TL;Dr: I see this feature as a dead end now, but if you want to work
> on
> > it by my guest, as long as it doesn't cause us more (support) work in the
> > long term.
> >
> > -ash
> >
> > On 12 July 2024 14:20:14 BST, "Michał Modras" <michalmodras@google.com
> .INVALID>
> > wrote:
> > >I think we should finish the AIP within Airflow 2 - it will take time
> > until
> > >Airflow 3 is out, and I believe some learnings from finishing and
> running
> > >this AIP might be useful for Airflow 3. We plan to contribute to
> finishing
> > >this AIP.
> > >
> > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> > ><Je...@de.bosch.com.invalid> wrote:
> > >
> > >> Hi,
> > >>
> > >> I'd favor to make it usable - especially as we are at 80%.
> > >>
> > >> Main motivation is that with our environment we see stability problems
> > >> with the distributed setup and using Celery, which was the main
> > motivation
> > >> to spin the discussion about AIP-69. AIP-69 is depending on the
> feature.
> > >> Waiting another 12-18 months to be able to host a stable distributed
> > setup
> > >> based on Airflow 3 is something hard to argue. And I can confirm it is
> > >> working already in my AIP-69 PoC.
> > >>
> > >> In this light I could offer to move it to at least the level that it
> can
> > >> be used and is properly CI tested as using it for AIP-69 as first
> > consumer
> > >> (which could reduce the scope to task execution, DAG parsing and
> > triggered
> > >> could be taken out-of-scope for AIP-69 dependency for example). I
> could
> > >> offer supporting to close the gaps to completion.
> > >>
> > >> In regards of workload the completion should be a target before the
> > >> cut-off to Airflow 3, so I would assume only "keeping the lights on"
> > would
> > >> be a distraction while developing Airflow 3.
> > >>
> > >> Mit freundlichen Grüßen / Best regards
> > >>
> > >> Jens Scheffler
> > >>
> > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
> > >> GERMANY | www.bosch.com
> > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 90417410
> > >> <+49%20160%2090417410> | Jens.Scheffler@de.bosch.com
> > >>
> > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr.
> Markus
> > >> Forschner,
> > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
> > >>
> > >> -----Original Message-----
> > >> From: Jed Cunningham <je...@apache.org>
> > >> Sent: Thursday, July 11, 2024 10:41 PM
> > >> To: dev@airflow.apache.org
> > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
> > >>
> > >> It feels a little weird to add a new "forever" experimental feature in
> > >> Airflow 2 that we already know won't be there in Airflow 3. Not
> > something
> > >> I'd want to be really user facing at this point in time either.
> > >>
> > >> Given the short timeline for Airflow 3, I imagine we'd be better off
> > >> spending those cycles elsewhere. My 2c - not my cycles :)
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Amogh Desai <am...@gmail.com>.
I'd prefer to move it completely to Airflow 3, because I don't really see
spending cycles
on developing something for the short term here.
With Airflow 3, it would come out more complete and something that could be
usable by users without
having to put a halt to it and starting it again.
I understand that it would affect the AIP 69 but as I whole i don't see
doing AIP 44 for Airflow 2 as beneficial.
Thanks & Regards,
Amogh Desai
On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> -0.25 to me continuing - I don't see the value long term in putting more
> effort into this, and any lessons from this have already been learnt, as
> development on Airflow 3 has already started, so this would literally be
> happening in parallel.
>
> Given we are planning, at most, 1 or two more releases of Airflow 2 this
> feature will have to stay marked experimental so that it doesn't increase
> our support burden.
>
> My TL;Dr: I see this feature as a dead end now, but if you want to work on
> it by my guest, as long as it doesn't cause us more (support) work in the
> long term.
>
> -ash
>
> On 12 July 2024 14:20:14 BST, "Michał Modras" <mi...@google.com.INVALID>
> wrote:
> >I think we should finish the AIP within Airflow 2 - it will take time
> until
> >Airflow 3 is out, and I believe some learnings from finishing and running
> >this AIP might be useful for Airflow 3. We plan to contribute to finishing
> >this AIP.
> >
> >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> ><Je...@de.bosch.com.invalid> wrote:
> >
> >> Hi,
> >>
> >> I'd favor to make it usable - especially as we are at 80%.
> >>
> >> Main motivation is that with our environment we see stability problems
> >> with the distributed setup and using Celery, which was the main
> motivation
> >> to spin the discussion about AIP-69. AIP-69 is depending on the feature.
> >> Waiting another 12-18 months to be able to host a stable distributed
> setup
> >> based on Airflow 3 is something hard to argue. And I can confirm it is
> >> working already in my AIP-69 PoC.
> >>
> >> In this light I could offer to move it to at least the level that it can
> >> be used and is properly CI tested as using it for AIP-69 as first
> consumer
> >> (which could reduce the scope to task execution, DAG parsing and
> triggered
> >> could be taken out-of-scope for AIP-69 dependency for example). I could
> >> offer supporting to close the gaps to completion.
> >>
> >> In regards of workload the completion should be a target before the
> >> cut-off to Airflow 3, so I would assume only "keeping the lights on"
> would
> >> be a distraction while developing Airflow 3.
> >>
> >> Mit freundlichen Grüßen / Best regards
> >>
> >> Jens Scheffler
> >>
> >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
> >> GERMANY | www.bosch.com
> >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 90417410
> >> <+49%20160%2090417410> | Jens.Scheffler@de.bosch.com
> >>
> >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> >> Forschner,
> >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
> >>
> >> -----Original Message-----
> >> From: Jed Cunningham <je...@apache.org>
> >> Sent: Thursday, July 11, 2024 10:41 PM
> >> To: dev@airflow.apache.org
> >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
> >>
> >> It feels a little weird to add a new "forever" experimental feature in
> >> Airflow 2 that we already know won't be there in Airflow 3. Not
> something
> >> I'd want to be really user facing at this point in time either.
> >>
> >> Given the short timeline for Airflow 3, I imagine we'd be better off
> >> spending those cycles elsewhere. My 2c - not my cycles :)
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Ash Berlin-Taylor <as...@apache.org>.
-0.25 to me continuing - I don't see the value long term in putting more effort into this, and any lessons from this have already been learnt, as development on Airflow 3 has already started, so this would literally be happening in parallel.
Given we are planning, at most, 1 or two more releases of Airflow 2 this feature will have to stay marked experimental so that it doesn't increase our support burden.
My TL;Dr: I see this feature as a dead end now, but if you want to work on it by my guest, as long as it doesn't cause us more (support) work in the long term.
-ash
On 12 July 2024 14:20:14 BST, "Michał Modras" <mi...@google.com.INVALID> wrote:
>I think we should finish the AIP within Airflow 2 - it will take time until
>Airflow 3 is out, and I believe some learnings from finishing and running
>this AIP might be useful for Airflow 3. We plan to contribute to finishing
>this AIP.
>
>On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T)
><Je...@de.bosch.com.invalid> wrote:
>
>> Hi,
>>
>> I'd favor to make it usable - especially as we are at 80%.
>>
>> Main motivation is that with our environment we see stability problems
>> with the distributed setup and using Celery, which was the main motivation
>> to spin the discussion about AIP-69. AIP-69 is depending on the feature.
>> Waiting another 12-18 months to be able to host a stable distributed setup
>> based on Airflow 3 is something hard to argue. And I can confirm it is
>> working already in my AIP-69 PoC.
>>
>> In this light I could offer to move it to at least the level that it can
>> be used and is properly CI tested as using it for AIP-69 as first consumer
>> (which could reduce the scope to task execution, DAG parsing and triggered
>> could be taken out-of-scope for AIP-69 dependency for example). I could
>> offer supporting to close the gaps to completion.
>>
>> In regards of workload the completion should be a target before the
>> cut-off to Airflow 3, so I would assume only "keeping the lights on" would
>> be a distraction while developing Airflow 3.
>>
>> Mit freundlichen Grüßen / Best regards
>>
>> Jens Scheffler
>>
>> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
>> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
>> GERMANY | www.bosch.com
>> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 90417410
>> <+49%20160%2090417410> | Jens.Scheffler@de.bosch.com
>>
>> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
>> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
>> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
>> Forschner,
>> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
>>
>> -----Original Message-----
>> From: Jed Cunningham <je...@apache.org>
>> Sent: Thursday, July 11, 2024 10:41 PM
>> To: dev@airflow.apache.org
>> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
>>
>> It feels a little weird to add a new "forever" experimental feature in
>> Airflow 2 that we already know won't be there in Airflow 3. Not something
>> I'd want to be really user facing at this point in time either.
>>
>> Given the short timeline for Airflow 3, I imagine we'd be better off
>> spending those cycles elsewhere. My 2c - not my cycles :)
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Michał Modras <mi...@google.com.INVALID>.
I think we should finish the AIP within Airflow 2 - it will take time until
Airflow 3 is out, and I believe some learnings from finishing and running
this AIP might be useful for Airflow 3. We plan to contribute to finishing
this AIP.
On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T)
<Je...@de.bosch.com.invalid> wrote:
> Hi,
>
> I'd favor to make it usable - especially as we are at 80%.
>
> Main motivation is that with our environment we see stability problems
> with the distributed setup and using Celery, which was the main motivation
> to spin the discussion about AIP-69. AIP-69 is depending on the feature.
> Waiting another 12-18 months to be able to host a stable distributed setup
> based on Airflow 3 is something hard to argue. And I can confirm it is
> working already in my AIP-69 PoC.
>
> In this light I could offer to move it to at least the level that it can
> be used and is properly CI tested as using it for AIP-69 as first consumer
> (which could reduce the scope to task execution, DAG parsing and triggered
> could be taken out-of-scope for AIP-69 dependency for example). I could
> offer supporting to close the gaps to completion.
>
> In regards of workload the completion should be a target before the
> cut-off to Airflow 3, so I would assume only "keeping the lights on" would
> be a distraction while developing Airflow 3.
>
> Mit freundlichen Grüßen / Best regards
>
> Jens Scheffler
>
> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
> GERMANY | www.bosch.com
> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 90417410
> <+49%20160%2090417410> | Jens.Scheffler@de.bosch.com
>
> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> Forschner,
> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
>
> -----Original Message-----
> From: Jed Cunningham <je...@apache.org>
> Sent: Thursday, July 11, 2024 10:41 PM
> To: dev@airflow.apache.org
> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
>
> It feels a little weird to add a new "forever" experimental feature in
> Airflow 2 that we already know won't be there in Airflow 3. Not something
> I'd want to be really user facing at this point in time either.
>
> Given the short timeline for Airflow 3, I imagine we'd be better off
> spending those cycles elsewhere. My 2c - not my cycles :)
>
RE: [DISCUSS] To AIP-44 or not to AIP-44
Posted by "Scheffler Jens (XC-AS/EAE-ADA-T)" <Je...@de.bosch.com.INVALID>.
Hi,
I'd favor to make it usable - especially as we are at 80%.
Main motivation is that with our environment we see stability problems with the distributed setup and using Celery, which was the main motivation to spin the discussion about AIP-69. AIP-69 is depending on the feature. Waiting another 12-18 months to be able to host a stable distributed setup based on Airflow 3 is something hard to argue. And I can confirm it is working already in my AIP-69 PoC.
In this light I could offer to move it to at least the level that it can be used and is properly CI tested as using it for AIP-69 as first consumer (which could reduce the scope to task execution, DAG parsing and triggered could be taken out-of-scope for AIP-69 dependency for example). I could offer supporting to close the gaps to completion.
In regards of workload the completion should be a target before the cut-off to Airflow 3, so I would assume only "keeping the lights on" would be a distraction while developing Airflow 3.
Mit freundlichen Grüßen / Best regards
Jens Scheffler
Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY | www.bosch.com
Tel. +49 711 811-91508 | Mobil +49 160 90417410 | Jens.Scheffler@de.bosch.com
Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus Forschner,
Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
-----Original Message-----
From: Jed Cunningham <je...@apache.org>
Sent: Thursday, July 11, 2024 10:41 PM
To: dev@airflow.apache.org
Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44
It feels a little weird to add a new "forever" experimental feature in Airflow 2 that we already know won't be there in Airflow 3. Not something I'd want to be really user facing at this point in time either.
Given the short timeline for Airflow 3, I imagine we'd be better off spending those cycles elsewhere. My 2c - not my cycles :)
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Jed Cunningham <je...@apache.org>.
It feels a little weird to add a new "forever" experimental feature in
Airflow 2 that we already know won't be there in Airflow 3. Not something
I'd want to be really user facing at this point in time either.
Given the short timeline for Airflow 3, I imagine we'd be better off
spending those cycles elsewhere. My 2c - not my cycles :)
Re: [DISCUSS] To AIP-44 or not to AIP-44
Posted by Kaxil Naik <ka...@gmail.com>.
I would favor moving towards Airflow 3 and delivering a fully-baked feature.
On Thu, 11 Jul 2024 at 18:28, Jarek Potiuk <ja...@potiuk.com> wrote:
> Following up after today's Airflow 3 call, I wanted to start
> discussion on whether we should continue implementing AIP-44 and
> release it in Airflow 2 or whether we should focus on AIP-72 and
> Airflow 3.
>
> We could go both ways.
>
> The implementation is somewhat 80% done - what we really miss there is
> automation of testing it in CI to make sure that all DB access is
> going through RPC (and complete remaining things that aren't). Daniel
> particularly put a lot of effort into implementing some of the
> remaining pieces and manually got the "task" working. also Jens had
> done some manual testing with "remote executor" (AIP-69) and got it to
> "work". But thorough testing - especially "forward-looking-testing" is
> needed. Also wiring the authentication is needed (following the
> webserver/log authentication we already have).
>
> On one hand - I know Google Team is particularly interested in having
> it completed, also Jens would like to use it to run the "AIP-69"
> prototype in Airflow 2. if we don't implement AIP-44, AIP-69 will only
> land in Airflow 3.
>
> On the other hand, it will drag out some efforts from Airflow 3 and
> will keep people out of implementing some of the changes we need. Also
> if we implement and release it now - we will have to support and solve
> issues Also in Airflow 2. Users who will deploy it, will have to redo
> their deployment for Airflow 3. And without multi-team AIP-67 it will
> only partially separate out the DB access (there will be no team
> isolation, DAG authors will still be able to access "all other" DAGs,
> tasks, connections, variables etc., but they won't be able to run
> arbitrary DB operations).
>
> We could go both ways, and yes - there is a lot of "Sunken cost".
>
> I'd love to hear what others think about it.
>
> J.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>