You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Mateusz Henc <mh...@google.com.INVALID> on 2021/12/02 13:20:15 UTC
[Discuss] AIP-43 DAG Processor separation
Hi,
I just submitted my proposal for separating the DagProcessor from Scheduler
and running it as an independent component. This is part of AIP-1 Airflow
multi-tenancy effort.
PTAL
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
Best regards,
Mateusz Henc
Re: [Discuss] AIP-43 DAG Processor separation
Posted by Mateusz Henc <mh...@google.com.INVALID>.
Hi,
If you have any final comments or concerns please let me know.
Otherwise I will start voting on Friday tomorrow.
Best regards,
Mateusz Henc
On Mon, Jan 3, 2022 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Happy New Year everyone:
>
> Just a short reminder we have the AIP-43 up for discussion - Mateusz
> is back in a few days, but I am happy to discuss details there:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
>
> J.
>
> On Sat, Dec 18, 2021 at 6:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Just as a comment - I think a number of people expressed that they
> > want to comment on AIP-43 / AIP-44. Mateusz mentioned that also maybe
> > some people will have more time during the slower period around Xmas -
> > so feel free to make your comments still - we aim - if possible - to
> > have the discussions conclude ~ mid Jan and that should give enough
> > time for different contributors. Do not wait till the last moment
> > though and feel free to comment on both AIPs as soon as you can. I
> > will be around also to answer those.
> >
> > On Wed, Dec 15, 2021 at 9:47 PM Mateusz Henc <mh...@google.com.invalid>
> wrote:
> > >
> > > Hi,
> > > FYI I start my vacation on Monday and I am off for 3 weeks (back on
> 01/10) with limited access to my laptop.
> > >
> > > If you have any comments please add by the end of this week, so I
> could address them while I am still there.
> > > If you are fine then maybe we could start voting soon?
> > >
> > > // Leaving more time for AIP-44 as it is longer and more complicated.
> > >
> > > Best regards,
> > > Mateusz Henc
> > >
> > >
> > > On Tue, Dec 14, 2021 at 1:09 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >>
> > >> Comment on that one too - we also provided more details (including
> > >> possible deployment options). Similarly as in case of AIP-44 we had a
> > >> few goals - which also resulted from earlier discussions:
> > >>
> > >> * Keep current behavior practically untouched
> > >> * Allow for different deployments "no separation", "dag processor
> > >> separation", "multiple dag processors for tenant isolation"
> > >> * the "multiple dag processor" case does NOT change the ways dags are
> > >> stored or parsed or where dags are stored. They are all parts of
> > >> "DAG_FOLDER". The change is merely there to allow each dag processor
> > >> to be "responsible" for parsing only selected part of the "dag" tree.
> > >> This is purely to isolate code execution between tenants and has no
> > >> impact whatsoever on other components (and it has no impact on other
> > >> ideas there - such as DAG fetcher or DAG submission). It's purely to
> > >> allow the "deployment" to better isolate code execution of user code
> > >> coming from different tenants.
> > >>
> > >> I hope this one will be faster to discuss and raise less
> > >> controversies/doubts - but feel absolutely free to comment/criticize
> > >> etc.
> > >>
> > >> J.
> > >>
> > >> On Thu, Dec 9, 2021 at 4:36 PM Mateusz Henc <mh...@google.com.invalid>
> wrote:
> > >> >
> > >> > Hi,
> > >> > We just updated the AIP-43 adding some details and deployment
> options.
> > >> > Please take a look. Any comments are appreciated.
> > >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
> > >> >
> > >> > BTW Update for AIP-44 is coming soon.
> > >> >
> > >> > Best regards,
> > >> > Mateusz Henc
> > >> >
> > >> >
> > >> > On Thu, Dec 2, 2021 at 3:04 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >> >>
> > >> >> Excited :)
> > >> >>
> > >> >> On Thu, Dec 2, 2021 at 2:20 PM Mateusz Henc
> <mh...@google.com.invalid> wrote:
> > >> >> >
> > >> >> > Hi,
> > >> >> > I just submitted my proposal for separating the DagProcessor
> from Scheduler and running it as an independent component. This is part of
> AIP-1 Airflow multi-tenancy effort.
> > >> >> >
> > >> >> > PTAL
> > >> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
> > >> >> >
> > >> >> > Best regards,
> > >> >> > Mateusz Henc
>
Re: [Discuss] AIP-43 DAG Processor separation
Posted by Jarek Potiuk <ja...@potiuk.com>.
Happy New Year everyone:
Just a short reminder we have the AIP-43 up for discussion - Mateusz
is back in a few days, but I am happy to discuss details there:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
J.
On Sat, Dec 18, 2021 at 6:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Just as a comment - I think a number of people expressed that they
> want to comment on AIP-43 / AIP-44. Mateusz mentioned that also maybe
> some people will have more time during the slower period around Xmas -
> so feel free to make your comments still - we aim - if possible - to
> have the discussions conclude ~ mid Jan and that should give enough
> time for different contributors. Do not wait till the last moment
> though and feel free to comment on both AIPs as soon as you can. I
> will be around also to answer those.
>
> On Wed, Dec 15, 2021 at 9:47 PM Mateusz Henc <mh...@google.com.invalid> wrote:
> >
> > Hi,
> > FYI I start my vacation on Monday and I am off for 3 weeks (back on 01/10) with limited access to my laptop.
> >
> > If you have any comments please add by the end of this week, so I could address them while I am still there.
> > If you are fine then maybe we could start voting soon?
> >
> > // Leaving more time for AIP-44 as it is longer and more complicated.
> >
> > Best regards,
> > Mateusz Henc
> >
> >
> > On Tue, Dec 14, 2021 at 1:09 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> Comment on that one too - we also provided more details (including
> >> possible deployment options). Similarly as in case of AIP-44 we had a
> >> few goals - which also resulted from earlier discussions:
> >>
> >> * Keep current behavior practically untouched
> >> * Allow for different deployments "no separation", "dag processor
> >> separation", "multiple dag processors for tenant isolation"
> >> * the "multiple dag processor" case does NOT change the ways dags are
> >> stored or parsed or where dags are stored. They are all parts of
> >> "DAG_FOLDER". The change is merely there to allow each dag processor
> >> to be "responsible" for parsing only selected part of the "dag" tree.
> >> This is purely to isolate code execution between tenants and has no
> >> impact whatsoever on other components (and it has no impact on other
> >> ideas there - such as DAG fetcher or DAG submission). It's purely to
> >> allow the "deployment" to better isolate code execution of user code
> >> coming from different tenants.
> >>
> >> I hope this one will be faster to discuss and raise less
> >> controversies/doubts - but feel absolutely free to comment/criticize
> >> etc.
> >>
> >> J.
> >>
> >> On Thu, Dec 9, 2021 at 4:36 PM Mateusz Henc <mh...@google.com.invalid> wrote:
> >> >
> >> > Hi,
> >> > We just updated the AIP-43 adding some details and deployment options.
> >> > Please take a look. Any comments are appreciated.
> >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
> >> >
> >> > BTW Update for AIP-44 is coming soon.
> >> >
> >> > Best regards,
> >> > Mateusz Henc
> >> >
> >> >
> >> > On Thu, Dec 2, 2021 at 3:04 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >> >>
> >> >> Excited :)
> >> >>
> >> >> On Thu, Dec 2, 2021 at 2:20 PM Mateusz Henc <mh...@google.com.invalid> wrote:
> >> >> >
> >> >> > Hi,
> >> >> > I just submitted my proposal for separating the DagProcessor from Scheduler and running it as an independent component. This is part of AIP-1 Airflow multi-tenancy effort.
> >> >> >
> >> >> > PTAL
> >> >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
> >> >> >
> >> >> > Best regards,
> >> >> > Mateusz Henc
Re: [Discuss] AIP-43 DAG Processor separation
Posted by Jarek Potiuk <ja...@potiuk.com>.
Just as a comment - I think a number of people expressed that they
want to comment on AIP-43 / AIP-44. Mateusz mentioned that also maybe
some people will have more time during the slower period around Xmas -
so feel free to make your comments still - we aim - if possible - to
have the discussions conclude ~ mid Jan and that should give enough
time for different contributors. Do not wait till the last moment
though and feel free to comment on both AIPs as soon as you can. I
will be around also to answer those.
On Wed, Dec 15, 2021 at 9:47 PM Mateusz Henc <mh...@google.com.invalid> wrote:
>
> Hi,
> FYI I start my vacation on Monday and I am off for 3 weeks (back on 01/10) with limited access to my laptop.
>
> If you have any comments please add by the end of this week, so I could address them while I am still there.
> If you are fine then maybe we could start voting soon?
>
> // Leaving more time for AIP-44 as it is longer and more complicated.
>
> Best regards,
> Mateusz Henc
>
>
> On Tue, Dec 14, 2021 at 1:09 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Comment on that one too - we also provided more details (including
>> possible deployment options). Similarly as in case of AIP-44 we had a
>> few goals - which also resulted from earlier discussions:
>>
>> * Keep current behavior practically untouched
>> * Allow for different deployments "no separation", "dag processor
>> separation", "multiple dag processors for tenant isolation"
>> * the "multiple dag processor" case does NOT change the ways dags are
>> stored or parsed or where dags are stored. They are all parts of
>> "DAG_FOLDER". The change is merely there to allow each dag processor
>> to be "responsible" for parsing only selected part of the "dag" tree.
>> This is purely to isolate code execution between tenants and has no
>> impact whatsoever on other components (and it has no impact on other
>> ideas there - such as DAG fetcher or DAG submission). It's purely to
>> allow the "deployment" to better isolate code execution of user code
>> coming from different tenants.
>>
>> I hope this one will be faster to discuss and raise less
>> controversies/doubts - but feel absolutely free to comment/criticize
>> etc.
>>
>> J.
>>
>> On Thu, Dec 9, 2021 at 4:36 PM Mateusz Henc <mh...@google.com.invalid> wrote:
>> >
>> > Hi,
>> > We just updated the AIP-43 adding some details and deployment options.
>> > Please take a look. Any comments are appreciated.
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
>> >
>> > BTW Update for AIP-44 is coming soon.
>> >
>> > Best regards,
>> > Mateusz Henc
>> >
>> >
>> > On Thu, Dec 2, 2021 at 3:04 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>
>> >> Excited :)
>> >>
>> >> On Thu, Dec 2, 2021 at 2:20 PM Mateusz Henc <mh...@google.com.invalid> wrote:
>> >> >
>> >> > Hi,
>> >> > I just submitted my proposal for separating the DagProcessor from Scheduler and running it as an independent component. This is part of AIP-1 Airflow multi-tenancy effort.
>> >> >
>> >> > PTAL
>> >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
>> >> >
>> >> > Best regards,
>> >> > Mateusz Henc
Re: [Discuss] AIP-43 DAG Processor separation
Posted by Mateusz Henc <mh...@google.com.INVALID>.
Hi,
FYI I start my vacation on Monday and I am off for 3 weeks (back on 01/10)
with limited access to my laptop.
If you have any comments please add by the end of this week, so I could
address them while I am still there.
If you are fine then maybe we could start voting soon?
// Leaving more time for AIP-44 as it is longer and more complicated.
Best regards,
Mateusz Henc
On Tue, Dec 14, 2021 at 1:09 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Comment on that one too - we also provided more details (including
> possible deployment options). Similarly as in case of AIP-44 we had a
> few goals - which also resulted from earlier discussions:
>
> * Keep current behavior practically untouched
> * Allow for different deployments "no separation", "dag processor
> separation", "multiple dag processors for tenant isolation"
> * the "multiple dag processor" case does NOT change the ways dags are
> stored or parsed or where dags are stored. They are all parts of
> "DAG_FOLDER". The change is merely there to allow each dag processor
> to be "responsible" for parsing only selected part of the "dag" tree.
> This is purely to isolate code execution between tenants and has no
> impact whatsoever on other components (and it has no impact on other
> ideas there - such as DAG fetcher or DAG submission). It's purely to
> allow the "deployment" to better isolate code execution of user code
> coming from different tenants.
>
> I hope this one will be faster to discuss and raise less
> controversies/doubts - but feel absolutely free to comment/criticize
> etc.
>
> J.
>
> On Thu, Dec 9, 2021 at 4:36 PM Mateusz Henc <mh...@google.com.invalid>
> wrote:
> >
> > Hi,
> > We just updated the AIP-43 adding some details and deployment options.
> > Please take a look. Any comments are appreciated.
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
> >
> > BTW Update for AIP-44 is coming soon.
> >
> > Best regards,
> > Mateusz Henc
> >
> >
> > On Thu, Dec 2, 2021 at 3:04 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> Excited :)
> >>
> >> On Thu, Dec 2, 2021 at 2:20 PM Mateusz Henc <mh...@google.com.invalid>
> wrote:
> >> >
> >> > Hi,
> >> > I just submitted my proposal for separating the DagProcessor from
> Scheduler and running it as an independent component. This is part of AIP-1
> Airflow multi-tenancy effort.
> >> >
> >> > PTAL
> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
> >> >
> >> > Best regards,
> >> > Mateusz Henc
>
Re: [Discuss] AIP-43 DAG Processor separation
Posted by Jarek Potiuk <ja...@potiuk.com>.
Comment on that one too - we also provided more details (including
possible deployment options). Similarly as in case of AIP-44 we had a
few goals - which also resulted from earlier discussions:
* Keep current behavior practically untouched
* Allow for different deployments "no separation", "dag processor
separation", "multiple dag processors for tenant isolation"
* the "multiple dag processor" case does NOT change the ways dags are
stored or parsed or where dags are stored. They are all parts of
"DAG_FOLDER". The change is merely there to allow each dag processor
to be "responsible" for parsing only selected part of the "dag" tree.
This is purely to isolate code execution between tenants and has no
impact whatsoever on other components (and it has no impact on other
ideas there - such as DAG fetcher or DAG submission). It's purely to
allow the "deployment" to better isolate code execution of user code
coming from different tenants.
I hope this one will be faster to discuss and raise less
controversies/doubts - but feel absolutely free to comment/criticize
etc.
J.
On Thu, Dec 9, 2021 at 4:36 PM Mateusz Henc <mh...@google.com.invalid> wrote:
>
> Hi,
> We just updated the AIP-43 adding some details and deployment options.
> Please take a look. Any comments are appreciated.
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
>
> BTW Update for AIP-44 is coming soon.
>
> Best regards,
> Mateusz Henc
>
>
> On Thu, Dec 2, 2021 at 3:04 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Excited :)
>>
>> On Thu, Dec 2, 2021 at 2:20 PM Mateusz Henc <mh...@google.com.invalid> wrote:
>> >
>> > Hi,
>> > I just submitted my proposal for separating the DagProcessor from Scheduler and running it as an independent component. This is part of AIP-1 Airflow multi-tenancy effort.
>> >
>> > PTAL
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
>> >
>> > Best regards,
>> > Mateusz Henc
Re: [Discuss] AIP-43 DAG Processor separation
Posted by Mateusz Henc <mh...@google.com.INVALID>.
Hi,
We just updated the AIP-43 adding some details and deployment options.
Please take a look. Any comments are appreciated.
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
BTW Update for AIP-44 is coming soon.
Best regards,
Mateusz Henc
On Thu, Dec 2, 2021 at 3:04 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Excited :)
>
> On Thu, Dec 2, 2021 at 2:20 PM Mateusz Henc <mh...@google.com.invalid>
> wrote:
> >
> > Hi,
> > I just submitted my proposal for separating the DagProcessor from
> Scheduler and running it as an independent component. This is part of AIP-1
> Airflow multi-tenancy effort.
> >
> > PTAL
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
> >
> > Best regards,
> > Mateusz Henc
>
Re: [Discuss] AIP-43 DAG Processor separation
Posted by Jarek Potiuk <ja...@potiuk.com>.
Excited :)
On Thu, Dec 2, 2021 at 2:20 PM Mateusz Henc <mh...@google.com.invalid> wrote:
>
> Hi,
> I just submitted my proposal for separating the DagProcessor from Scheduler and running it as an independent component. This is part of AIP-1 Airflow multi-tenancy effort.
>
> PTAL
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
>
> Best regards,
> Mateusz Henc