You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2024/08/01 20:13:00 UTC
[VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Hello here,
I reviewed and - I hope - addressed all the comments so far. One most
recent change in the doc was to allow a connection/variable to belong to
multiple teams. Also I mentioned a potential future change that will allow
us to run "per-team" schedulers - leaving the UI/DB schema as the main
components that are shared between teams.
I am also fine if this one's delivery will slip to 3.1 - because it has
many dependencies on other AIPs - and also makes some assumptions on
the other AIP-s scope when they are not yet fully fleshed out. So some
implementation details **might** change. the general approach is going to
stay:
* not targeting manageable "mutliple tenants" - but more of a small-ish set
of teams within the same organisation that want to maintain separate
environments and isolated execution environment
* common UI and scheduler (potentially per-team schedule) as the main
"common" piece
* each team being able to manage their team separately - to the extent of
parsing DAGs and executing tasks
The AIP proposal with updates:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
J.
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Hussein Awala <hu...@awala.fr>.
+1 binding
On Thu, Aug 8, 2024 at 5:38 PM Oliveira, Niko <on...@amazon.com.invalid>
wrote:
> +1 (binding)
>
> A lot of effort went into this one, great job Jarek. I'm happy to see this
> one get across the line because we hear from so many users and customers
> who want this feature. It's important to focus on building what users are
> asking for.
>
> ________________________________
> From: Shahar Epstein <sh...@apache.org>
> Sent: Wednesday, August 7, 2024 10:58:27 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [VOTE] AIP-67 - Multi-team deplyment of Airflow
> Components
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 (binding)
> Looking forward!
>
> On 2024/08/01 20:13:00 Jarek Potiuk wrote:
> > Hello here,
> >
> > I reviewed and - I hope - addressed all the comments so far. One most
> > recent change in the doc was to allow a connection/variable to belong to
> > multiple teams. Also I mentioned a potential future change that will
> allow
> > us to run "per-team" schedulers - leaving the UI/DB schema as the main
> > components that are shared between teams.
> >
> > I am also fine if this one's delivery will slip to 3.1 - because it has
> > many dependencies on other AIPs - and also makes some assumptions on
> > the other AIP-s scope when they are not yet fully fleshed out. So some
> > implementation details **might** change. the general approach is going to
> > stay:
> >
> > * not targeting manageable "mutliple tenants" - but more of a small-ish
> set
> > of teams within the same organisation that want to maintain separate
> > environments and isolated execution environment
> >
> > * common UI and scheduler (potentially per-team schedule) as the main
> > "common" piece
> >
> > * each team being able to manage their team separately - to the extent of
> > parsing DAGs and executing tasks
> >
> > The AIP proposal with updates:
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> >
> > J.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Pavankumar Gopidesu <go...@gmail.com>.
+1 (non-binding)
Regards,
Pavan
On Thu, Aug 8, 2024, 16:38 Oliveira, Niko <on...@amazon.com.invalid>
wrote:
> +1 (binding)
>
> A lot of effort went into this one, great job Jarek. I'm happy to see this
> one get across the line because we hear from so many users and customers
> who want this feature. It's important to focus on building what users are
> asking for.
>
> ________________________________
> From: Shahar Epstein <sh...@apache.org>
> Sent: Wednesday, August 7, 2024 10:58:27 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [VOTE] AIP-67 - Multi-team deplyment of Airflow
> Components
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 (binding)
> Looking forward!
>
> On 2024/08/01 20:13:00 Jarek Potiuk wrote:
> > Hello here,
> >
> > I reviewed and - I hope - addressed all the comments so far. One most
> > recent change in the doc was to allow a connection/variable to belong to
> > multiple teams. Also I mentioned a potential future change that will
> allow
> > us to run "per-team" schedulers - leaving the UI/DB schema as the main
> > components that are shared between teams.
> >
> > I am also fine if this one's delivery will slip to 3.1 - because it has
> > many dependencies on other AIPs - and also makes some assumptions on
> > the other AIP-s scope when they are not yet fully fleshed out. So some
> > implementation details **might** change. the general approach is going to
> > stay:
> >
> > * not targeting manageable "mutliple tenants" - but more of a small-ish
> set
> > of teams within the same organisation that want to maintain separate
> > environments and isolated execution environment
> >
> > * common UI and scheduler (potentially per-team schedule) as the main
> > "common" piece
> >
> > * each team being able to manage their team separately - to the extent of
> > parsing DAGs and executing tasks
> >
> > The AIP proposal with updates:
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> >
> > J.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by "Oliveira, Niko" <on...@amazon.com.INVALID>.
+1 (binding)
A lot of effort went into this one, great job Jarek. I'm happy to see this one get across the line because we hear from so many users and customers who want this feature. It's important to focus on building what users are asking for.
________________________________
From: Shahar Epstein <sh...@apache.org>
Sent: Wednesday, August 7, 2024 10:58:27 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
+1 (binding)
Looking forward!
On 2024/08/01 20:13:00 Jarek Potiuk wrote:
> Hello here,
>
> I reviewed and - I hope - addressed all the comments so far. One most
> recent change in the doc was to allow a connection/variable to belong to
> multiple teams. Also I mentioned a potential future change that will allow
> us to run "per-team" schedulers - leaving the UI/DB schema as the main
> components that are shared between teams.
>
> I am also fine if this one's delivery will slip to 3.1 - because it has
> many dependencies on other AIPs - and also makes some assumptions on
> the other AIP-s scope when they are not yet fully fleshed out. So some
> implementation details **might** change. the general approach is going to
> stay:
>
> * not targeting manageable "mutliple tenants" - but more of a small-ish set
> of teams within the same organisation that want to maintain separate
> environments and isolated execution environment
>
> * common UI and scheduler (potentially per-team schedule) as the main
> "common" piece
>
> * each team being able to manage their team separately - to the extent of
> parsing DAGs and executing tasks
>
> The AIP proposal with updates:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>
>
> J.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Shahar Epstein <sh...@apache.org>.
+1 (binding)
Looking forward!
On 2024/08/01 20:13:00 Jarek Potiuk wrote:
> Hello here,
>
> I reviewed and - I hope - addressed all the comments so far. One most
> recent change in the doc was to allow a connection/variable to belong to
> multiple teams. Also I mentioned a potential future change that will allow
> us to run "per-team" schedulers - leaving the UI/DB schema as the main
> components that are shared between teams.
>
> I am also fine if this one's delivery will slip to 3.1 - because it has
> many dependencies on other AIPs - and also makes some assumptions on
> the other AIP-s scope when they are not yet fully fleshed out. So some
> implementation details **might** change. the general approach is going to
> stay:
>
> * not targeting manageable "mutliple tenants" - but more of a small-ish set
> of teams within the same organisation that want to maintain separate
> environments and isolated execution environment
>
> * common UI and scheduler (potentially per-team schedule) as the main
> "common" piece
>
> * each team being able to manage their team separately - to the extent of
> parsing DAGs and executing tasks
>
> The AIP proposal with updates:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>
>
> J.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Buğra Öztürk <oz...@gmail.com>.
+1 non-binding
I’m looking forward to seeing this AIP as it will pave the way for
follow-up features that enable fully manageable multi-tenancy. I’m also
excited about the opportunity to work on this. 🙌
On Thu, Aug 1, 2024 at 10:40 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Voting ends a week from now - i.e Thursday 8th of August 2023 (midnight).
>
> On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hello here,
> >
> > I reviewed and - I hope - addressed all the comments so far. One most
> > recent change in the doc was to allow a connection/variable to belong to
> > multiple teams. Also I mentioned a potential future change that will
> allow
> > us to run "per-team" schedulers - leaving the UI/DB schema as the main
> > components that are shared between teams.
> >
> > I am also fine if this one's delivery will slip to 3.1 - because it has
> > many dependencies on other AIPs - and also makes some assumptions on
> > the other AIP-s scope when they are not yet fully fleshed out. So some
> > implementation details **might** change. the general approach is going to
> > stay:
> >
> > * not targeting manageable "mutliple tenants" - but more of a small-ish
> > set of teams within the same organisation that want to maintain separate
> > environments and isolated execution environment
> >
> > * common UI and scheduler (potentially per-team schedule) as the main
> > "common" piece
> >
> > * each team being able to manage their team separately - to the extent of
> > parsing DAGs and executing tasks
> >
> > The AIP proposal with updates:
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> >
> > J.
> >
> >
>
--
Bugra Ozturk
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Jorrick Sleijster <jo...@gmail.com>.
+1 non-binding.
We have always believed that a single airflow makes it easier for our
users. Unfortunately that leaves much to be desired on isolation
perspective. For that reason our most requested functionality remains;
'can I get a Variable or Connection only accessible to my team?'. Our
users, and especially security team, will be delighted with this
improvement.
On Mon, 5 Aug 2024, 05:19 Guangyang Li, <li...@gmail.com> wrote:
> +1 non-binding
>
> Very happy to see this support! I've seen Airflow in several companies with
> a long long requirements.txt, in order to support the needs from multiple
> teams. I'm sure those maintainers are super excited to have this feature.
>
> On Fri, Aug 2, 2024 at 7:00 PM Vikram Koka <vi...@astronomer.io.invalid>
> wrote:
>
> > +1 binding
> >
> > Thank you Jarek for being very patient and working through all the twists
> > and turns on this AIP.
> >
> > I do believe that the changes are all positive and I really like the
> > evolution towards a more focused multi-team target.
> > This has become an uber-AIP, which is now building on top of AIPs such as
> > the AIP-72 Task Execution Interface, and
> > AIP-73 Data Awareness (only referencing other in-flight AIPs at this
> time,
> > rather than already completed ones).
> > That's fantastic.
> >
> > I think this may require the "Auth manager" work to be done as well,
> which
> > you brought up in a different thread.
> > I noticed that you kept it out of scope of this AIP, which I think is
> > right, but may be needed for the overall use case.
> >
> > Based on the dependencies, I also agree that this would be an Airflow 3.1
> > target, but having a future target use case in mind
> > is also a good thing for the developers of the underlying AIPs to not
> > preclude those scenarios and to flag additional work
> > needed for this use case during implementation.
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Aug 2, 2024 at 1:13 PM Vincent Beck <vi...@apache.org> wrote:
> >
> > > +1 binding. Very good demonstration of perseverance on this one :)
> > >
> > > On 2024/08/02 20:02:00 "Bishundeo, Rajeshwar" wrote:
> > > > +1 non-binding.
> > > >
> > > > Great job to all the folks getting this feature 1-step closer to
> > reality!
> > > >
> > > > -- Rajesh
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 2024-08-02, 3:57 PM, "Mehta, Shubham" <shubhx@amazon.com.INVA
> > > <ma...@amazon.com.INVA>LID> wrote:
> > > >
> > > >
> > > > CAUTION: This email originated from outside of the organization. Do
> not
> > > click links or open attachments unless you can confirm the sender and
> > know
> > > the content is safe.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> > vous
> > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes
> pas
> > > certain que le contenu ne présente aucun risque.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > +1 non-binding.
> > > >
> > > >
> > > > Thanks, Jarek, for continuing to evolve this piece of work based on
> the
> > > hundreds of comments you received from everyone. While there are
> > > dependencies on other AIPs, I'd love to see this implemented in Airflow
> > > 3.0. I hope other AIP authors also consider this as a reality and plan
> > for
> > > a multi-team setup during implementation. Ideally, users migrating to
> > > Airflow 3 should be able to plan for an optimal team structure and
> access
> > > permissions from the outset. The AWS team will definitely help in any
> way
> > > possible to ship this on time. Looking forward to the execution!
> > > >
> > > >
> > > > Shubham
> > > >
> > > >
> > > > On 2024-08-02, 12:46 PM, "Kaxil Naik" <kaxilnaik@gmail.com <mailto:
> > > kaxilnaik@gmail.com> <mailto:kaxilnaik@gmail.com <mailto:
> > > kaxilnaik@gmail.com>>> wrote:
> > > >
> > > >
> > > >
> > > >
> > > > CAUTION: This email originated from outside of the organization. Do
> not
> > > click links or open attachments unless you can confirm the sender and
> > know
> > > the content is safe.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> > vous
> > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes
> pas
> > > certain que le contenu ne présente aucun risque.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > +1 binding
> > > >
> > > >
> > > >
> > > >
> > > > Thank you for persisting with the questions & changes, great job.
> > > >
> > > >
> > > >
> > > >
> > > > Realistically, though, I think this would happen in 3.1 due to
> > > > dependencies, as you mentioned in the email.
> > > >
> > > >
> > > >
> > > >
> > > > Looking forward.
> > > >
> > > >
> > > >
> > > >
> > > > Regards,
> > > > Kaxil
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T)
> > > > <Jens.Scheffler@de.bosch.com.inva <mailto:
> > > Jens.Scheffler@de.bosch.com.inva> <mailto:
> > Jens.Scheffler@de.bosch.com.inva
> > > <ma...@de.bosch.com.inva>>lid> wrote:
> > > >
> > > >
> > > >
> > > >
> > > > > +1 binding.
> > > > >
> > > > > Many thougths and discussions went into this AIP. Also User survery
> > > showed
> > > > > a strong demand for this - same we have also in our house. A multi
> > team
> > > > > setup is eagerly needed.
> > > > > Nevertheless the complexity and missing turn-key delivery will be a
> > > > > challenge. I assume it needs to evolve.
> > > > > Besides a bit sceptic words the possibilities rule-out the
> additional
> > > > > complexity. I think we should commit making it - especially not
> with
> > > > > options to have some more breaking changes in AF3 is the right
> point
> > to
> > > > > plan all meede structural changes in! Thanks Jarek for driving
> this!
> > > > >
> > > > > Sent from Outlook for iOS<https://aka.ms/o0ukef> <
> > > https://aka.ms/o0ukef>> <https://aka.ms/o0ukef>> <
> > > https://aka.ms/o0ukef&gt;>>
> > > > > ________________________________
> > > > > From: Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>
> > > <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> > > > > Sent: Thursday, August 1, 2024 10:40:16 PM
> > > > > To: dev@airflow.apache.org <ma...@airflow.apache.org>
> <mailto:
> > > dev@airflow.apache.org <ma...@airflow.apache.org>> <
> > > dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > > dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > > > > Subject: Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow
> > Components
> > > > >
> > > > > Voting ends a week from now - i.e Thursday 8th of August 2023
> > > (midnight).
> > > > >
> > > > > On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <jarek@potiuk.com
> > > <ma...@potiuk.com> <mailto:jarek@potiuk.com <mailto:
> > > jarek@potiuk.com>>> wrote:
> > > > >
> > > > > > Hello here,
> > > > > >
> > > > > > I reviewed and - I hope - addressed all the comments so far. One
> > most
> > > > > > recent change in the doc was to allow a connection/variable to
> > > belong to
> > > > > > multiple teams. Also I mentioned a potential future change that
> > will
> > > > > allow
> > > > > > us to run "per-team" schedulers - leaving the UI/DB schema as the
> > > main
> > > > > > components that are shared between teams.
> > > > > >
> > > > > > I am also fine if this one's delivery will slip to 3.1 - because
> it
> > > has
> > > > > > many dependencies on other AIPs - and also makes some assumptions
> > on
> > > > > > the other AIP-s scope when they are not yet fully fleshed out. So
> > > some
> > > > > > implementation details **might** change. the general approach is
> > > going to
> > > > > > stay:
> > > > > >
> > > > > > * not targeting manageable "mutliple tenants" - but more of a
> > > small-ish
> > > > > > set of teams within the same organisation that want to maintain
> > > separate
> > > > > > environments and isolated execution environment
> > > > > >
> > > > > > * common UI and scheduler (potentially per-team schedule) as the
> > main
> > > > > > "common" piece
> > > > > >
> > > > > > * each team being able to manage their team separately - to the
> > > extent of
> > > > > > parsing DAGs and executing tasks
> > > > > >
> > > > > > The AIP proposal with updates:
> > > > > >
> > > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0
> > > <
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0
> > >
> > > <
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0
> > >
> > > <
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&amp;reserved=0>
> > > ;>
> > > > > <
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > >
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > >
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>
> > > ;>
> > > > > >
> > > > > >
> > > > > >
> > > > > > J.
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> Т���������������������������������������������������������������������ХF�?V�7V'67&�&R�?R��?�â?FWb�V�7V'67&�&T??�&f��r�???6�R��&pФf�"??FF�F���?�?6���?�G2�?R��?�â?FWbֆV�???�&f��r�???6�R��&p
> > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Guangyang Li <li...@gmail.com>.
+1 non-binding
Very happy to see this support! I've seen Airflow in several companies with
a long long requirements.txt, in order to support the needs from multiple
teams. I'm sure those maintainers are super excited to have this feature.
On Fri, Aug 2, 2024 at 7:00 PM Vikram Koka <vi...@astronomer.io.invalid>
wrote:
> +1 binding
>
> Thank you Jarek for being very patient and working through all the twists
> and turns on this AIP.
>
> I do believe that the changes are all positive and I really like the
> evolution towards a more focused multi-team target.
> This has become an uber-AIP, which is now building on top of AIPs such as
> the AIP-72 Task Execution Interface, and
> AIP-73 Data Awareness (only referencing other in-flight AIPs at this time,
> rather than already completed ones).
> That's fantastic.
>
> I think this may require the "Auth manager" work to be done as well, which
> you brought up in a different thread.
> I noticed that you kept it out of scope of this AIP, which I think is
> right, but may be needed for the overall use case.
>
> Based on the dependencies, I also agree that this would be an Airflow 3.1
> target, but having a future target use case in mind
> is also a good thing for the developers of the underlying AIPs to not
> preclude those scenarios and to flag additional work
> needed for this use case during implementation.
>
>
>
>
>
>
>
>
> On Fri, Aug 2, 2024 at 1:13 PM Vincent Beck <vi...@apache.org> wrote:
>
> > +1 binding. Very good demonstration of perseverance on this one :)
> >
> > On 2024/08/02 20:02:00 "Bishundeo, Rajeshwar" wrote:
> > > +1 non-binding.
> > >
> > > Great job to all the folks getting this feature 1-step closer to
> reality!
> > >
> > > -- Rajesh
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 2024-08-02, 3:57 PM, "Mehta, Shubham" <shubhx@amazon.com.INVA
> > <ma...@amazon.com.INVA>LID> wrote:
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> > >
> > >
> > >
> > >
> > >
> > >
> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> vous
> > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > certain que le contenu ne présente aucun risque.
> > >
> > >
> > >
> > >
> > >
> > >
> > > +1 non-binding.
> > >
> > >
> > > Thanks, Jarek, for continuing to evolve this piece of work based on the
> > hundreds of comments you received from everyone. While there are
> > dependencies on other AIPs, I'd love to see this implemented in Airflow
> > 3.0. I hope other AIP authors also consider this as a reality and plan
> for
> > a multi-team setup during implementation. Ideally, users migrating to
> > Airflow 3 should be able to plan for an optimal team structure and access
> > permissions from the outset. The AWS team will definitely help in any way
> > possible to ship this on time. Looking forward to the execution!
> > >
> > >
> > > Shubham
> > >
> > >
> > > On 2024-08-02, 12:46 PM, "Kaxil Naik" <kaxilnaik@gmail.com <mailto:
> > kaxilnaik@gmail.com> <mailto:kaxilnaik@gmail.com <mailto:
> > kaxilnaik@gmail.com>>> wrote:
> > >
> > >
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> vous
> > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > certain que le contenu ne présente aucun risque.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > +1 binding
> > >
> > >
> > >
> > >
> > > Thank you for persisting with the questions & changes, great job.
> > >
> > >
> > >
> > >
> > > Realistically, though, I think this would happen in 3.1 due to
> > > dependencies, as you mentioned in the email.
> > >
> > >
> > >
> > >
> > > Looking forward.
> > >
> > >
> > >
> > >
> > > Regards,
> > > Kaxil
> > >
> > >
> > >
> > >
> > > On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T)
> > > <Jens.Scheffler@de.bosch.com.inva <mailto:
> > Jens.Scheffler@de.bosch.com.inva> <mailto:
> Jens.Scheffler@de.bosch.com.inva
> > <ma...@de.bosch.com.inva>>lid> wrote:
> > >
> > >
> > >
> > >
> > > > +1 binding.
> > > >
> > > > Many thougths and discussions went into this AIP. Also User survery
> > showed
> > > > a strong demand for this - same we have also in our house. A multi
> team
> > > > setup is eagerly needed.
> > > > Nevertheless the complexity and missing turn-key delivery will be a
> > > > challenge. I assume it needs to evolve.
> > > > Besides a bit sceptic words the possibilities rule-out the additional
> > > > complexity. I think we should commit making it - especially not with
> > > > options to have some more breaking changes in AF3 is the right point
> to
> > > > plan all meede structural changes in! Thanks Jarek for driving this!
> > > >
> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef> <
> > https://aka.ms/o0ukef>> <https://aka.ms/o0ukef>> <
> > https://aka.ms/o0ukef&gt;>>
> > > > ________________________________
> > > > From: Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>
> > <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> > > > Sent: Thursday, August 1, 2024 10:40:16 PM
> > > > To: dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > dev@airflow.apache.org <ma...@airflow.apache.org>> <
> > dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > > > Subject: Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow
> Components
> > > >
> > > > Voting ends a week from now - i.e Thursday 8th of August 2023
> > (midnight).
> > > >
> > > > On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <jarek@potiuk.com
> > <ma...@potiuk.com> <mailto:jarek@potiuk.com <mailto:
> > jarek@potiuk.com>>> wrote:
> > > >
> > > > > Hello here,
> > > > >
> > > > > I reviewed and - I hope - addressed all the comments so far. One
> most
> > > > > recent change in the doc was to allow a connection/variable to
> > belong to
> > > > > multiple teams. Also I mentioned a potential future change that
> will
> > > > allow
> > > > > us to run "per-team" schedulers - leaving the UI/DB schema as the
> > main
> > > > > components that are shared between teams.
> > > > >
> > > > > I am also fine if this one's delivery will slip to 3.1 - because it
> > has
> > > > > many dependencies on other AIPs - and also makes some assumptions
> on
> > > > > the other AIP-s scope when they are not yet fully fleshed out. So
> > some
> > > > > implementation details **might** change. the general approach is
> > going to
> > > > > stay:
> > > > >
> > > > > * not targeting manageable "mutliple tenants" - but more of a
> > small-ish
> > > > > set of teams within the same organisation that want to maintain
> > separate
> > > > > environments and isolated execution environment
> > > > >
> > > > > * common UI and scheduler (potentially per-team schedule) as the
> main
> > > > > "common" piece
> > > > >
> > > > > * each team being able to manage their team separately - to the
> > extent of
> > > > > parsing DAGs and executing tasks
> > > > >
> > > > > The AIP proposal with updates:
> > > > >
> > > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0
> > <
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0
> >
> > <
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0
> >
> > <
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&amp;reserved=0>
> > ;>
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>
> > ;>
> > > > >
> > > > >
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> Т���������������������������������������������������������������������ХF�?V�7V'67&�&R�?R��?�â?FWb�V�7V'67&�&T??�&f��r�???6�R��&pФf�"??FF�F���?�?6���?�G2�?R��?�â?FWbֆV�???�&f��r�???6�R��&p
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
+1 binding
Thank you Jarek for being very patient and working through all the twists
and turns on this AIP.
I do believe that the changes are all positive and I really like the
evolution towards a more focused multi-team target.
This has become an uber-AIP, which is now building on top of AIPs such as
the AIP-72 Task Execution Interface, and
AIP-73 Data Awareness (only referencing other in-flight AIPs at this time,
rather than already completed ones).
That's fantastic.
I think this may require the "Auth manager" work to be done as well, which
you brought up in a different thread.
I noticed that you kept it out of scope of this AIP, which I think is
right, but may be needed for the overall use case.
Based on the dependencies, I also agree that this would be an Airflow 3.1
target, but having a future target use case in mind
is also a good thing for the developers of the underlying AIPs to not
preclude those scenarios and to flag additional work
needed for this use case during implementation.
On Fri, Aug 2, 2024 at 1:13 PM Vincent Beck <vi...@apache.org> wrote:
> +1 binding. Very good demonstration of perseverance on this one :)
>
> On 2024/08/02 20:02:00 "Bishundeo, Rajeshwar" wrote:
> > +1 non-binding.
> >
> > Great job to all the folks getting this feature 1-step closer to reality!
> >
> > -- Rajesh
> >
> >
> >
> >
> >
> >
> > On 2024-08-02, 3:57 PM, "Mehta, Shubham" <shubhx@amazon.com.INVA
> <ma...@amazon.com.INVA>LID> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain que le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > +1 non-binding.
> >
> >
> > Thanks, Jarek, for continuing to evolve this piece of work based on the
> hundreds of comments you received from everyone. While there are
> dependencies on other AIPs, I'd love to see this implemented in Airflow
> 3.0. I hope other AIP authors also consider this as a reality and plan for
> a multi-team setup during implementation. Ideally, users migrating to
> Airflow 3 should be able to plan for an optimal team structure and access
> permissions from the outset. The AWS team will definitely help in any way
> possible to ship this on time. Looking forward to the execution!
> >
> >
> > Shubham
> >
> >
> > On 2024-08-02, 12:46 PM, "Kaxil Naik" <kaxilnaik@gmail.com <mailto:
> kaxilnaik@gmail.com> <mailto:kaxilnaik@gmail.com <mailto:
> kaxilnaik@gmail.com>>> wrote:
> >
> >
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain que le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > +1 binding
> >
> >
> >
> >
> > Thank you for persisting with the questions & changes, great job.
> >
> >
> >
> >
> > Realistically, though, I think this would happen in 3.1 due to
> > dependencies, as you mentioned in the email.
> >
> >
> >
> >
> > Looking forward.
> >
> >
> >
> >
> > Regards,
> > Kaxil
> >
> >
> >
> >
> > On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T)
> > <Jens.Scheffler@de.bosch.com.inva <mailto:
> Jens.Scheffler@de.bosch.com.inva> <mailto:Jens.Scheffler@de.bosch.com.inva
> <ma...@de.bosch.com.inva>>lid> wrote:
> >
> >
> >
> >
> > > +1 binding.
> > >
> > > Many thougths and discussions went into this AIP. Also User survery
> showed
> > > a strong demand for this - same we have also in our house. A multi team
> > > setup is eagerly needed.
> > > Nevertheless the complexity and missing turn-key delivery will be a
> > > challenge. I assume it needs to evolve.
> > > Besides a bit sceptic words the possibilities rule-out the additional
> > > complexity. I think we should commit making it - especially not with
> > > options to have some more breaking changes in AF3 is the right point to
> > > plan all meede structural changes in! Thanks Jarek for driving this!
> > >
> > > Sent from Outlook for iOS<https://aka.ms/o0ukef> <
> https://aka.ms/o0ukef>> <https://aka.ms/o0ukef>> <
> https://aka.ms/o0ukef&gt;>>
> > > ________________________________
> > > From: Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>
> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> > > Sent: Thursday, August 1, 2024 10:40:16 PM
> > > To: dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> dev@airflow.apache.org <ma...@airflow.apache.org>> <
> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > > Subject: Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
> > >
> > > Voting ends a week from now - i.e Thursday 8th of August 2023
> (midnight).
> > >
> > > On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com> <mailto:jarek@potiuk.com <mailto:
> jarek@potiuk.com>>> wrote:
> > >
> > > > Hello here,
> > > >
> > > > I reviewed and - I hope - addressed all the comments so far. One most
> > > > recent change in the doc was to allow a connection/variable to
> belong to
> > > > multiple teams. Also I mentioned a potential future change that will
> > > allow
> > > > us to run "per-team" schedulers - leaving the UI/DB schema as the
> main
> > > > components that are shared between teams.
> > > >
> > > > I am also fine if this one's delivery will slip to 3.1 - because it
> has
> > > > many dependencies on other AIPs - and also makes some assumptions on
> > > > the other AIP-s scope when they are not yet fully fleshed out. So
> some
> > > > implementation details **might** change. the general approach is
> going to
> > > > stay:
> > > >
> > > > * not targeting manageable "mutliple tenants" - but more of a
> small-ish
> > > > set of teams within the same organisation that want to maintain
> separate
> > > > environments and isolated execution environment
> > > >
> > > > * common UI and scheduler (potentially per-team schedule) as the main
> > > > "common" piece
> > > >
> > > > * each team being able to manage their team separately - to the
> extent of
> > > > parsing DAGs and executing tasks
> > > >
> > > > The AIP proposal with updates:
> > > >
> > >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0>
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0>
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&amp;reserved=0>
> ;>
> > > <
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>
> ;>
> > > >
> > > >
> > > >
> > > > J.
> > > >
> > > >
> > >
> >
> >
> >
> >
> >
> >
> >
> Т���������������������������������������������������������������������ХF�?V�7V'67&�&R�?R��?�â?FWb�V�7V'67&�&T??�&f��r�???6�R��&pФf�"??FF�F���?�?6���?�G2�?R��?�â?FWbֆV�???�&f��r�???6�R��&p
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Vincent Beck <vi...@apache.org>.
+1 binding. Very good demonstration of perseverance on this one :)
On 2024/08/02 20:02:00 "Bishundeo, Rajeshwar" wrote:
> +1 non-binding.
>
> Great job to all the folks getting this feature 1-step closer to reality!
>
> -- Rajesh
>
>
>
>
>
>
> On 2024-08-02, 3:57 PM, "Mehta, Shubham" <shubhx@amazon.com.INVA <ma...@amazon.com.INVA>LID> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
>
>
>
>
>
>
> +1 non-binding.
>
>
> Thanks, Jarek, for continuing to evolve this piece of work based on the hundreds of comments you received from everyone. While there are dependencies on other AIPs, I'd love to see this implemented in Airflow 3.0. I hope other AIP authors also consider this as a reality and plan for a multi-team setup during implementation. Ideally, users migrating to Airflow 3 should be able to plan for an optimal team structure and access permissions from the outset. The AWS team will definitely help in any way possible to ship this on time. Looking forward to the execution!
>
>
> Shubham
>
>
> On 2024-08-02, 12:46 PM, "Kaxil Naik" <kaxilnaik@gmail.com <ma...@gmail.com> <mailto:kaxilnaik@gmail.com <ma...@gmail.com>>> wrote:
>
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
>
>
>
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
>
>
>
>
>
>
>
>
>
>
>
>
> +1 binding
>
>
>
>
> Thank you for persisting with the questions & changes, great job.
>
>
>
>
> Realistically, though, I think this would happen in 3.1 due to
> dependencies, as you mentioned in the email.
>
>
>
>
> Looking forward.
>
>
>
>
> Regards,
> Kaxil
>
>
>
>
> On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T)
> <Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva> <mailto:Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva>>lid> wrote:
>
>
>
>
> > +1 binding.
> >
> > Many thougths and discussions went into this AIP. Also User survery showed
> > a strong demand for this - same we have also in our house. A multi team
> > setup is eagerly needed.
> > Nevertheless the complexity and missing turn-key delivery will be a
> > challenge. I assume it needs to evolve.
> > Besides a bit sceptic words the possibilities rule-out the additional
> > complexity. I think we should commit making it - especially not with
> > options to have some more breaking changes in AF3 is the right point to
> > plan all meede structural changes in! Thanks Jarek for driving this!
> >
> > Sent from Outlook for iOS<https://aka.ms/o0ukef> <https://aka.ms/o0ukef>> <https://aka.ms/o0ukef>> <https://aka.ms/o0ukef&gt;>>
> > ________________________________
> > From: Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> > Sent: Thursday, August 1, 2024 10:40:16 PM
> > To: dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>> <dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > Subject: Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
> >
> > Voting ends a week from now - i.e Thursday 8th of August 2023 (midnight).
> >
> > On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> wrote:
> >
> > > Hello here,
> > >
> > > I reviewed and - I hope - addressed all the comments so far. One most
> > > recent change in the doc was to allow a connection/variable to belong to
> > > multiple teams. Also I mentioned a potential future change that will
> > allow
> > > us to run "per-team" schedulers - leaving the UI/DB schema as the main
> > > components that are shared between teams.
> > >
> > > I am also fine if this one's delivery will slip to 3.1 - because it has
> > > many dependencies on other AIPs - and also makes some assumptions on
> > > the other AIP-s scope when they are not yet fully fleshed out. So some
> > > implementation details **might** change. the general approach is going to
> > > stay:
> > >
> > > * not targeting manageable "mutliple tenants" - but more of a small-ish
> > > set of teams within the same organisation that want to maintain separate
> > > environments and isolated execution environment
> > >
> > > * common UI and scheduler (potentially per-team schedule) as the main
> > > "common" piece
> > >
> > > * each team being able to manage their team separately - to the extent of
> > > parsing DAGs and executing tasks
> > >
> > > The AIP proposal with updates:
> > >
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0 <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&amp;reserved=0>>
> > <
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>>
> > >
> > >
> > >
> > > J.
> > >
> > >
> >
>
>
>
>
>
>
> Т���������������������������������������������������������������������ХF�?V�7V'67&�&R�?R��?�â?FWb�V�7V'67&�&T??�&f��r�???6�R��&pФf�"??FF�F���?�?6���?�G2�?R��?�â?FWbֆV�???�&f��r�???6�R��&p
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by "Bishundeo, Rajeshwar" <rb...@amazon.com.INVALID>.
+1 non-binding.
Great job to all the folks getting this feature 1-step closer to reality!
-- Rajesh
On 2024-08-02, 3:57 PM, "Mehta, Shubham" <shubhx@amazon.com.INVA <ma...@amazon.com.INVA>LID> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
+1 non-binding.
Thanks, Jarek, for continuing to evolve this piece of work based on the hundreds of comments you received from everyone. While there are dependencies on other AIPs, I'd love to see this implemented in Airflow 3.0. I hope other AIP authors also consider this as a reality and plan for a multi-team setup during implementation. Ideally, users migrating to Airflow 3 should be able to plan for an optimal team structure and access permissions from the outset. The AWS team will definitely help in any way possible to ship this on time. Looking forward to the execution!
Shubham
On 2024-08-02, 12:46 PM, "Kaxil Naik" <kaxilnaik@gmail.com <ma...@gmail.com> <mailto:kaxilnaik@gmail.com <ma...@gmail.com>>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
+1 binding
Thank you for persisting with the questions & changes, great job.
Realistically, though, I think this would happen in 3.1 due to
dependencies, as you mentioned in the email.
Looking forward.
Regards,
Kaxil
On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T)
<Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva> <mailto:Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva>>lid> wrote:
> +1 binding.
>
> Many thougths and discussions went into this AIP. Also User survery showed
> a strong demand for this - same we have also in our house. A multi team
> setup is eagerly needed.
> Nevertheless the complexity and missing turn-key delivery will be a
> challenge. I assume it needs to evolve.
> Besides a bit sceptic words the possibilities rule-out the additional
> complexity. I think we should commit making it - especially not with
> options to have some more breaking changes in AF3 is the right point to
> plan all meede structural changes in! Thanks Jarek for driving this!
>
> Sent from Outlook for iOS<https://aka.ms/o0ukef> <https://aka.ms/o0ukef>> <https://aka.ms/o0ukef>> <https://aka.ms/o0ukef&gt;>>
> ________________________________
> From: Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> Sent: Thursday, August 1, 2024 10:40:16 PM
> To: dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>> <dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>>
> Subject: Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
>
> Voting ends a week from now - i.e Thursday 8th of August 2023 (midnight).
>
> On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> wrote:
>
> > Hello here,
> >
> > I reviewed and - I hope - addressed all the comments so far. One most
> > recent change in the doc was to allow a connection/variable to belong to
> > multiple teams. Also I mentioned a potential future change that will
> allow
> > us to run "per-team" schedulers - leaving the UI/DB schema as the main
> > components that are shared between teams.
> >
> > I am also fine if this one's delivery will slip to 3.1 - because it has
> > many dependencies on other AIPs - and also makes some assumptions on
> > the other AIP-s scope when they are not yet fully fleshed out. So some
> > implementation details **might** change. the general approach is going to
> > stay:
> >
> > * not targeting manageable "mutliple tenants" - but more of a small-ish
> > set of teams within the same organisation that want to maintain separate
> > environments and isolated execution environment
> >
> > * common UI and scheduler (potentially per-team schedule) as the main
> > "common" piece
> >
> > * each team being able to manage their team separately - to the extent of
> > parsing DAGs and executing tasks
> >
> > The AIP proposal with updates:
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0 <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&amp;reserved=0>>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>>
> >
> >
> >
> > J.
> >
> >
>
Т���������������������������������������������������������������������ХF�?V�7V'67&�&R�?R��?�â?FWb�V�7V'67&�&T??�&f��r�???6�R��&pФf�"??FF�F���?�?6���?�G2�?R��?�â?FWbֆV�???�&f��r�???6�R��&p
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by "Mehta, Shubham" <sh...@amazon.com.INVALID>.
+1 non-binding.
Thanks, Jarek, for continuing to evolve this piece of work based on the hundreds of comments you received from everyone. While there are dependencies on other AIPs, I'd love to see this implemented in Airflow 3.0. I hope other AIP authors also consider this as a reality and plan for a multi-team setup during implementation. Ideally, users migrating to Airflow 3 should be able to plan for an optimal team structure and access permissions from the outset. The AWS team will definitely help in any way possible to ship this on time. Looking forward to the execution!
Shubham
On 2024-08-02, 12:46 PM, "Kaxil Naik" <kaxilnaik@gmail.com <ma...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
+1 binding
Thank you for persisting with the questions & changes, great job.
Realistically, though, I think this would happen in 3.1 due to
dependencies, as you mentioned in the email.
Looking forward.
Regards,
Kaxil
On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T)
<Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva>lid> wrote:
> +1 binding.
>
> Many thougths and discussions went into this AIP. Also User survery showed
> a strong demand for this - same we have also in our house. A multi team
> setup is eagerly needed.
> Nevertheless the complexity and missing turn-key delivery will be a
> challenge. I assume it needs to evolve.
> Besides a bit sceptic words the possibilities rule-out the additional
> complexity. I think we should commit making it - especially not with
> options to have some more breaking changes in AF3 is the right point to
> plan all meede structural changes in! Thanks Jarek for driving this!
>
> Sent from Outlook for iOS<https://aka.ms/o0ukef> <https://aka.ms/o0ukef>>
> ________________________________
> From: Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>>
> Sent: Thursday, August 1, 2024 10:40:16 PM
> To: dev@airflow.apache.org <ma...@airflow.apache.org> <dev@airflow.apache.org <ma...@airflow.apache.org>>
> Subject: Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
>
> Voting ends a week from now - i.e Thursday 8th of August 2023 (midnight).
>
> On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
>
> > Hello here,
> >
> > I reviewed and - I hope - addressed all the comments so far. One most
> > recent change in the doc was to allow a connection/variable to belong to
> > multiple teams. Also I mentioned a potential future change that will
> allow
> > us to run "per-team" schedulers - leaving the UI/DB schema as the main
> > components that are shared between teams.
> >
> > I am also fine if this one's delivery will slip to 3.1 - because it has
> > many dependencies on other AIPs - and also makes some assumptions on
> > the other AIP-s scope when they are not yet fully fleshed out. So some
> > implementation details **might** change. the general approach is going to
> > stay:
> >
> > * not targeting manageable "mutliple tenants" - but more of a small-ish
> > set of teams within the same organisation that want to maintain separate
> > environments and isolated execution environment
> >
> > * common UI and scheduler (potentially per-team schedule) as the main
> > "common" piece
> >
> > * each team being able to manage their team separately - to the extent of
> > parsing DAGs and executing tasks
> >
> > The AIP proposal with updates:
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0 <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>
> >
> >
> >
> > J.
> >
> >
>
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Kaxil Naik <ka...@gmail.com>.
+1 binding
Thank you for persisting with the questions & changes, great job.
Realistically, though, I think this would happen in 3.1 due to
dependencies, as you mentioned in the email.
Looking forward.
Regards,
Kaxil
On Fri, 2 Aug 2024 at 20:30, Scheffler Jens (XC-AS/EAE-ADA-T)
<Je...@de.bosch.com.invalid> wrote:
> +1 binding.
>
> Many thougths and discussions went into this AIP. Also User survery showed
> a strong demand for this - same we have also in our house. A multi team
> setup is eagerly needed.
> Nevertheless the complexity and missing turn-key delivery will be a
> challenge. I assume it needs to evolve.
> Besides a bit sceptic words the possibilities rule-out the additional
> complexity. I think we should commit making it - especially not with
> options to have some more breaking changes in AF3 is the right point to
> plan all meede structural changes in! Thanks Jarek for driving this!
>
> Sent from Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Thursday, August 1, 2024 10:40:16 PM
> To: dev@airflow.apache.org <de...@airflow.apache.org>
> Subject: Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
>
> Voting ends a week from now - i.e Thursday 8th of August 2023 (midnight).
>
> On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hello here,
> >
> > I reviewed and - I hope - addressed all the comments so far. One most
> > recent change in the doc was to allow a connection/variable to belong to
> > multiple teams. Also I mentioned a potential future change that will
> allow
> > us to run "per-team" schedulers - leaving the UI/DB schema as the main
> > components that are shared between teams.
> >
> > I am also fine if this one's delivery will slip to 3.1 - because it has
> > many dependencies on other AIPs - and also makes some assumptions on
> > the other AIP-s scope when they are not yet fully fleshed out. So some
> > implementation details **might** change. the general approach is going to
> > stay:
> >
> > * not targeting manageable "mutliple tenants" - but more of a small-ish
> > set of teams within the same organisation that want to maintain separate
> > environments and isolated execution environment
> >
> > * common UI and scheduler (potentially per-team schedule) as the main
> > "common" piece
> >
> > * each team being able to manage their team separately - to the extent of
> > parsing DAGs and executing tasks
> >
> > The AIP proposal with updates:
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> >
> >
> > J.
> >
> >
>
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by "Scheffler Jens (XC-AS/EAE-ADA-T)" <Je...@de.bosch.com.INVALID>.
+1 binding.
Many thougths and discussions went into this AIP. Also User survery showed a strong demand for this - same we have also in our house. A multi team setup is eagerly needed.
Nevertheless the complexity and missing turn-key delivery will be a challenge. I assume it needs to evolve.
Besides a bit sceptic words the possibilities rule-out the additional complexity. I think we should commit making it - especially not with options to have some more breaking changes in AF3 is the right point to plan all meede structural changes in! Thanks Jarek for driving this!
Sent from Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Thursday, August 1, 2024 10:40:16 PM
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Voting ends a week from now - i.e Thursday 8th of August 2023 (midnight).
On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello here,
>
> I reviewed and - I hope - addressed all the comments so far. One most
> recent change in the doc was to allow a connection/variable to belong to
> multiple teams. Also I mentioned a potential future change that will allow
> us to run "per-team" schedulers - leaving the UI/DB schema as the main
> components that are shared between teams.
>
> I am also fine if this one's delivery will slip to 3.1 - because it has
> many dependencies on other AIPs - and also makes some assumptions on
> the other AIP-s scope when they are not yet fully fleshed out. So some
> implementation details **might** change. the general approach is going to
> stay:
>
> * not targeting manageable "mutliple tenants" - but more of a small-ish
> set of teams within the same organisation that want to maintain separate
> environments and isolated execution environment
>
> * common UI and scheduler (potentially per-team schedule) as the main
> "common" piece
>
> * each team being able to manage their team separately - to the extent of
> parsing DAGs and executing tasks
>
> The AIP proposal with updates:
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-67%2BMulti-team%2Bdeployment%2Bof%2BAirflow%2Bcomponents&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C3f883fcfd40c41a9724608dcb26a3538%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638581416424218968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=LXRWcx%2B0xBpi6jeIb%2BJd4v72y8GojvYhbonctkx1sZ8%3D&reserved=0<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components>
>
>
> J.
>
>
Re: [VOTE] AIP-67 - Multi-team deplyment of Airflow Components
Posted by Jarek Potiuk <ja...@potiuk.com>.
Voting ends a week from now - i.e Thursday 8th of August 2023 (midnight).
On Thu, Aug 1, 2024 at 10:13 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello here,
>
> I reviewed and - I hope - addressed all the comments so far. One most
> recent change in the doc was to allow a connection/variable to belong to
> multiple teams. Also I mentioned a potential future change that will allow
> us to run "per-team" schedulers - leaving the UI/DB schema as the main
> components that are shared between teams.
>
> I am also fine if this one's delivery will slip to 3.1 - because it has
> many dependencies on other AIPs - and also makes some assumptions on
> the other AIP-s scope when they are not yet fully fleshed out. So some
> implementation details **might** change. the general approach is going to
> stay:
>
> * not targeting manageable "mutliple tenants" - but more of a small-ish
> set of teams within the same organisation that want to maintain separate
> environments and isolated execution environment
>
> * common UI and scheduler (potentially per-team schedule) as the main
> "common" piece
>
> * each team being able to manage their team separately - to the extent of
> parsing DAGs and executing tasks
>
> The AIP proposal with updates:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>
>
> J.
>
>