Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Ash Berlin-Taylor <as...@apache.org>]

Hi everyone,

One thing I’ve been struggling with while reading the other thread about multi-team DB changes[0] is what is the end-user problem we are trying to address with it.

The main impetus for opening this discussion is that a lot has changed in Airflow since this AIP was created in early 2024 and voted on mid-2024, and I'm wondering if those changes are big enough to invalidate the design and assumptions made at the time.

Reading the DB changes thread I see that that changes are far reaching and necessarily have to touch most of the Airflow object models, and this got me thinking about what value do we actually get with the change, since as stated in the AIP some of the non-goals are[1] (slightly edited here for brevity with the “[…]"):


> • Sharing broker/backend for celery executors between teams. This MAY be covered by future AIPs
> • Implementation of FAB-based multi-team Auth Manager. […]
> • Per-team concurrency and prioritization of tasks. […].
> • Resource allocation per-executor. In the current proposal, executors are run as sub-processes of Scheduler and we have very little control over their individual resource usage. […]
> • Turn-key multi-team Deployment of Airflow (for example via Helm chart). This is unlikely to happen.[…]
> • team management tools (creation, removal, rename etc.). […]
> • Combining "global" execution with "team" execution. While it should be possible in the proposed architecture to have a "team" execution and "global" execution in a single instance of Airflow, this has it's own unique set of challenges and assumption is that Airflow Deployment is either "global" (today) or "multi-team" (After this AIP is implemented) - but it cannot be combined (yet). This is possible to be implemented in the future.
> • Running multiple schedulers - one-per team. While it should be possible if we add support to select DAGs "per team" per scheduler, this is not implemented in this AIP and left for the future

And also Design Non-goals from the AIP [2]: 

> • It’s not a primary goal of this proposal to significantly decrease resource consumption for Airflow installation compared to the current ways of achieving “multi-tenant” setup. […]
> • It’s not a goal of the proposal to provide a one-stop installation mechanism for “Multi-team” Airflow. […]
> • It’s not a goal to decrease the overall maintenance effort involved in responding to needs of different teams, […]

The main pain point that we seem to be addressing with this AIP is this[3]:

> The main reason for having multi-team deployment of Airflow is achieving security and isolation between the teams, coupled with ability of the isolated teams to collaborate via shared Datasets.


So what’s changed since we collectively (myself included) voted on and accepted this AIP? Well, we now have AIP-82 — External event driven dags. That could be used to achieve this goal right now in 3.0 with no changes to Airflow itself, and is perhaps a more robust mechanism of doing it too.

So my main question, given the wide reaching code changes need for AIP-67, and (IMO) the imperfect/limited scope of team completion I wonder if using AIP-82 would not be a better solution to the problem.

1. It’s much simpler from a code level, as nothing need to change
2. It’s not _that_ much more complex from an operational point of view (you have to run an extra scheduler and web server,  but those would likely need scaling up.)
3. We won’t disappoint people by not implementing the part of multi-team that they want (Someone being part of multiple teams, sharing connections/vars between teams)

And using this mechanism (of external dataset/asset polling) also negates one of the biggest cons of the AIP-67, that of the tight coupling of Airflow versions between the teams. In larger companies this is a _huge_ problem already, and this would only make it worse.

So what’s my idea (and at this stage is it only an idea for discussion) is that we re-evalute AIP-67 in light of what exists in Airflow 3.0 now and decide if it’s still worth the added complexity of DB, code and operational overhead, and decided if we still want it.

Please, please, please point out if there are other benefits that I have missed, I'm not trying to be selective and get my way, I'm trying to make sure Airflow continues to meet the need of users, and can also continue to evolve (where I worry that complexity of code/datamodel materially hurts that final point)

Thoughts?

[0]: https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
[1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope?
[2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals
[3]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

To be honest I am glad you raised this argument - because it means that the
goal I had in mind has been achieved. People will not think that it is easy
to add more and more teams to the picture - the more they add, the more
messy it will get.

That's quite cool if you take into account what AIP-67 goal is.

pon., 7 lip 2025, 21:53 użytkownik Jarek Potiuk <ja...@potiuk.com> napisał:

> To be honest ugliness of it and the messiness of supporting many teams is
> deliberate choice. This is also to add friction in supporting 'many' teams.
> Supporting 'many' teams had never been a goal for multi-team even in the
> last incarnation. Due to scaling of scheduler especially - we should not
> give the impression that this solution can scale to 'many' teams.
>
> So that choice is pretty deliberate.
>
> pon., 7 lip 2025, 21:27 użytkownik Oliveira, Niko
> <on...@amazon.com.invalid> napisał:
>
>> Thanks for getting back to me and sharing a concrete example Jarek!
>>
>> I think overloading the executor field, yet again, with another dimension
>> is going to get quite messy. If you have many teams and many executors per
>> team you now have colons (serving two purposes), semi-colons and commas in
>> one config that will be tremendously long. I think this is a poor user
>> experience. Same goes for the executor configurations themselves.
>> And for someone who's looked through the code, I honestly and
>> respectfully, disagree that this is going to be simpler code to write and
>> maintain. I think it's actually cleaner and easier for both the users and
>> our code base to implement this with first class support, rather than
>> embedding team-ness into configs one-by-one and hard coding that
>> parsing/support across the code base in specific locations where those
>> particular configs are used. It's messy and does not scale well (it smells
>> of executor coupling problem we had years ago, this is going to be
>> team-coupling where random bits of our code base are going to have to have
>> these hacks instead of a first class solution flowing through more
>> transparently).
>>
>> Cheers,
>> Niko
>>
>> ________________________________
>> From: Jarek Potiuk <ja...@potiuk.com>
>> Sent: Monday, July 7, 2025 11:15:16 AM
>> To: dev@airflow.apache.org
>> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External
>> event driven dags) exists
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>>
>>
>>
>> I realized that I owe Niko an explanation of configuration changes. Again
>> -
>> following the philosophy above - minimal set of changes to "airflow
>> internals". the "minimum" set of changes that will work. I propose the
>> change below that has **no** changes to the way how the
>> current configuration "shared" feature works - it will change the way
>> executors will retrieve their configuration if they are configured
>> "per-team" - and we can 100% bank on existing multi-executors.
>> I believe that will absolutely minimise the set of changes needed to
>> implement multi-team and we will be able to get it "faster" and with "far
>> lower risk" of impacting airflow code and say - 3.1 or 3.2 delivery.
>>
>> Existing multi-executor configuration will be extended to include team
>> prefix. The prefix will be separated with ":", entries for different teams
>> will be separated with ";"
>>
>> [core]
>>
>> executor = team1:KubernetesExecutor,my.custom.module.Executor
>> Class;team2:CeleryExecutor
>>
>> The configuration of executors will also be prefixed with the same team:
>>
>> [team1:kubernetes_executor]
>>
>> api_client_retry_configuration = { "total": 3, "backoff_factor": 0.5 }
>>
>> The environment variables keeping configuration will use ___  (three
>> underscores) to replace ":". For example:
>>
>> AIRFLOW__TEAM1___KUBERNETES_EXECUTOR__API_CLIENT_RETRY_CONFIGURATION`
>>
>>
>> J.
>>
>> On Thu, Jul 3, 2025 at 8:47 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > > The direction this one is taking is interesting. If you're really just
>> > trying to make the feature barely possible and mostly targeted towards
>> > managed providers to implement the rest, then I suppose this hits the
>> mark.
>> >
>> > Well actually by taking the direction I took, it's not "mostly for
>> managed
>> > providers" - i see it as it is equally, for managed providers and
>> on-prem
>> > users, but also, following the open-source spirit, philosophically, I
>> think
>> > in Airflow, any such change should be done with those things in mind,
>> > because we are at the stage where we are already "established' and by
>> > innovating on top what we have we have sometimes more to lose than to
>> gain
>> > - so I feel with "deployment' features we should be very careful to
>> > distinguish 'enabling things" vs. 'doing things". My focus with this
>> > iteration was to remove all the roadblocks that make it impossible (or
>> > extremely difficult) to implement "real" multi-team and separation
>> without
>> > modifying airflow core. I though "what is the minimal set of features
>> that
>> > will make it "possible" for someone motivated to deploy a single airflow
>> > for multiple teams.
>> >
>> > * minimise maintenance effort increase
>> > * do not "spoil" the "simple case" - we do not want to add features that
>> > make "simple" implementation more complex the current `docker run -it
>> > apache/airflow standalone` - should be simple and straightforward to run
>> > * if there is anything that involves complex deployment, we should not
>> aim
>> > to make a "turn-key" solution that we will have to support - similarly
>> like
>> > we do with our configuration parameters, we have 100s knobs to turn,
>> and as
>> > long as default settings are reasonable and someone "motivated" can
>> > configure and fine-tune - this configuration and fine-tuning should be
>> left
>> > to them - regardless if they are on-prem or managed. And both should be
>> > able to do it.
>> >
>> > I think it's not only smart technically (we support the low-level basic
>> > features and when someone puts them together and makes it more of a
>> > turn-key solution they are responsible for designing and implementing
>> it -
>> > so we have less maintenance effort. But also it's good from a simple
>> > "open-source business model point of view" - i.e. it's a smart product
>> > decision we should make.
>> >
>> > Why airflow is #18 in OSS rank - of course we have a huge community and
>> > people contributing in their free time, completely voluntarily. And we
>> > cherish, support and encourage it. But let's be honest - if not all
>> those
>> > that make business on top of airflow did not invest literally millions
>> of
>> > dollars (in terms of engineering salary, sponsoring Airflow Summit,
>> > supporting people like me (some smart stakeholders at least who
>> understand
>> > the value of it) who can be good "community spirit" - Airflow would have
>> > order of magnitude less activity, reach, Airflow 3 would not be simply
>> > possible. And this is a good thing that we have those stakeholders that
>> are
>> > interested and make money by turning Airflow into a "turn-key" solution.
>> > This is a fantastic, symbiotic relationship.
>> >
>> > So - what my thinking is - we should NOT make things that make airflow
>> > more turn-key for those complex cases. We should leave it up to those
>> who
>> > want to make it and want to charge money for it. This is cool and great
>> > that they can do - and we should not do it "for them" - but on the other
>> > hand - we should make it possible that those who want to turn airflow
>> into
>> > more complex (say multi-team solution) to make it happen - by providing
>> > them with minimal set of features that make it possible.
>> >
>> > And that also - in a way - keeps the balance between on-prem and managed
>> > implementation.
>> >
>> > Something that I've learned as a rule of thumb is that making a feature
>> > "generic" compared to custom implementation is 3x-10x more expensive
>> (both
>> > in implementation and maintenance). And it means that if an on-prem user
>> > wants to implement something for them (say turn-key multi-team solution
>> for
>> > their case) it will cost `x` , but when a managed provider wants to
>> > implement a generic multi-team it will cost `10x`. But also managed
>> > providers can spread the cost over the premium they will charge to their
>> > users so that they don't have to manage Airflow on their own and pay `x`
>> > for this mult-team feature to develop on their own. And this is a "fair"
>> > choice to make by on-prem users. They might choose what they want to do
>> > then. Also it's fair for managed provider - yes they need to invest
>> more,
>> > but also they have a chance to shine on promoting it and making it more
>> > optimised at scale etc. etc.
>> >
>> > That is my line of thinking.
>> >
>> >
>> > J.
>> >
>> >
>> > On Thu, Jul 3, 2025 at 1:41 AM Oliveira, Niko
>> <on...@amazon.com.invalid>
>> > wrote:
>> >
>> >> Hey Jarek,
>> >>
>> >>
>> >> The direction this one is taking is interesting. If you're really just
>> >> trying to make the feature barely possible and mostly targeted towards
>> >> managed providers to implement the rest, then I suppose this hits the
>> mark.
>> >>
>> >> But this is not something we're asking for at Amazon and personally I
>> >> think we should make the feature reasonably usable for those running
>> >> self-managed OSS Airflow as well. There are many users running an
>> on-prem
>> >> Airflow. Getting too hyper-fixated on an implementation that's so
>> >> simplified that it's obtuse and difficult to use by most users seems
>> like
>> >> the wrong approach to me. But you and I have already discussed this at
>> >> length and I haven't convinced you so far, so if I'm the only one with
>> this
>> >> thinking then I'm happy to disagree and commit as we say at Amazon :)
>> >>
>> >>
>> >> > So I would be rather strong on **not** touching the current
>> >> configuration and
>> >>
>> >> simply adding configuration for per-team executors in executor config -
>> >> even if it is uglier and more "low-level".
>> >>
>> >> Can you explain what "adding configuration for per-team executors in
>> >> executor config" would look like? I don't have a concrete sense of
>> what you
>> >> mean by this.
>> >>
>> >> Thanks for your efforts on trying to get this feature agreed to and
>> voted
>> >> on. Looking forward to working on the project in the coming weeks!
>> >>
>> >> Cheers,
>> >> Niko
>> >>
>> >> ________________________________
>> >> From: Jarek Potiuk <ja...@potiuk.com>
>> >> Sent: Tuesday, July 1, 2025 10:26:55 PM
>> >> To: dev@airflow.apache.org
>> >> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82
>> (External
>> >> event driven dags) exists
>> >>
>> >> CAUTION: This email originated from outside of the organization. Do not
>> >> click links or open attachments unless you can confirm the sender and
>> know
>> >> the content is safe.
>> >>
>> >>
>> >>
>> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>> externe.
>> >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
>> pouvez
>> >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
>> que
>> >> le contenu ne présente aucun risque.
>> >>
>> >>
>> >>
>> >> Any last comments ? There is a long weekend coming up in the US, so I
>> will
>> >> likely start voting on the updated AIP on Monday 7th.
>> >>
>> >> On Fri, Jun 27, 2025 at 12:41 PM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> >>
>> >> > I'd really love to finalise discussion and put it up to a vote some
>> time
>> >> > after the recording from the last dev call is posted - so that more
>> >> > context, details and the LONG discussion we had on it. There is no
>> >> *huge*
>> >> > hurry  - we have strong dependency on Task Isolation and it seems
>> that
>> >> it
>> >> > will still take a bit of time to complete, so I'd say I would love to
>> >> start
>> >> > voting in about a week time - so that maybe at the next dev call we
>> can
>> >> > "seal" the subject. Happy to see any more comments - especially from
>> >> those
>> >> > who have opinions but they had no opportunity to express them.
>> >> >
>> >> > I am personally very happy with the direction it took -
>> simplification
>> >> and
>> >> > "MVP" kind of approach - also I invite the stakeholders of ours to
>> take
>> >> a
>> >> > close look at the scope and what we really propose - I have a feeling
>> >> that
>> >> > we can balance it out - there is something we can make to make it not
>> >> > "worse" for the offerings they have. I think we have a really good
>> >> > symbiotic relationship here, and I would love to leverage that. For
>> one
>> >> -
>> >> > my goal here is to have a minimum number of changes that are
>> impacting
>> >> > maintainability of the open-source airflow - but mostly "opening up
>> some
>> >> > possibilities" - rather than provide turn-key solutions. And mostly
>> >> because
>> >> > this is good for all sides - less maintenance and complexity for OSS
>> >> > maintainers, but more opportunities to make it into "turn-key"
>> >> solutions by
>> >> > the stakeholders, while also allowing the "on-prem" users - if they
>> are
>> >> > highly motivated - to use those features by adding the "turn-key"
>> layer
>> >> on
>> >> > their own. Also adding multi-team should not be at the expense of
>> >> "simple"
>> >> > installations - they should be virtually unaffected.
>> >> >
>> >> > One example of applying this is cutting on "separate config files". I
>> >> > think it moves us closer to a "turn-key" solution but it is not
>> really
>> >> > necessary to achieve the three goals above - that's why in the
>> current
>> >> > proposal this part is completely removed - Sorry Niko, but I still
>> think
>> >> > it's one of the things that falls into this bucket. We can easily
>> remove
>> >> > it, they complicate code, documentation and options the users have,
>> and
>> >> > even if it is a "little" more complex to manage configuration by
>> >> motivated
>> >> > users, it's also an opportunity for "turn-key" option that
>> stakeholders
>> >> can
>> >> > build in their products - and we do not have to maintain it in the
>> >> > open-source. So I would be rather strong on **not** touching the
>> current
>> >> > configuration and simply adding configuration for per-team executors
>> in
>> >> > executor config - even if it is uglier and more "low-level".
>> >> >
>> >> > So if there are some constructive ideas on what can be done to make
>> it
>> >> > "simpler" and less "turn-key" in that respect - I would highly value
>> >> such
>> >> > ideas and comments. If we can cut down something more that is not
>> >> > "necessary" for the three primary goals I came up with - I am more
>> than
>> >> > happy to do it.
>> >> >
>> >> > Just to remind - those are the "extracted" goals. I slightly updated
>> >> them
>> >> > and added to the preamble of the AIP:
>> >> >
>> >> > * less operational overhead for managing multi-team (once AIP-72 is
>> >> > complete) where separate execution environments are important
>> >> > * virtual assets sharing between teams
>> >> > * ability of having "admin" and "team sharing" capability where dags
>> >> from
>> >> > multiple teams can be seen in a single Airflow UI  (requires custom
>> >> RBAC an
>> >> > AIP-56 implementation of Auth Manager - with KeyCloak Auth Manager
>> >> being a
>> >> > reference implementation)
>> >> >
>> >> > J.
>> >> >
>> >> >
>> >> > On Thu, Jun 26, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> >> >
>> >> >>
>> >> >>> One technical observation: Now that the dag table no longer has a
>> >> >>> team_id in it, what would the behaviour be when a DAG is attempted
>> to
>> >> move
>> >> >>> between bundles? How do we detect this? (I’m not all convinced
>> that we
>> >> >>> correctly detect duplicate dag ids across bundles today, so I
>> wouldn’t
>> >> >>> assume or rely on the current behaviour.)
>> >> >>>
>> >> >>
>> >> >> Of course - yes, I realise that - that problem was also not handled
>> in
>> >> >> the previous iteration to be honest. That is something that dag
>> bundle
>> >> >> solution allows to solve eventually - but I do not think it's a
>> >> blocker for
>> >> >> the proposed implementation. We will have to eventually add some
>> way of
>> >> >> blocking dags to jump between bundles, we might tackle this
>> >> separately. I
>> >> >> already wanted to propose a separate update to that - but I did not
>> >> want to
>> >> >> complicate the current proposal. One thing at a time. I can,
>> however -
>> >> if
>> >> >> you consider that as a blocker, extend the current AIP with it. Not
>> a
>> >> big
>> >> >> problem. This is however a bit independent from the team_id
>> >> introduction.
>> >> >>
>> >> >> Overall, I am still unconvinced this proposal has enough real user
>> >> >>> benefit over actually separate deployments, and on balance of the
>> >> added
>> >> >>> complexity and maintenance burden I do not think it is worth it.
>> >> >>>
>> >> >>
>> >> >> That makes me sad, I thought that over the course of the discussion
>> I
>> >> >> addressed all the concerns (in this case the concern was "is it
>> worth
>> >> with
>> >> >> the cost and little benefit", but when I did it and heavily limited
>> the
>> >> >> impact, now the concern is "is it worth at all as changes are really
>> >> >> minimal" - and surely, anyone can change and adapt their concerns,
>> over
>> >> >> time,  but that one seems like ever-moving target. I hoped at least
>> for
>> >> >> some acknowledgment of some concerns (complexity in this case) is
>> >> >> addressed, but it seems that you are deeply convinced that we do not
>> >> need
>> >> >> multi-team at all (which is in stark contrast with at least a dozen
>> of
>> >> >> bigger and smaller users of Airflow who submitted talks to Airflow
>> >> summit
>> >> >> (including about 5 or 6 submissions for Airflow 2025) on how they
>> spent
>> >> >> their engineering effort, time and money on trying to achieve
>> something
>> >> >> similar - they assessed that it's worth, you  assess that it's not
>> >> worth.
>> >> >> Somehow I trust our users that they were not spending the money,
>> time
>> >> and
>> >> >> engineering effort to achieve this because they wanted to spend more
>> >> money.
>> >> >> I think they assessed it's worth it. So I want to make it a bit
>> easier
>> >> and
>> >> >> more "proper" way for them to do that.
>> >> >>
>> >> >>>
>> >> >>> Upgrades: it is not easier to upgrade under this multi team
>> proposal,
>> >> >>> but much much harder. This is based on hard earned experience from
>> >> helping
>> >> >>> Astronomer users — having to coordinate upgrades between multiple
>> >> teams
>> >> >>> turns in to a months long slog of the hardest kind of work —
>> people
>> >> work:
>> >> >>> getting other teams to agree to do things that they don’t directly
>> >> care
>> >> >>> about — “It’s working for me, I don’t care about upgrading, we’ll
>> get
>> >> to it
>> >> >>> next quarter” is a refrain I’ve heard many times.
>> >> >>>
>> >> >>
>> >> >> Yes. absolutely - this is why we deferred it until we knew what
>> shape
>> >> >> task isolation and other AIPs we depend on take on. Because it is
>> clear
>> >> >> that pretty much all the problem you explain above are going to be
>> >> solved
>> >> >> with task isolation. And it's not just my opinion. If you want to
>> argue
>> >> >> with it, you likely need to argue with yourself:
>> >> >>
>> https://github.com/apache/airflow/issues/51545#issuecomment-2980038478
>> >> .
>> >> >> Let me quote what you wrote there last week:
>> >> >>
>> >> >> Ash Berlin Taylor wrote:
>> >> >>
>> >> >> > A tight coupling between task-sdk and any "server side" component
>> is
>> >> >> the opposite to one of the goals of AIP-72 (I'm not sure we ever
>> >> explicitly
>> >> >> said this, but the first point of motivation for the AIP says
>> >> "Dependency
>> >> >> conflicts for administrators supporting data teams using different
>> >> versions
>> >> >> of providers, libraries, or python packages")
>> >> >> > In short, my goal with TaskSDK, and the reason for introducing
>> CalVer
>> >> >> and Cadwyn with the execution API is to end up in a world where you
>> can
>> >> >> upgrade the Airflow Scheduler/API server interdependently of any
>> worker
>> >> >> nodes (with the exception that the server must be at least as new as
>> >> the
>> >> >> clients)
>> >> >> > This ability to have version-skew is pretty much non-negotiable
>> to me
>> >> >> and is (other than other languages) one of primary benefits of
>> AIP-72
>> >> >>
>> >> >> If you read yourself from that quote it basically means "it will be
>> >> easy
>> >> >> to upgrade airflow independently of workers". So I am a bit confused
>> >> here.
>> >> >> Yes, I agree it was difficult, but you yourself explain that when
>> >> AIP-72
>> >> >> (which since API-67 has been accepted has always beem prerequisite
>> of
>> >> it)
>> >> >> wrote it will be "easy". So I am not sure why you are bringing it
>> now.
>> >> We
>> >> >> assume AIP-72 will be completed and this problem will be gone. Let's
>> >> not
>> >> >> mention it any more please.
>> >> >>
>> >> >> The true separation from TaskSDK will likely only land in about 3.2
>> >> time
>> >> >>> frame. We are actively working on it, but it’s a slow process of
>> >> untangling
>> >> >>> lots of assumptions made in the code base over the years. Maybe
>> once
>> >> we
>> >> >>> have that my view would be different, but right now I think this
>> >> makes the
>> >> >>> proposal a non-starter. Especially as you are saying that most
>> teams
>> >> will
>> >> >>> have unique connections. If they’ve got those already, then having
>> an
>> >> asset
>> >> >>> trigger use those conns to watch/poll for activity is a much easier
>> >> >>> solution to operate and crucially, to scale and upgrade.
>> >> >>>
>> >> >>
>> >> >> Yes. I perfectly understand that and I am fully aware of potentially
>> >> 3.2
>> >> >> time-frame. And that's fine. Actually I heartily invite you to
>> listen
>> >> to
>> >> >> the part of my talk from Berlin Buzzwords when I was asked for the
>> >> timeline
>> >> >> - https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the
>> exact
>> >> >> timeline in my talk . My answer was basically - "3.1" or "3.2", and
>> I
>> >> >> sincerely hope "3.1" but we might not be able to complete it
>> because we
>> >> >> have other things to do (other - is indeed the Task Isolation work
>> >> that you
>> >> >> are leading). And that's perfectly fine. And it absolutely does not
>> >> prevent
>> >> >> us from voting on the AIP now - similarly as we voted on the
>> previous
>> >> >> version of the AIP - knowing that it has some prerequisites a few
>> >> months
>> >> >> ago. Especially that we know that the feature we need from task
>> >> isolation
>> >> >> is "non-negotiable". I.e. it WILL happen. We don't hope for it, we
>> >> know it
>> >> >> will be there. Those are your own words.
>> >> >>
>> >> >>
>> >> >>> >  I think we can’t compare AIP-82 to sharing virtual assets due to
>> >> >>> complexity of it.
>> >> >>>
>> >> >>> Virtual Assets was a mistake, and not how users actually want to
>> use
>> >> >>> them. Mea culpa
>> >> >>>
>> >> >>
>> >> >> This is the first time I hear this - certainly you never raised this
>> >> >> concern on the devlist. So if you have some concerns about virtual
>> >> assets I
>> >> >> think you should raise it on the devlist, because I think everyone
>> >> here is
>> >> >> missing some conversation (or maybe it's just your private opinion
>> that
>> >> >> you never shared with anyone, but maybe it's worth). I would be
>> >> >> interested to hear how the feature that was absolutely most
>> successful
>> >> >> feature of airflow 2 was a mistake. According to the 2024 survey
>> >> >> https://airflow.apache.org/blog/airflow-survey-2024/  - 48% of
>> Airflow
>> >> >> users have been using it, even if it was added as one of the last
>> >> >> big features of Airflow 2. It's the MOST used feature out of all the
>> >> >> features out there. I would be really curious to see how it was a
>> >> mistake
>> >> >> (but  please start a separate thread explaining why you think it
>> was a
>> >> >> mistake, what are your data points and what do you think should be
>> >> fixed.
>> >> >> Just dropping "virtual assets were a mistake" in the middle of
>> >> multi-team
>> >> >> conversation seems completely unjustified without knowing what you
>> are
>> >> >> talking about. So I think, until we know more, this argument has no
>> >> base.
>> >> >>
>> >> >>
>> >> >>>
>> >> >>> S
>> >> >>> To restate my points:
>> >> >>>
>> >> >>> - Sharing a deployment between teams today/in 3.1 is operationally
>> >> more
>> >> >>> complex (both scaling, and upgrades) — this is a con, not a plus.
>> >> >>>
>> >> >>
>> >> >> Surely. But it will be easier when AIP-72 is complete (which I am
>> >> >> definitely looking forward to and as clearly explained in AIP-82,
>> is a
>> >> >> prerequisite of it). Nothing changed here.
>> >> >>
>> >> >>
>> >> >>> - The main user benefit appears to be “allow teams’ DAGs to
>> >> communicate
>> >> >>> via Assets”, in which case we can do that today by putting more
>> work
>> >> in to
>> >> >>> AIP-82’s Asset triggers
>> >> >>>
>> >> >>
>> >> >> No. Lower operational complexity for multi-teams (providing that we
>> >> >> deliver AIP-72) is another benefit. Virtual assets is another, and
>> >> since
>> >> >> there is no ground in "virtual assets is a mistake" statement (not
>> >> until
>> >> >> you explain what you mean by that in a separate discussion) - this
>> is
>> >> also
>> >> >> still a very valid point.
>> >> >>
>> >> >>
>> >> >>> Soon, we will have then be asked about cross-team governance,
>> policy
>> >> >>> enforcement, and potentially unbounded edge cases (e.g.,
>> team-specific
>> >> >>> secrets, roles, quotas). ain, you get this for free with truely
>> >> separate
>> >> >>> deployments already
>> >> >>> allow different teams to use different executors (including
>> multiple
>> >> >>> executors per-team following AIP-61)
>> >> >>>
>> >> >>
>> >> >> Not really. We very explicitly say in the AIP that his is not a goal
>> >> and
>> >> >> that we have no plans for. And yes, using separate executors per
>> team
>> >> is
>> >> >> actually back in the AIP-82 in case you did not notice (and the code
>> >> needed
>> >> >> for it's even implemented and merged already in main by Vincent).
>> >> >>
>> >> >>
>> >> >>> Provably not true right now, and until ~3.2 delivers the full Task
>> >> >>> SDK/Core dependency separation this would be _more_ work to
>> upgrade,
>> >> not
>> >> >>> less, and that work is not shared but still on a central team.
>> >> >>>
>> >> >>
>> >> >> Absolutely - we will wait for AIP-72 completion. I do not want to
>> say
>> >> 3.1
>> >> >> or 3.2 directly - because there are - as you said - a lot of moving
>> >> pieces.
>> >> >> So my target for multi-team is "After AIP-72 is completed". Full
>> stop.
>> >> But
>> >> >> there is nothing wrong with accepting the AIP now and doing
>> preparatory
>> >> >> work in parallel. Similarly as there is no way to have a baby in 1
>> >> month by
>> >> >> 9 women, there is no way adding more effort to task-sdk isolation
>> will
>> >> >> speed it up - we alredy have not only 3 people (you leading it,
>> Kaxil
>> >> and
>> >> >> Amog) but also all the help from me and even 10s of different
>> >> contributors
>> >> >> (for example with the recent db_test cleanup that I took leadership
>> >> on) -
>> >> >> and there are people who wish to work on adding multi-team features.
>> >> Since
>> >> >> the design heavily limits impact on airflow codebase and
>> interactions
>> >> with
>> >> >> task-sdk implementation, there is nothing wrong with starting
>> >> >> implementation in parallel either- amazon team is keen to move it
>> >> forward -
>> >> >> they even already implemented SQS trigger for assets, and we are
>> >> working
>> >> >> together on FAB removal, Keycloak authentication manager - and they
>> >> seem to
>> >> >> still have capacity and drive to progress multi-team. So I am not
>> sure
>> >> if
>> >> >> we are trading off something. There is no "if we work on more on
>> task
>> >> sdk
>> >> >> and drop multi-team things will be faster". Generally in open source
>> >> people
>> >> >> work in the area where they feel they can provide best value - such
>> as
>> >> you
>> >> >> working on task-sdk, me on CI,dev env, they will deliver more value
>> on
>> >> >> multi-team
>> >> >>
>> >> >>
>> >> >>>
>> >> >>> So please, as succinctly as possible, please tell me what the
>> direct
>> >> >>> benefit to users this proposal is over us putting this effort in to
>> >> writing
>> >> >>> better Asset triggers instead?
>> >> >>>
>> >> >>
>> >> >>
>> >> >> * less operational overhead for managing multi-team (once AIP-72 is
>> >> >> complete) where separate execution environments are important
>> >> >> * virtual assets sharing
>> >> >> * ability of having "admin" and "team sharing" capability where dags
>> >> from
>> >> >> multiple teams can be seen in a single Airflow UI  (requires custom
>> >> RBAC)
>> >> >>
>> >> >> None of this can be done via beter asset triggers
>> >> >>
>> >> >>
>> >> >>>
>> >> >>> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >> >>> >
>> >> >>> > My counter-points:
>> >> >>> >
>> >> >>> >
>> >> >>> >> 1. Managing a multi team deployment is not materially different
>> >> from
>> >> >>> >> managing a deployment per team
>> >> >>> >>
>> >> >>> >
>> >> >>> > It's a bit easier - especially when it comes to upgrades
>> (especially
>> >> >>> in the
>> >> >>> > case we are targetting when we are not targetting multi-tenant,
>> but
>> >> >>> several
>> >> >>> > relatively closely cooperating teams with different dependncy
>> >> >>> requiremens
>> >> >>> > and isolation need.
>> >> >>> >
>> >> >>> > 2. The database changes were quite wide-reaching
>> >> >>> >>
>> >> >>> >
>> >> >>> > Yes. that is addressed.
>> >> >>> >
>> >> >>> >
>> >> >>> >> 3. I don’t believe the original AIP (again, I haven’t read the
>> >> updated
>> >> >>> >> proposal or recent messages on the thread. yet) will meet what
>> many
>> >> >>> users
>> >> >>> >> want out of a multiteam solution
>> >> >>> >>
>> >> >>> >
>> >> >>> > I think we will only see when we try. A lot of people thing they
>> >> would,
>> >> >>> > even if they are warned. I know at least one user (Wealthsimple)
>> who
>> >> >>> > definitely want to use it and they got a very detailed
>> explanation
>> >> of
>> >> >>> the
>> >> >>> > idea and understand it well. So I am sure that **some** users
>> would.
>> >> >>> But we
>> >> >>> > do not know how many.
>> >> >>> >
>> >> >>> >
>> >> >>> >> To expand on those points a bit more
>> >> >>> >>
>> >> >>> >> On 1. The only components that are shared are, I think, the
>> >> scheduler
>> >> >>> and
>> >> >>> >> the API server, and it’s arguable if that is actually a good
>> idea
>> >> >>> given
>> >> >>> >> those are likely to be the most performance sensitive components
>> >> >>> anyway.
>> >> >>> >>
>> >> >>> >> Additionally the fact that the scheduler is a shared component
>> >> makes
>> >> >>> >> upgrading it almost a non starter as you would likely need
>> buy-in,
>> >> >>> changes,
>> >> >>> >> and testing form ALL teams using it. I’d argue that this is a
>> huge
>> >> >>> negative
>> >> >>> >> until we finish off the version indepence work of AIP-72.
>> >> >>> >>
>> >> >>> >
>> >> >>> > Quite disagree here - especially that our target is that
>> task-sdk is
>> >> >>> > supposed to provide all isolation that is needed. There should
>> be 0
>> >> >>> changes
>> >> >>> > in the dags needed to upgrade scheduler, api_server, triggerer -
>> >> >>> precisely
>> >> >>> > because we introduced backwards-compatible task-sdk.
>> >> >>> >
>> >> >>> > On 3 my complaint is essentially that this doesn’t go nearly far
>> >> >>> enough. It
>> >> >>> >> doesn’t allow read only views to other teams dags. I don’t
>> think it
>> >> >>> allows
>> >> >>> >> you to be in multiple teams at once. You can’t share a
>> connection
>> >> >>> between
>> >> >>> >> teams but only allow certain specified dags to access it, but
>> would
>> >> >>> have to
>> >> >>> >> either be globally usable, or duplicated-and-kept-in-sync
>> between
>> >> >>> teams. In
>> >> >>> >> short I think it fall short of being useful..
>> >> >>> >>
>> >> >>> >
>> >> >>> > Oh absolutely all that is possible (except sharing single
>> >> connections
>> >> >>> > between multiple teams - which is a very niche use cases and
>> >> >>> duplication
>> >> >>> > here is perfectly ok as first approximation - and if we need
>> more we
>> >> >>> can
>> >> >>> > add it later).
>> >> >>> >
>> >> >>> > Auth manager RBAC and access is abstracted away, and the Keyclock
>> >> >>> Manager
>> >> >>> > implemented by Vincent allows to manage completely independent
>> and
>> >> >>> separate
>> >> >>> > RBAC based on arguments and resources provided by Airflow. There
>> is
>> >> >>> nothing
>> >> >>> > to prevent the user who configures KeyCloak RBAC to define it in
>> the
>> >> >>> way:
>> >> >>> >
>> >> >>> > if group a > allow to read a and write b
>> >> >>> > if group b > alllow to write b but not a
>> >> >>> >
>> >> >>> > and any other combinations. KeyCloak implementation - pretty
>> >> advanced
>> >> >>> > already - (and design of auth manager) completely abstracts away
>> >> both
>> >> >>> > authentication and authorization to KeyCloak and KeyCloak has
>> RBAC
>> >> >>> > management built in. Also any of the users can write their own -
>> >> even
>> >> >>> > hard-coded authentication manager to do the same if they do not
>> >> want to
>> >> >>> > have configurable KeyCloak. Even SimpleAuthManager could be
>> >> hard-coded
>> >> >>> to
>> >> >>> > provide thiose features.
>> >> >>> >
>> >> >>> >
>> >> >>> >>
>> >> >>> >> So on the surface, I’m no more in favour of using dag bundle as
>> a
>> >> >>> >> replacement for team id as I think most of the above points
>> still
>> >> >>> stand.
>> >> >>> >>
>> >> >>> >
>> >> >>> > We disagree here.
>> >> >>> >
>> >> >>> >>
>> >> >>> >> My counter proposal: We do _nothing_ to core airflow. We work on
>> >> >>> improving
>> >> >>> >> the event-based trigger o fdags (write more triggers for
>> read/check
>> >> >>> remote
>> >> >>> >> Assets etc) so that teams can have 100% isolated deployments but
>> >> still
>> >> >>> >> trigger dags based on asset events from other teams.
>> >> >>> >>
>> >> >>> >
>> >> >>> > That does not solve any of the other design goals - only allows
>> to
>> >> >>> trigger
>> >> >>> > assets a bit more easily (but also it's not entirely solved by
>> >> AIP-82
>> >> >>> > because it does not solve virtual assets - only ones that have
>> >> defined
>> >> >>> > triggerer and "something" to listen on - which is way more
>> complex
>> >> than
>> >> >>> > just defining asset in a Dag and using it in another). I think we
>> >> can't
>> >> >>> > compare AIP-82 to sharing virtual assets due to complexity of
>> it. I
>> >> >>> > explained it in the doc.
>> >> >>> >
>> >> >>> >
>> >> >>> > I will now go and catch up with the long thread and updated
>> proposal
>> >> >>> and
>> >> >>> >> come back.
>> >> >>> >>
>> >> >>> >
>> >> >>> > Please. I hope the above explaination will help in better
>> >> >>> understanding of
>> >> >>> > the proposal, because I think you had some assumptions that do
>> not
>> >> >>> hold any
>> >> >>> > more with the new proposal.
>> >> >>> >
>> >> >>> > J.
>> >> >>> >
>> >> >>> >
>> >> >>> >>
>> >> >>> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> >> >>> >>>
>> >> >>> >>> Just to clarify the relation - I updated the AIP now to refer
>> to
>> >> >>> AIP-82
>> >> >>> >> and
>> >> >>> >>> to explain relation between the "cross-team" and
>> "cross-airflow"
>> >> >>> asset
>> >> >>> >>> triggering - this is what I added:
>> >> >>> >>>
>> >> >>> >>> Note that there is a relation between AIP-82 ("External Driven
>> >> >>> >> Scheduling")
>> >> >>> >>> and this part of the functionality. When you have multiple
>> >> instances
>> >> >>> of
>> >> >>> >>> Airflow, you can use shared datasets - "Physical datasets" -
>> that
>> >> >>> several
>> >> >>> >>> Airflow Instances can use - for example there could be an S3
>> >> object
>> >> >>> that
>> >> >>> >> is
>> >> >>> >>> produced by one airflow instance, and consumed by another. That
>> >> >>> requires
>> >> >>> >>> deferred trigger to monitor for such datasets, and appropriate
>> >> >>> >> permissions
>> >> >>> >>> to the external dataset, and you could achive similar result to
>> >> >>> >> cross-team
>> >> >>> >>> dataset triggering (but cross airflow). However the feature of
>> >> >>> sharing
>> >> >>> >>> datasets between the teams also works for virtual assets, that
>> do
>> >> not
>> >> >>> >> have
>> >> >>> >>> physically shared "objects" and trigger that is monitoring for
>> >> >>> changes in
>> >> >>> >>> such asset.
>> >> >>> >>>
>> >> >>> >>> J.
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <jarek@potiuk.com
>> >
>> >> >>> wrote:
>> >> >>> >>>
>> >> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>> >> >>> reference
>> >> >>> >> to
>> >> >>> >>>>> AIP-82, which surprised me, but will take a more detailed
>> read
>> >> >>> through.
>> >> >>> >>>>
>> >> >>> >>>> Yep. It did not - because I did not think it was needed or
>> even
>> >> very
>> >> >>> >>>> important after the simplifications. AIP-82 has a different
>> >> scope,
>> >> >>> >> really.
>> >> >>> >>>> It only helps when the Assets are "real" data files which we
>> have
>> >> >>> >> physical
>> >> >>> >>>> triggers for, it's slightly related - sharing datasets between
>> >> teams
>> >> >>> >>>> (including those that do not require physical files and
>> >> triggers) is
>> >> >>> >> still
>> >> >>> >>>> possible in the design we have now, but it's not (and never
>> was)
>> >> the
>> >> >>> >>>> **only** reason for having multi-team. There always was (and
>> >> still
>> >> >>> is)
>> >> >>> >> the
>> >> >>> >>>> possibility of having a common, distinct environments (i.e.
>> >> >>> dependencies
>> >> >>> >>>> and providers) per team, the possibility of having connections
>> >> and
>> >> >>> >>>> variables that are only accessible to one team and not the
>> other,
>> >> >>> and
>> >> >>> >>>> isolating workload execution (all that while allowing to
>> manage
>> >> >>> multiple
>> >> >>> >>>> team and schedule things with single deployment). That did not
>> >> >>> change.
>> >> >>> >> What
>> >> >>> >>>> changed a lot is that it is now way simpler, something that we
>> >> can
>> >> >>> >>>> implement without heavy changes to the codebase - and give it
>> to
>> >> our
>> >> >>> >> users,
>> >> >>> >>>> so that they can assess if this is something they need without
>> >> too
>> >> >>> much
>> >> >>> >>>> risk and effort.
>> >> >>> >>>>
>> >> >>> >>>> This was - I believe the main concern, that the value we get
>> from
>> >> >>> it is
>> >> >>> >>>> not dramatic, but the required changes are huge. This
>> "redesign"
>> >> >>> changes
>> >> >>> >>>> the equation - the value is still unchanged, but the cost of
>> >> >>> >> implementing
>> >> >>> >>>> it and impact on the Airflow codebase is much smaller. I still
>> >> have
>> >> >>> not
>> >> >>> >>>> heard back from Ash if my proposal responds to his original
>> >> concern
>> >> >>> >> though,
>> >> >>> >>>> so I am mostly guessing (also based on the positive impact of
>> >> >>> others)
>> >> >>> >> that
>> >> >>> >>>> yes it does. But to be honest I am not sure and I would love
>> to
>> >> hear
>> >> >>> >> back,
>> >> >>> >>>> I decided to update the AIP to reflect it - regardless,
>> because I
>> >> >>> think
>> >> >>> >> the
>> >> >>> >>>> simplification I proposed keeps the original goals, but is
>> indeed
>> >> >>> way
>> >> >>> >>>> simpler.
>> >> >>> >>>>
>> >> >>> >>>>> This is a very difficult thread to catch up on.
>> >> >>> >>>>
>> >> >>> >>>> Valid point. Let me summarize what is the result:
>> >> >>> >>>>
>> >> >>> >>>> * I significantly simplified the implementation proposal
>> >> comparing
>> >> >>> to
>> >> >>> >> the
>> >> >>> >>>> original version
>> >> >>> >>>> * main simplification is very limited impact on existing
>> >> database -
>> >> >>> >>>> without "ripple effect" that would require us to change a lot
>> of
>> >> >>> tables,
>> >> >>> >>>> including their primary keys, and heavily impact the UI
>> >> >>> >>>> * this is now more of an incremental change that can be
>> >> implemented
>> >> >>> way
>> >> >>> >>>> faster and with far less risk
>> >> >>> >>>> * updated idea is based on leveraging bundles (already part of
>> >> our
>> >> >>> data
>> >> >>> >>>> model) to map them (many-to-one) to a team - which requires to
>> >> just
>> >> >>> >> extend
>> >> >>> >>>> the data model with bundle mapping and add team_id to
>> connections
>> >> >>> and
>> >> >>> >>>> variables. Those are all needed DB changes.
>> >> >>> >>>>
>> >> >>> >>>> The AIP is updated - in a one single big change so It should
>> be
>> >> >>> easy to
>> >> >>> >>>> compare the changes:
>> >> >>> >>>>
>> >> >>> >>
>> >> >>>
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
>> >> >>> >>>> -> I even named the version appropriately "Simplified
>> multi-team
>> >> >>> AIP" -
>> >> >>> >> you
>> >> >>> >>>> can select and compare v.65 with v.66 to see the exact
>> >> differences I
>> >> >>> >>>> proposed.
>> >> >>> >>>>
>> >> >>> >>>> I hope it will be helpful to catch up and for those who did
>> not
>> >> >>> follow,
>> >> >>> >> to
>> >> >>> >>>> be able to make up their minds about it.
>> >> >>> >>>>
>> >> >>> >>>> J.
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
>> >> >>> >> <vi...@astronomer.io.invalid>
>> >> >>> >>>> wrote:
>> >> >>> >>>>
>> >> >>> >>>>> This is a very difficult thread to catch up on.
>> >> >>> >>>>> I will take a detailed look at the AIP update to try to
>> figure
>> >> out
>> >> >>> the
>> >> >>> >>>>> changes in the proposal.
>> >> >>> >>>>>
>> >> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>> >> >>> reference
>> >> >>> >> to
>> >> >>> >>>>> AIP-82, which surprised me, but will take a more detailed
>> read
>> >> >>> through.
>> >> >>> >>>>>
>> >> >>> >>>>> Vikram
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>> >> >>> >>>>> gopidesupavan@gmail.com>
>> >> >>> >>>>> wrote:
>> >> >>> >>>>>
>> >> >>> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's
>> much
>> >> >>> more
>> >> >>> >> slim
>> >> >>> >>>>>> down.
>> >> >>> >>>>>>
>> >> >>> >>>>>> left a minor comment. :) Overall looking great.
>> >> >>> >>>>>>
>> >> >>> >>>>>> Pavan
>> >> >>> >>>>>>
>> >> >>> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>> >> >>> >>>>> <j_scheffler@gmx.de.invalid
>> >> >>> >>>>>>>
>> >> >>> >>>>>> wrote:
>> >> >>> >>>>>>
>> >> >>> >>>>>>> Thanks for the rework/update of the AIP-72!
>> >> >>> >>>>>>>
>> >> >>> >>>>>>> Just a few small comments but overall I like it as it is
>> much
>> >> >>> leaner
>> >> >>> >>>>>>> than originally planned and is in a level of complexity
>> that
>> >> it
>> >> >>> >> really
>> >> >>> >>>>>>> seems to be a benefit to close the gap as described.
>> >> >>> >>>>>>>
>> >> >>> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
>> >> >>> >>>>>>>> I updated the AIP - including architecture images and
>> >> reviewed
>> >> >>> it
>> >> >>> >>>>>> (again)
>> >> >>> >>>>>>>> and corrected any ambiguities and places where it needed
>> to
>> >> be
>> >> >>> >>>>> changed.
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>> I think the current state
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>
>> >> >>> >>>>>>
>> >> >>> >>>>>
>> >> >>> >>
>> >> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>> >> >>> >>>>>>>> - nicely describes the proposal.
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>> Comparing to the previous one:
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>> 1. The DB changes are far less intrusive - no ripple
>> effect
>> >> on
>> >> >>> >>>>> Airflow
>> >> >>> >>>>>>>> 2. There is no need to merge configurations and provide
>> >> >>> different
>> >> >>> >>>>> set
>> >> >>> >>>>>> of
>> >> >>> >>>>>>>> configs per team - we can add it later but I do not see
>> why
>> >> we
>> >> >>> need
>> >> >>> >>>>> it
>> >> >>> >>>>>> in
>> >> >>> >>>>>>>> this simplified version
>> >> >>> >>>>>>>> 3. We can still configure a different set of executors per
>> >> team
>> >> >>> -
>> >> >>> >>>>> that
>> >> >>> >>>>>> is
>> >> >>> >>>>>>>> already implemented (we just need to wire it to the
>> bundle ->
>> >> >>> team
>> >> >>> >>>>>>> mapping).
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>> I think it will be way simpler and faster to implement
>> this
>> >> way
>> >> >>> and
>> >> >>> >>>>> it
>> >> >>> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we
>> can
>> >> >>> give
>> >> >>> >>>>> our
>> >> >>> >>>>>>>> users so that they can provide feedback.
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>> J.
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <
>> >> jarek@potiuk.com>
>> >> >>> >>>>> wrote:
>> >> >>> >>>>>>>>
>> >> >>> >>>>>>>>>
>> >> >>> >>>>>>>>>
>> >> >>> >>>>>>>>>> I like this iteration a bit more now for sure, thanks
>> for
>> >> >>> being
>> >> >>> >>>>>>> receptive
>> >> >>> >>>>>>>>>> to feedback! :)
>> >> >>> >>>>>>>>>>
>> >> >>> >>>>>>>>>
>> >> >>> >>>>>>>>>> This now becomes quite close to what was proposing
>> before,
>> >> we
>> >> >>> now
>> >> >>> >>>>>> again
>> >> >>> >>>>>>>>>> have a team ID (which I think is really needed here,
>> glad
>> >> to
>> >> >>> see
>> >> >>> >>>>> it
>> >> >>> >>>>>>> back)
>> >> >>> >>>>>>>>>> and it will be used for auth management, configuration
>> >> >>> >>>>> specification,
>> >> >>> >>>>>>> etc
>> >> >>> >>>>>>>>>> but will be carried by Bundle instead of the dag model.
>> >> Which
>> >> >>> as
>> >> >>> >>>>> you
>> >> >>> >>>>>>> say
>> >> >>> >>>>>>>>>> “For that we will need to make sure that both
>> api-server,
>> >> >>> >>>>> scheduler
>> >> >>> >>>>>> and
>> >> >>> >>>>>>>>>> triggerer have access to the "bundle definition" (to
>> >> perform
>> >> >>> the
>> >> >>> >>>>>>> mapping)"
>> >> >>> >>>>>>>>>> which honestly doesn’t feel too much different from the
>> >> >>> original
>> >> >>> >>>>>>> proposal
>> >> >>> >>>>>>>>>> we had last week of adding it to Dag table and ensuring
>> >> it’s
>> >> >>> >>>>>> available
>> >> >>> >>>>>>>>>> everywhere. but either way I’m happy to meet in the
>> middle
>> >> and
>> >> >>> >>>>> keep
>> >> >>> >>>>>> it
>> >> >>> >>>>>>> on
>> >> >>> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable
>> >> location.
>> >> >>> >>>>>>>>>>
>> >> >>> >>>>>>>>> I think the big difference is the "ripple effect" that
>> was
>> >> >>> >>>>> discussed
>> >> >>> >>>>>> in
>> >> >>> >>>>>>>>>
>> >> >>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>> >> >>> >>>>>> (and I
>> >> >>> >>>>>>>>> believe - correct me if I am wrong Ash - important
>> trigger
>> >> for
>> >> >>> the
>> >> >>> >>>>>>>>> discussion) so far what we wanted is to extend the
>> primary
>> >> key
>> >> >>> and
>> >> >>> >>>>> it
>> >> >>> >>>>>>> would
>> >> >>> >>>>>>>>> ripple through all the pieces of Airflow -> models, API,
>> UI
>> >> >>> etc.
>> >> >>> >>>>> ...
>> >> >>> >>>>>>>>> However - we already have `bundle_name" and
>> "bundle_version"
>> >> >>> in the
>> >> >>> >>>>>> Dag
>> >> >>> >>>>>>>>> model. So I think when we add a separate table where we
>> map
>> >> the
>> >> >>> >>>>> bundle
>> >> >>> >>>>>>> to
>> >> >>> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not
>> >> want
>> >> >>> to
>> >> >>> >>>>>> change
>> >> >>> >>>>>>>>> primary key, we do not want to change UI in any way
>> (except
>> >> >>> >>>>> filtering
>> >> >>> >>>>>> of
>> >> >>> >>>>>>>>> DAGs available based on your team - but that will be
>> >> handled in
>> >> >>> >>>>> Auth
>> >> >>> >>>>>>>>> Manager and will not impact UI in any way, I think
>> that's a
>> >> >>> huge
>> >> >>> >>>>>>>>> simplification of the implementation, and if we agree to
>> it
>> >> - i
>> >> >>> >>>>> think
>> >> >>> >>>>>> it
>> >> >>> >>>>>>>>> should speed up the implementation significantly. There
>> are
>> >> >>> only a
>> >> >>> >>>>>>> limited
>> >> >>> >>>>>>>>> number of times where you need to look up the team_id -
>> so
>> >> >>> having
>> >> >>> >>>>> the
>> >> >>> >>>>>>>>> bundle -> team mapping in a separate table and having to
>> >> look
>> >> >>> them
>> >> >>> >>>>> up
>> >> >>> >>>>>>>>> should not be a problem. And it has much less complexity
>> and
>> >> >>> >>>>>>>>> "ripple-effect" through the codebase (for example I could
>> >> >>> imagine
>> >> >>> >>>>> 100s
>> >> >>> >>>>>>> or
>> >> >>> >>>>>>>>> thousands already written tests that would have to be
>> >> adapted
>> >> >>> if we
>> >> >>> >>>>>>> changed
>> >> >>> >>>>>>>>> the primary key - where there will be pretty much zero
>> >> impact
>> >> >>> on
>> >> >>> >>>>>>> existing
>> >> >>> >>>>>>>>> tests if we just add bundle -> team lookup table.
>> >> >>> >>>>>>>>>
>> >> >>> >>>>>>>>>
>> >> >>> >>>>>>>>>> One other thing I’d point out is that I think including
>> >> >>> executors
>> >> >>> >>>>> per
>> >> >>> >>>>>>>>>> team is a very easy win and quite possible without much
>> >> work.
>> >> >>> I
>> >> >>> >>>>>> already
>> >> >>> >>>>>>>>>> have much of the code written. Executors are already
>> aware
>> >> of
>> >> >>> >>>>> Teams
>> >> >>> >>>>>>> that
>> >> >>> >>>>>>>>>> own them (merged), I have a PR open to have
>> configuration
>> >> per
>> >> >>> team
>> >> >>> >>>>>>> (with a
>> >> >>> >>>>>>>>>> quite simple and isolated approach, I believe you
>> approved
>> >> >>> Jarek).
>> >> >>> >>>>>> The
>> >> >>> >>>>>>> last
>> >> >>> >>>>>>>>>> piece is updating the scheduling logic to route tasks
>> from
>> >> a
>> >> >>> >>>>>> particular
>> >> >>> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much
>> >> work
>> >> >>> >>>>> (though
>> >> >>> >>>>>> it
>> >> >>> >>>>>>>>>> would be easier if the Task models had a column for the
>> >> team
>> >> >>> they
>> >> >>> >>>>>>> belong
>> >> >>> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to
>> get
>> >> >>> the
>> >> >>> >>>>>> team) I
>> >> >>> >>>>>>>>>> have a branch where I was experimenting with this logic
>> >> >>> already.
>> >> >>> >>>>>>>>>> Any who, long story short, I don’t think we necessarily
>> >> need
>> >> >>> to
>> >> >>> >>>>>> remove
>> >> >>> >>>>>>>>>> this piece from the project's scope if it is already
>> partly
>> >> >>> done
>> >> >>> >>>>> and
>> >> >>> >>>>>>> not
>> >> >>> >>>>>>>>>> too difficult.
>> >> >>> >>>>>>>>>>
>> >> >>> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want
>> to
>> >> just
>> >> >>> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we
>> >> have
>> >> >>> it
>> >> >>> >>>>> in.
>> >> >>> >>>>>>> And
>> >> >>> >>>>>>>>> if we can make it in, easily (which it seems we can) - we
>> >> can
>> >> >>> also
>> >> >>> >>>>>>> include
>> >> >>> >>>>>>>>> it in the first iteration. What I wanted to avoid really
>> >> (from
>> >> >>> the
>> >> >>> >>>>>>> original
>> >> >>> >>>>>>>>> design) - again trying to simplify it, limit the changes,
>> >> and
>> >> >>> >>>>> speed up
>> >> >>> >>>>>>>>> implementation. And there is one "complexity" that I
>> wanted
>> >> to
>> >> >>> >>>>> avoid
>> >> >>> >>>>>>>>> specifically - having to have separate , additional
>> >> >>> configuration
>> >> >>> >>>>> per
>> >> >>> >>>>>>> team.
>> >> >>> >>>>>>>>> Not only because it complicates already complex
>> >> configuration
>> >> >>> >>>>> handling
>> >> >>> >>>>>>> (I
>> >> >>> >>>>>>>>> know we have PR for that) but mostly because if it is not
>> >> >>> needed,
>> >> >>> >>>>> we
>> >> >>> >>>>>> can
>> >> >>> >>>>>>>>> simplify documentation and explain to our users easier
>> what
>> >> >>> they
>> >> >>> >>>>> need
>> >> >>> >>>>>>> to do
>> >> >>> >>>>>>>>> to have their own multi-team setup. And I am quite open
>> to
>> >> >>> keeping
>> >> >>> >>>>>>>>> multiple-executors if we can avoid complicating
>> >> configuration.
>> >> >>> >>>>>>>>>
>> >> >>> >>>>>>>>> But I think some details of that and whether we really
>> need
>> >> >>> >>>>> separate
>> >> >>> >>>>>>>>> configuration might also come as a result of updating the
>> >> AIP
>> >> >>> - I
>> >> >>> >>>>> am
>> >> >>> >>>>>> not
>> >> >>> >>>>>>>>> quite sure now if we need it, but we can discuss it when
>> we
>> >> >>> >>>>> iterate on
>> >> >>> >>>>>>> the
>> >> >>> >>>>>>>>> AIP.
>> >> >>> >>>>>>>>>
>> >> >>> >>>>>>>>> J.
>> >> >>> >>>>>>>>>
>> >> >>> >>>>>>>>>
>> >> >>> >>>>>>>
>> >> >>> >>>>>>>
>> >> >>>
>> ---------------------------------------------------------------------
>> >> >>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >> >>> >>>>>>> For additional commands, e-mail:
>> dev-help@airflow.apache.org
>> >> >>> >>>>>>>
>> >> >>> >>>>>>>
>> >> >>> >>>>>>
>> >> >>> >>>>>
>> >> >>> >>>>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> ---------------------------------------------------------------------
>> >> >>> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >> >>> >> For additional commands, e-mail: dev-help@airflow.apache.org
>> >> >>>
>> >> >>>
>> >>
>> >
>>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

To be honest ugliness of it and the messiness of supporting many teams is
deliberate choice. This is also to add friction in supporting 'many' teams.
Supporting 'many' teams had never been a goal for multi-team even in the
last incarnation. Due to scaling of scheduler especially - we should not
give the impression that this solution can scale to 'many' teams.

So that choice is pretty deliberate.

pon., 7 lip 2025, 21:27 użytkownik Oliveira, Niko
<on...@amazon.com.invalid> napisał:

> Thanks for getting back to me and sharing a concrete example Jarek!
>
> I think overloading the executor field, yet again, with another dimension
> is going to get quite messy. If you have many teams and many executors per
> team you now have colons (serving two purposes), semi-colons and commas in
> one config that will be tremendously long. I think this is a poor user
> experience. Same goes for the executor configurations themselves.
> And for someone who's looked through the code, I honestly and
> respectfully, disagree that this is going to be simpler code to write and
> maintain. I think it's actually cleaner and easier for both the users and
> our code base to implement this with first class support, rather than
> embedding team-ness into configs one-by-one and hard coding that
> parsing/support across the code base in specific locations where those
> particular configs are used. It's messy and does not scale well (it smells
> of executor coupling problem we had years ago, this is going to be
> team-coupling where random bits of our code base are going to have to have
> these hacks instead of a first class solution flowing through more
> transparently).
>
> Cheers,
> Niko
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Monday, July 7, 2025 11:15:16 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External
> event driven dags) exists
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> I realized that I owe Niko an explanation of configuration changes. Again -
> following the philosophy above - minimal set of changes to "airflow
> internals". the "minimum" set of changes that will work. I propose the
> change below that has **no** changes to the way how the
> current configuration "shared" feature works - it will change the way
> executors will retrieve their configuration if they are configured
> "per-team" - and we can 100% bank on existing multi-executors.
> I believe that will absolutely minimise the set of changes needed to
> implement multi-team and we will be able to get it "faster" and with "far
> lower risk" of impacting airflow code and say - 3.1 or 3.2 delivery.
>
> Existing multi-executor configuration will be extended to include team
> prefix. The prefix will be separated with ":", entries for different teams
> will be separated with ";"
>
> [core]
>
> executor = team1:KubernetesExecutor,my.custom.module.Executor
> Class;team2:CeleryExecutor
>
> The configuration of executors will also be prefixed with the same team:
>
> [team1:kubernetes_executor]
>
> api_client_retry_configuration = { "total": 3, "backoff_factor": 0.5 }
>
> The environment variables keeping configuration will use ___  (three
> underscores) to replace ":". For example:
>
> AIRFLOW__TEAM1___KUBERNETES_EXECUTOR__API_CLIENT_RETRY_CONFIGURATION`
>
>
> J.
>
> On Thu, Jul 3, 2025 at 8:47 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > The direction this one is taking is interesting. If you're really just
> > trying to make the feature barely possible and mostly targeted towards
> > managed providers to implement the rest, then I suppose this hits the
> mark.
> >
> > Well actually by taking the direction I took, it's not "mostly for
> managed
> > providers" - i see it as it is equally, for managed providers and on-prem
> > users, but also, following the open-source spirit, philosophically, I
> think
> > in Airflow, any such change should be done with those things in mind,
> > because we are at the stage where we are already "established' and by
> > innovating on top what we have we have sometimes more to lose than to
> gain
> > - so I feel with "deployment' features we should be very careful to
> > distinguish 'enabling things" vs. 'doing things". My focus with this
> > iteration was to remove all the roadblocks that make it impossible (or
> > extremely difficult) to implement "real" multi-team and separation
> without
> > modifying airflow core. I though "what is the minimal set of features
> that
> > will make it "possible" for someone motivated to deploy a single airflow
> > for multiple teams.
> >
> > * minimise maintenance effort increase
> > * do not "spoil" the "simple case" - we do not want to add features that
> > make "simple" implementation more complex the current `docker run -it
> > apache/airflow standalone` - should be simple and straightforward to run
> > * if there is anything that involves complex deployment, we should not
> aim
> > to make a "turn-key" solution that we will have to support - similarly
> like
> > we do with our configuration parameters, we have 100s knobs to turn, and
> as
> > long as default settings are reasonable and someone "motivated" can
> > configure and fine-tune - this configuration and fine-tuning should be
> left
> > to them - regardless if they are on-prem or managed. And both should be
> > able to do it.
> >
> > I think it's not only smart technically (we support the low-level basic
> > features and when someone puts them together and makes it more of a
> > turn-key solution they are responsible for designing and implementing it
> -
> > so we have less maintenance effort. But also it's good from a simple
> > "open-source business model point of view" - i.e. it's a smart product
> > decision we should make.
> >
> > Why airflow is #18 in OSS rank - of course we have a huge community and
> > people contributing in their free time, completely voluntarily. And we
> > cherish, support and encourage it. But let's be honest - if not all those
> > that make business on top of airflow did not invest literally millions of
> > dollars (in terms of engineering salary, sponsoring Airflow Summit,
> > supporting people like me (some smart stakeholders at least who
> understand
> > the value of it) who can be good "community spirit" - Airflow would have
> > order of magnitude less activity, reach, Airflow 3 would not be simply
> > possible. And this is a good thing that we have those stakeholders that
> are
> > interested and make money by turning Airflow into a "turn-key" solution.
> > This is a fantastic, symbiotic relationship.
> >
> > So - what my thinking is - we should NOT make things that make airflow
> > more turn-key for those complex cases. We should leave it up to those who
> > want to make it and want to charge money for it. This is cool and great
> > that they can do - and we should not do it "for them" - but on the other
> > hand - we should make it possible that those who want to turn airflow
> into
> > more complex (say multi-team solution) to make it happen - by providing
> > them with minimal set of features that make it possible.
> >
> > And that also - in a way - keeps the balance between on-prem and managed
> > implementation.
> >
> > Something that I've learned as a rule of thumb is that making a feature
> > "generic" compared to custom implementation is 3x-10x more expensive
> (both
> > in implementation and maintenance). And it means that if an on-prem user
> > wants to implement something for them (say turn-key multi-team solution
> for
> > their case) it will cost `x` , but when a managed provider wants to
> > implement a generic multi-team it will cost `10x`. But also managed
> > providers can spread the cost over the premium they will charge to their
> > users so that they don't have to manage Airflow on their own and pay `x`
> > for this mult-team feature to develop on their own. And this is a "fair"
> > choice to make by on-prem users. They might choose what they want to do
> > then. Also it's fair for managed provider - yes they need to invest more,
> > but also they have a chance to shine on promoting it and making it more
> > optimised at scale etc. etc.
> >
> > That is my line of thinking.
> >
> >
> > J.
> >
> >
> > On Thu, Jul 3, 2025 at 1:41 AM Oliveira, Niko
> <on...@amazon.com.invalid>
> > wrote:
> >
> >> Hey Jarek,
> >>
> >>
> >> The direction this one is taking is interesting. If you're really just
> >> trying to make the feature barely possible and mostly targeted towards
> >> managed providers to implement the rest, then I suppose this hits the
> mark.
> >>
> >> But this is not something we're asking for at Amazon and personally I
> >> think we should make the feature reasonably usable for those running
> >> self-managed OSS Airflow as well. There are many users running an
> on-prem
> >> Airflow. Getting too hyper-fixated on an implementation that's so
> >> simplified that it's obtuse and difficult to use by most users seems
> like
> >> the wrong approach to me. But you and I have already discussed this at
> >> length and I haven't convinced you so far, so if I'm the only one with
> this
> >> thinking then I'm happy to disagree and commit as we say at Amazon :)
> >>
> >>
> >> > So I would be rather strong on **not** touching the current
> >> configuration and
> >>
> >> simply adding configuration for per-team executors in executor config -
> >> even if it is uglier and more "low-level".
> >>
> >> Can you explain what "adding configuration for per-team executors in
> >> executor config" would look like? I don't have a concrete sense of what
> you
> >> mean by this.
> >>
> >> Thanks for your efforts on trying to get this feature agreed to and
> voted
> >> on. Looking forward to working on the project in the coming weeks!
> >>
> >> Cheers,
> >> Niko
> >>
> >> ________________________________
> >> From: Jarek Potiuk <ja...@potiuk.com>
> >> Sent: Tuesday, July 1, 2025 10:26:55 PM
> >> To: dev@airflow.apache.org
> >> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82
> (External
> >> event driven dags) exists
> >>
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you can confirm the sender and
> know
> >> the content is safe.
> >>
> >>
> >>
> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe.
> >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> >> le contenu ne présente aucun risque.
> >>
> >>
> >>
> >> Any last comments ? There is a long weekend coming up in the US, so I
> will
> >> likely start voting on the updated AIP on Monday 7th.
> >>
> >> On Fri, Jun 27, 2025 at 12:41 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> > I'd really love to finalise discussion and put it up to a vote some
> time
> >> > after the recording from the last dev call is posted - so that more
> >> > context, details and the LONG discussion we had on it. There is no
> >> *huge*
> >> > hurry  - we have strong dependency on Task Isolation and it seems that
> >> it
> >> > will still take a bit of time to complete, so I'd say I would love to
> >> start
> >> > voting in about a week time - so that maybe at the next dev call we
> can
> >> > "seal" the subject. Happy to see any more comments - especially from
> >> those
> >> > who have opinions but they had no opportunity to express them.
> >> >
> >> > I am personally very happy with the direction it took - simplification
> >> and
> >> > "MVP" kind of approach - also I invite the stakeholders of ours to
> take
> >> a
> >> > close look at the scope and what we really propose - I have a feeling
> >> that
> >> > we can balance it out - there is something we can make to make it not
> >> > "worse" for the offerings they have. I think we have a really good
> >> > symbiotic relationship here, and I would love to leverage that. For
> one
> >> -
> >> > my goal here is to have a minimum number of changes that are impacting
> >> > maintainability of the open-source airflow - but mostly "opening up
> some
> >> > possibilities" - rather than provide turn-key solutions. And mostly
> >> because
> >> > this is good for all sides - less maintenance and complexity for OSS
> >> > maintainers, but more opportunities to make it into "turn-key"
> >> solutions by
> >> > the stakeholders, while also allowing the "on-prem" users - if they
> are
> >> > highly motivated - to use those features by adding the "turn-key"
> layer
> >> on
> >> > their own. Also adding multi-team should not be at the expense of
> >> "simple"
> >> > installations - they should be virtually unaffected.
> >> >
> >> > One example of applying this is cutting on "separate config files". I
> >> > think it moves us closer to a "turn-key" solution but it is not really
> >> > necessary to achieve the three goals above - that's why in the current
> >> > proposal this part is completely removed - Sorry Niko, but I still
> think
> >> > it's one of the things that falls into this bucket. We can easily
> remove
> >> > it, they complicate code, documentation and options the users have,
> and
> >> > even if it is a "little" more complex to manage configuration by
> >> motivated
> >> > users, it's also an opportunity for "turn-key" option that
> stakeholders
> >> can
> >> > build in their products - and we do not have to maintain it in the
> >> > open-source. So I would be rather strong on **not** touching the
> current
> >> > configuration and simply adding configuration for per-team executors
> in
> >> > executor config - even if it is uglier and more "low-level".
> >> >
> >> > So if there are some constructive ideas on what can be done to make it
> >> > "simpler" and less "turn-key" in that respect - I would highly value
> >> such
> >> > ideas and comments. If we can cut down something more that is not
> >> > "necessary" for the three primary goals I came up with - I am more
> than
> >> > happy to do it.
> >> >
> >> > Just to remind - those are the "extracted" goals. I slightly updated
> >> them
> >> > and added to the preamble of the AIP:
> >> >
> >> > * less operational overhead for managing multi-team (once AIP-72 is
> >> > complete) where separate execution environments are important
> >> > * virtual assets sharing between teams
> >> > * ability of having "admin" and "team sharing" capability where dags
> >> from
> >> > multiple teams can be seen in a single Airflow UI  (requires custom
> >> RBAC an
> >> > AIP-56 implementation of Auth Manager - with KeyCloak Auth Manager
> >> being a
> >> > reference implementation)
> >> >
> >> > J.
> >> >
> >> >
> >> > On Thu, Jun 26, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >> >
> >> >>
> >> >>> One technical observation: Now that the dag table no longer has a
> >> >>> team_id in it, what would the behaviour be when a DAG is attempted
> to
> >> move
> >> >>> between bundles? How do we detect this? (I’m not all convinced that
> we
> >> >>> correctly detect duplicate dag ids across bundles today, so I
> wouldn’t
> >> >>> assume or rely on the current behaviour.)
> >> >>>
> >> >>
> >> >> Of course - yes, I realise that - that problem was also not handled
> in
> >> >> the previous iteration to be honest. That is something that dag
> bundle
> >> >> solution allows to solve eventually - but I do not think it's a
> >> blocker for
> >> >> the proposed implementation. We will have to eventually add some way
> of
> >> >> blocking dags to jump between bundles, we might tackle this
> >> separately. I
> >> >> already wanted to propose a separate update to that - but I did not
> >> want to
> >> >> complicate the current proposal. One thing at a time. I can, however
> -
> >> if
> >> >> you consider that as a blocker, extend the current AIP with it. Not a
> >> big
> >> >> problem. This is however a bit independent from the team_id
> >> introduction.
> >> >>
> >> >> Overall, I am still unconvinced this proposal has enough real user
> >> >>> benefit over actually separate deployments, and on balance of the
> >> added
> >> >>> complexity and maintenance burden I do not think it is worth it.
> >> >>>
> >> >>
> >> >> That makes me sad, I thought that over the course of the discussion I
> >> >> addressed all the concerns (in this case the concern was "is it worth
> >> with
> >> >> the cost and little benefit", but when I did it and heavily limited
> the
> >> >> impact, now the concern is "is it worth at all as changes are really
> >> >> minimal" - and surely, anyone can change and adapt their concerns,
> over
> >> >> time,  but that one seems like ever-moving target. I hoped at least
> for
> >> >> some acknowledgment of some concerns (complexity in this case) is
> >> >> addressed, but it seems that you are deeply convinced that we do not
> >> need
> >> >> multi-team at all (which is in stark contrast with at least a dozen
> of
> >> >> bigger and smaller users of Airflow who submitted talks to Airflow
> >> summit
> >> >> (including about 5 or 6 submissions for Airflow 2025) on how they
> spent
> >> >> their engineering effort, time and money on trying to achieve
> something
> >> >> similar - they assessed that it's worth, you  assess that it's not
> >> worth.
> >> >> Somehow I trust our users that they were not spending the money, time
> >> and
> >> >> engineering effort to achieve this because they wanted to spend more
> >> money.
> >> >> I think they assessed it's worth it. So I want to make it a bit
> easier
> >> and
> >> >> more "proper" way for them to do that.
> >> >>
> >> >>>
> >> >>> Upgrades: it is not easier to upgrade under this multi team
> proposal,
> >> >>> but much much harder. This is based on hard earned experience from
> >> helping
> >> >>> Astronomer users — having to coordinate upgrades between multiple
> >> teams
> >> >>> turns in to a months long slog of the hardest kind of work —  people
> >> work:
> >> >>> getting other teams to agree to do things that they don’t directly
> >> care
> >> >>> about — “It’s working for me, I don’t care about upgrading, we’ll
> get
> >> to it
> >> >>> next quarter” is a refrain I’ve heard many times.
> >> >>>
> >> >>
> >> >> Yes. absolutely - this is why we deferred it until we knew what shape
> >> >> task isolation and other AIPs we depend on take on. Because it is
> clear
> >> >> that pretty much all the problem you explain above are going to be
> >> solved
> >> >> with task isolation. And it's not just my opinion. If you want to
> argue
> >> >> with it, you likely need to argue with yourself:
> >> >>
> https://github.com/apache/airflow/issues/51545#issuecomment-2980038478
> >> .
> >> >> Let me quote what you wrote there last week:
> >> >>
> >> >> Ash Berlin Taylor wrote:
> >> >>
> >> >> > A tight coupling between task-sdk and any "server side" component
> is
> >> >> the opposite to one of the goals of AIP-72 (I'm not sure we ever
> >> explicitly
> >> >> said this, but the first point of motivation for the AIP says
> >> "Dependency
> >> >> conflicts for administrators supporting data teams using different
> >> versions
> >> >> of providers, libraries, or python packages")
> >> >> > In short, my goal with TaskSDK, and the reason for introducing
> CalVer
> >> >> and Cadwyn with the execution API is to end up in a world where you
> can
> >> >> upgrade the Airflow Scheduler/API server interdependently of any
> worker
> >> >> nodes (with the exception that the server must be at least as new as
> >> the
> >> >> clients)
> >> >> > This ability to have version-skew is pretty much non-negotiable to
> me
> >> >> and is (other than other languages) one of primary benefits of AIP-72
> >> >>
> >> >> If you read yourself from that quote it basically means "it will be
> >> easy
> >> >> to upgrade airflow independently of workers". So I am a bit confused
> >> here.
> >> >> Yes, I agree it was difficult, but you yourself explain that when
> >> AIP-72
> >> >> (which since API-67 has been accepted has always beem prerequisite of
> >> it)
> >> >> wrote it will be "easy". So I am not sure why you are bringing it
> now.
> >> We
> >> >> assume AIP-72 will be completed and this problem will be gone. Let's
> >> not
> >> >> mention it any more please.
> >> >>
> >> >> The true separation from TaskSDK will likely only land in about 3.2
> >> time
> >> >>> frame. We are actively working on it, but it’s a slow process of
> >> untangling
> >> >>> lots of assumptions made in the code base over the years. Maybe once
> >> we
> >> >>> have that my view would be different, but right now I think this
> >> makes the
> >> >>> proposal a non-starter. Especially as you are saying that most teams
> >> will
> >> >>> have unique connections. If they’ve got those already, then having
> an
> >> asset
> >> >>> trigger use those conns to watch/poll for activity is a much easier
> >> >>> solution to operate and crucially, to scale and upgrade.
> >> >>>
> >> >>
> >> >> Yes. I perfectly understand that and I am fully aware of potentially
> >> 3.2
> >> >> time-frame. And that's fine. Actually I heartily invite you to listen
> >> to
> >> >> the part of my talk from Berlin Buzzwords when I was asked for the
> >> timeline
> >> >> - https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the exact
> >> >> timeline in my talk . My answer was basically - "3.1" or "3.2", and I
> >> >> sincerely hope "3.1" but we might not be able to complete it because
> we
> >> >> have other things to do (other - is indeed the Task Isolation work
> >> that you
> >> >> are leading). And that's perfectly fine. And it absolutely does not
> >> prevent
> >> >> us from voting on the AIP now - similarly as we voted on the previous
> >> >> version of the AIP - knowing that it has some prerequisites a few
> >> months
> >> >> ago. Especially that we know that the feature we need from task
> >> isolation
> >> >> is "non-negotiable". I.e. it WILL happen. We don't hope for it, we
> >> know it
> >> >> will be there. Those are your own words.
> >> >>
> >> >>
> >> >>> >  I think we can’t compare AIP-82 to sharing virtual assets due to
> >> >>> complexity of it.
> >> >>>
> >> >>> Virtual Assets was a mistake, and not how users actually want to use
> >> >>> them. Mea culpa
> >> >>>
> >> >>
> >> >> This is the first time I hear this - certainly you never raised this
> >> >> concern on the devlist. So if you have some concerns about virtual
> >> assets I
> >> >> think you should raise it on the devlist, because I think everyone
> >> here is
> >> >> missing some conversation (or maybe it's just your private opinion
> that
> >> >> you never shared with anyone, but maybe it's worth). I would be
> >> >> interested to hear how the feature that was absolutely most
> successful
> >> >> feature of airflow 2 was a mistake. According to the 2024 survey
> >> >> https://airflow.apache.org/blog/airflow-survey-2024/  - 48% of
> Airflow
> >> >> users have been using it, even if it was added as one of the last
> >> >> big features of Airflow 2. It's the MOST used feature out of all the
> >> >> features out there. I would be really curious to see how it was a
> >> mistake
> >> >> (but  please start a separate thread explaining why you think it was
> a
> >> >> mistake, what are your data points and what do you think should be
> >> fixed.
> >> >> Just dropping "virtual assets were a mistake" in the middle of
> >> multi-team
> >> >> conversation seems completely unjustified without knowing what you
> are
> >> >> talking about. So I think, until we know more, this argument has no
> >> base.
> >> >>
> >> >>
> >> >>>
> >> >>> S
> >> >>> To restate my points:
> >> >>>
> >> >>> - Sharing a deployment between teams today/in 3.1 is operationally
> >> more
> >> >>> complex (both scaling, and upgrades) — this is a con, not a plus.
> >> >>>
> >> >>
> >> >> Surely. But it will be easier when AIP-72 is complete (which I am
> >> >> definitely looking forward to and as clearly explained in AIP-82, is
> a
> >> >> prerequisite of it). Nothing changed here.
> >> >>
> >> >>
> >> >>> - The main user benefit appears to be “allow teams’ DAGs to
> >> communicate
> >> >>> via Assets”, in which case we can do that today by putting more work
> >> in to
> >> >>> AIP-82’s Asset triggers
> >> >>>
> >> >>
> >> >> No. Lower operational complexity for multi-teams (providing that we
> >> >> deliver AIP-72) is another benefit. Virtual assets is another, and
> >> since
> >> >> there is no ground in "virtual assets is a mistake" statement (not
> >> until
> >> >> you explain what you mean by that in a separate discussion) - this is
> >> also
> >> >> still a very valid point.
> >> >>
> >> >>
> >> >>> Soon, we will have then be asked about cross-team governance, policy
> >> >>> enforcement, and potentially unbounded edge cases (e.g.,
> team-specific
> >> >>> secrets, roles, quotas). ain, you get this for free with truely
> >> separate
> >> >>> deployments already
> >> >>> allow different teams to use different executors (including multiple
> >> >>> executors per-team following AIP-61)
> >> >>>
> >> >>
> >> >> Not really. We very explicitly say in the AIP that his is not a goal
> >> and
> >> >> that we have no plans for. And yes, using separate executors per team
> >> is
> >> >> actually back in the AIP-82 in case you did not notice (and the code
> >> needed
> >> >> for it's even implemented and merged already in main by Vincent).
> >> >>
> >> >>
> >> >>> Provably not true right now, and until ~3.2 delivers the full Task
> >> >>> SDK/Core dependency separation this would be _more_ work to upgrade,
> >> not
> >> >>> less, and that work is not shared but still on a central team.
> >> >>>
> >> >>
> >> >> Absolutely - we will wait for AIP-72 completion. I do not want to say
> >> 3.1
> >> >> or 3.2 directly - because there are - as you said - a lot of moving
> >> pieces.
> >> >> So my target for multi-team is "After AIP-72 is completed". Full
> stop.
> >> But
> >> >> there is nothing wrong with accepting the AIP now and doing
> preparatory
> >> >> work in parallel. Similarly as there is no way to have a baby in 1
> >> month by
> >> >> 9 women, there is no way adding more effort to task-sdk isolation
> will
> >> >> speed it up - we alredy have not only 3 people (you leading it, Kaxil
> >> and
> >> >> Amog) but also all the help from me and even 10s of different
> >> contributors
> >> >> (for example with the recent db_test cleanup that I took leadership
> >> on) -
> >> >> and there are people who wish to work on adding multi-team features.
> >> Since
> >> >> the design heavily limits impact on airflow codebase and interactions
> >> with
> >> >> task-sdk implementation, there is nothing wrong with starting
> >> >> implementation in parallel either- amazon team is keen to move it
> >> forward -
> >> >> they even already implemented SQS trigger for assets, and we are
> >> working
> >> >> together on FAB removal, Keycloak authentication manager - and they
> >> seem to
> >> >> still have capacity and drive to progress multi-team. So I am not
> sure
> >> if
> >> >> we are trading off something. There is no "if we work on more on task
> >> sdk
> >> >> and drop multi-team things will be faster". Generally in open source
> >> people
> >> >> work in the area where they feel they can provide best value - such
> as
> >> you
> >> >> working on task-sdk, me on CI,dev env, they will deliver more value
> on
> >> >> multi-team
> >> >>
> >> >>
> >> >>>
> >> >>> So please, as succinctly as possible, please tell me what the direct
> >> >>> benefit to users this proposal is over us putting this effort in to
> >> writing
> >> >>> better Asset triggers instead?
> >> >>>
> >> >>
> >> >>
> >> >> * less operational overhead for managing multi-team (once AIP-72 is
> >> >> complete) where separate execution environments are important
> >> >> * virtual assets sharing
> >> >> * ability of having "admin" and "team sharing" capability where dags
> >> from
> >> >> multiple teams can be seen in a single Airflow UI  (requires custom
> >> RBAC)
> >> >>
> >> >> None of this can be done via beter asset triggers
> >> >>
> >> >>
> >> >>>
> >> >>> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
> >> >>> >
> >> >>> > My counter-points:
> >> >>> >
> >> >>> >
> >> >>> >> 1. Managing a multi team deployment is not materially different
> >> from
> >> >>> >> managing a deployment per team
> >> >>> >>
> >> >>> >
> >> >>> > It's a bit easier - especially when it comes to upgrades
> (especially
> >> >>> in the
> >> >>> > case we are targetting when we are not targetting multi-tenant,
> but
> >> >>> several
> >> >>> > relatively closely cooperating teams with different dependncy
> >> >>> requiremens
> >> >>> > and isolation need.
> >> >>> >
> >> >>> > 2. The database changes were quite wide-reaching
> >> >>> >>
> >> >>> >
> >> >>> > Yes. that is addressed.
> >> >>> >
> >> >>> >
> >> >>> >> 3. I don’t believe the original AIP (again, I haven’t read the
> >> updated
> >> >>> >> proposal or recent messages on the thread. yet) will meet what
> many
> >> >>> users
> >> >>> >> want out of a multiteam solution
> >> >>> >>
> >> >>> >
> >> >>> > I think we will only see when we try. A lot of people thing they
> >> would,
> >> >>> > even if they are warned. I know at least one user (Wealthsimple)
> who
> >> >>> > definitely want to use it and they got a very detailed explanation
> >> of
> >> >>> the
> >> >>> > idea and understand it well. So I am sure that **some** users
> would.
> >> >>> But we
> >> >>> > do not know how many.
> >> >>> >
> >> >>> >
> >> >>> >> To expand on those points a bit more
> >> >>> >>
> >> >>> >> On 1. The only components that are shared are, I think, the
> >> scheduler
> >> >>> and
> >> >>> >> the API server, and it’s arguable if that is actually a good idea
> >> >>> given
> >> >>> >> those are likely to be the most performance sensitive components
> >> >>> anyway.
> >> >>> >>
> >> >>> >> Additionally the fact that the scheduler is a shared component
> >> makes
> >> >>> >> upgrading it almost a non starter as you would likely need
> buy-in,
> >> >>> changes,
> >> >>> >> and testing form ALL teams using it. I’d argue that this is a
> huge
> >> >>> negative
> >> >>> >> until we finish off the version indepence work of AIP-72.
> >> >>> >>
> >> >>> >
> >> >>> > Quite disagree here - especially that our target is that task-sdk
> is
> >> >>> > supposed to provide all isolation that is needed. There should be
> 0
> >> >>> changes
> >> >>> > in the dags needed to upgrade scheduler, api_server, triggerer -
> >> >>> precisely
> >> >>> > because we introduced backwards-compatible task-sdk.
> >> >>> >
> >> >>> > On 3 my complaint is essentially that this doesn’t go nearly far
> >> >>> enough. It
> >> >>> >> doesn’t allow read only views to other teams dags. I don’t think
> it
> >> >>> allows
> >> >>> >> you to be in multiple teams at once. You can’t share a connection
> >> >>> between
> >> >>> >> teams but only allow certain specified dags to access it, but
> would
> >> >>> have to
> >> >>> >> either be globally usable, or duplicated-and-kept-in-sync between
> >> >>> teams. In
> >> >>> >> short I think it fall short of being useful..
> >> >>> >>
> >> >>> >
> >> >>> > Oh absolutely all that is possible (except sharing single
> >> connections
> >> >>> > between multiple teams - which is a very niche use cases and
> >> >>> duplication
> >> >>> > here is perfectly ok as first approximation - and if we need more
> we
> >> >>> can
> >> >>> > add it later).
> >> >>> >
> >> >>> > Auth manager RBAC and access is abstracted away, and the Keyclock
> >> >>> Manager
> >> >>> > implemented by Vincent allows to manage completely independent and
> >> >>> separate
> >> >>> > RBAC based on arguments and resources provided by Airflow. There
> is
> >> >>> nothing
> >> >>> > to prevent the user who configures KeyCloak RBAC to define it in
> the
> >> >>> way:
> >> >>> >
> >> >>> > if group a > allow to read a and write b
> >> >>> > if group b > alllow to write b but not a
> >> >>> >
> >> >>> > and any other combinations. KeyCloak implementation - pretty
> >> advanced
> >> >>> > already - (and design of auth manager) completely abstracts away
> >> both
> >> >>> > authentication and authorization to KeyCloak and KeyCloak has RBAC
> >> >>> > management built in. Also any of the users can write their own -
> >> even
> >> >>> > hard-coded authentication manager to do the same if they do not
> >> want to
> >> >>> > have configurable KeyCloak. Even SimpleAuthManager could be
> >> hard-coded
> >> >>> to
> >> >>> > provide thiose features.
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >> So on the surface, I’m no more in favour of using dag bundle as a
> >> >>> >> replacement for team id as I think most of the above points still
> >> >>> stand.
> >> >>> >>
> >> >>> >
> >> >>> > We disagree here.
> >> >>> >
> >> >>> >>
> >> >>> >> My counter proposal: We do _nothing_ to core airflow. We work on
> >> >>> improving
> >> >>> >> the event-based trigger o fdags (write more triggers for
> read/check
> >> >>> remote
> >> >>> >> Assets etc) so that teams can have 100% isolated deployments but
> >> still
> >> >>> >> trigger dags based on asset events from other teams.
> >> >>> >>
> >> >>> >
> >> >>> > That does not solve any of the other design goals - only allows to
> >> >>> trigger
> >> >>> > assets a bit more easily (but also it's not entirely solved by
> >> AIP-82
> >> >>> > because it does not solve virtual assets - only ones that have
> >> defined
> >> >>> > triggerer and "something" to listen on - which is way more complex
> >> than
> >> >>> > just defining asset in a Dag and using it in another). I think we
> >> can't
> >> >>> > compare AIP-82 to sharing virtual assets due to complexity of it.
> I
> >> >>> > explained it in the doc.
> >> >>> >
> >> >>> >
> >> >>> > I will now go and catch up with the long thread and updated
> proposal
> >> >>> and
> >> >>> >> come back.
> >> >>> >>
> >> >>> >
> >> >>> > Please. I hope the above explaination will help in better
> >> >>> understanding of
> >> >>> > the proposal, because I think you had some assumptions that do not
> >> >>> hold any
> >> >>> > more with the new proposal.
> >> >>> >
> >> >>> > J.
> >> >>> >
> >> >>> >
> >> >>> >>
> >> >>> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >> >>> >>>
> >> >>> >>> Just to clarify the relation - I updated the AIP now to refer to
> >> >>> AIP-82
> >> >>> >> and
> >> >>> >>> to explain relation between the "cross-team" and "cross-airflow"
> >> >>> asset
> >> >>> >>> triggering - this is what I added:
> >> >>> >>>
> >> >>> >>> Note that there is a relation between AIP-82 ("External Driven
> >> >>> >> Scheduling")
> >> >>> >>> and this part of the functionality. When you have multiple
> >> instances
> >> >>> of
> >> >>> >>> Airflow, you can use shared datasets - "Physical datasets" -
> that
> >> >>> several
> >> >>> >>> Airflow Instances can use - for example there could be an S3
> >> object
> >> >>> that
> >> >>> >> is
> >> >>> >>> produced by one airflow instance, and consumed by another. That
> >> >>> requires
> >> >>> >>> deferred trigger to monitor for such datasets, and appropriate
> >> >>> >> permissions
> >> >>> >>> to the external dataset, and you could achive similar result to
> >> >>> >> cross-team
> >> >>> >>> dataset triggering (but cross airflow). However the feature of
> >> >>> sharing
> >> >>> >>> datasets between the teams also works for virtual assets, that
> do
> >> not
> >> >>> >> have
> >> >>> >>> physically shared "objects" and trigger that is monitoring for
> >> >>> changes in
> >> >>> >>> such asset.
> >> >>> >>>
> >> >>> >>> J.
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com>
> >> >>> wrote:
> >> >>> >>>
> >> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
> >> >>> reference
> >> >>> >> to
> >> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
> >> >>> through.
> >> >>> >>>>
> >> >>> >>>> Yep. It did not - because I did not think it was needed or even
> >> very
> >> >>> >>>> important after the simplifications. AIP-82 has a different
> >> scope,
> >> >>> >> really.
> >> >>> >>>> It only helps when the Assets are "real" data files which we
> have
> >> >>> >> physical
> >> >>> >>>> triggers for, it's slightly related - sharing datasets between
> >> teams
> >> >>> >>>> (including those that do not require physical files and
> >> triggers) is
> >> >>> >> still
> >> >>> >>>> possible in the design we have now, but it's not (and never
> was)
> >> the
> >> >>> >>>> **only** reason for having multi-team. There always was (and
> >> still
> >> >>> is)
> >> >>> >> the
> >> >>> >>>> possibility of having a common, distinct environments (i.e.
> >> >>> dependencies
> >> >>> >>>> and providers) per team, the possibility of having connections
> >> and
> >> >>> >>>> variables that are only accessible to one team and not the
> other,
> >> >>> and
> >> >>> >>>> isolating workload execution (all that while allowing to manage
> >> >>> multiple
> >> >>> >>>> team and schedule things with single deployment). That did not
> >> >>> change.
> >> >>> >> What
> >> >>> >>>> changed a lot is that it is now way simpler, something that we
> >> can
> >> >>> >>>> implement without heavy changes to the codebase - and give it
> to
> >> our
> >> >>> >> users,
> >> >>> >>>> so that they can assess if this is something they need without
> >> too
> >> >>> much
> >> >>> >>>> risk and effort.
> >> >>> >>>>
> >> >>> >>>> This was - I believe the main concern, that the value we get
> from
> >> >>> it is
> >> >>> >>>> not dramatic, but the required changes are huge. This
> "redesign"
> >> >>> changes
> >> >>> >>>> the equation - the value is still unchanged, but the cost of
> >> >>> >> implementing
> >> >>> >>>> it and impact on the Airflow codebase is much smaller. I still
> >> have
> >> >>> not
> >> >>> >>>> heard back from Ash if my proposal responds to his original
> >> concern
> >> >>> >> though,
> >> >>> >>>> so I am mostly guessing (also based on the positive impact of
> >> >>> others)
> >> >>> >> that
> >> >>> >>>> yes it does. But to be honest I am not sure and I would love to
> >> hear
> >> >>> >> back,
> >> >>> >>>> I decided to update the AIP to reflect it - regardless,
> because I
> >> >>> think
> >> >>> >> the
> >> >>> >>>> simplification I proposed keeps the original goals, but is
> indeed
> >> >>> way
> >> >>> >>>> simpler.
> >> >>> >>>>
> >> >>> >>>>> This is a very difficult thread to catch up on.
> >> >>> >>>>
> >> >>> >>>> Valid point. Let me summarize what is the result:
> >> >>> >>>>
> >> >>> >>>> * I significantly simplified the implementation proposal
> >> comparing
> >> >>> to
> >> >>> >> the
> >> >>> >>>> original version
> >> >>> >>>> * main simplification is very limited impact on existing
> >> database -
> >> >>> >>>> without "ripple effect" that would require us to change a lot
> of
> >> >>> tables,
> >> >>> >>>> including their primary keys, and heavily impact the UI
> >> >>> >>>> * this is now more of an incremental change that can be
> >> implemented
> >> >>> way
> >> >>> >>>> faster and with far less risk
> >> >>> >>>> * updated idea is based on leveraging bundles (already part of
> >> our
> >> >>> data
> >> >>> >>>> model) to map them (many-to-one) to a team - which requires to
> >> just
> >> >>> >> extend
> >> >>> >>>> the data model with bundle mapping and add team_id to
> connections
> >> >>> and
> >> >>> >>>> variables. Those are all needed DB changes.
> >> >>> >>>>
> >> >>> >>>> The AIP is updated - in a one single big change so It should be
> >> >>> easy to
> >> >>> >>>> compare the changes:
> >> >>> >>>>
> >> >>> >>
> >> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
> >> >>> >>>> -> I even named the version appropriately "Simplified
> multi-team
> >> >>> AIP" -
> >> >>> >> you
> >> >>> >>>> can select and compare v.65 with v.66 to see the exact
> >> differences I
> >> >>> >>>> proposed.
> >> >>> >>>>
> >> >>> >>>> I hope it will be helpful to catch up and for those who did not
> >> >>> follow,
> >> >>> >> to
> >> >>> >>>> be able to make up their minds about it.
> >> >>> >>>>
> >> >>> >>>> J.
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
> >> >>> >> <vi...@astronomer.io.invalid>
> >> >>> >>>> wrote:
> >> >>> >>>>
> >> >>> >>>>> This is a very difficult thread to catch up on.
> >> >>> >>>>> I will take a detailed look at the AIP update to try to figure
> >> out
> >> >>> the
> >> >>> >>>>> changes in the proposal.
> >> >>> >>>>>
> >> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
> >> >>> reference
> >> >>> >> to
> >> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
> >> >>> through.
> >> >>> >>>>>
> >> >>> >>>>> Vikram
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
> >> >>> >>>>> gopidesupavan@gmail.com>
> >> >>> >>>>> wrote:
> >> >>> >>>>>
> >> >>> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's
> much
> >> >>> more
> >> >>> >> slim
> >> >>> >>>>>> down.
> >> >>> >>>>>>
> >> >>> >>>>>> left a minor comment. :) Overall looking great.
> >> >>> >>>>>>
> >> >>> >>>>>> Pavan
> >> >>> >>>>>>
> >> >>> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
> >> >>> >>>>> <j_scheffler@gmx.de.invalid
> >> >>> >>>>>>>
> >> >>> >>>>>> wrote:
> >> >>> >>>>>>
> >> >>> >>>>>>> Thanks for the rework/update of the AIP-72!
> >> >>> >>>>>>>
> >> >>> >>>>>>> Just a few small comments but overall I like it as it is
> much
> >> >>> leaner
> >> >>> >>>>>>> than originally planned and is in a level of complexity that
> >> it
> >> >>> >> really
> >> >>> >>>>>>> seems to be a benefit to close the gap as described.
> >> >>> >>>>>>>
> >> >>> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
> >> >>> >>>>>>>> I updated the AIP - including architecture images and
> >> reviewed
> >> >>> it
> >> >>> >>>>>> (again)
> >> >>> >>>>>>>> and corrected any ambiguities and places where it needed to
> >> be
> >> >>> >>>>> changed.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> I think the current state
> >> >>> >>>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>
> >> >>> >>
> >> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >> >>> >>>>>>>> - nicely describes the proposal.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> Comparing to the previous one:
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> 1. The DB changes are far less intrusive - no ripple effect
> >> on
> >> >>> >>>>> Airflow
> >> >>> >>>>>>>> 2. There is no need to merge configurations and provide
> >> >>> different
> >> >>> >>>>> set
> >> >>> >>>>>> of
> >> >>> >>>>>>>> configs per team - we can add it later but I do not see why
> >> we
> >> >>> need
> >> >>> >>>>> it
> >> >>> >>>>>> in
> >> >>> >>>>>>>> this simplified version
> >> >>> >>>>>>>> 3. We can still configure a different set of executors per
> >> team
> >> >>> -
> >> >>> >>>>> that
> >> >>> >>>>>> is
> >> >>> >>>>>>>> already implemented (we just need to wire it to the bundle
> ->
> >> >>> team
> >> >>> >>>>>>> mapping).
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> I think it will be way simpler and faster to implement this
> >> way
> >> >>> and
> >> >>> >>>>> it
> >> >>> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we
> can
> >> >>> give
> >> >>> >>>>> our
> >> >>> >>>>>>>> users so that they can provide feedback.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> J.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <
> >> jarek@potiuk.com>
> >> >>> >>>>> wrote:
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>> I like this iteration a bit more now for sure, thanks for
> >> >>> being
> >> >>> >>>>>>> receptive
> >> >>> >>>>>>>>>> to feedback! :)
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>> This now becomes quite close to what was proposing
> before,
> >> we
> >> >>> now
> >> >>> >>>>>> again
> >> >>> >>>>>>>>>> have a team ID (which I think is really needed here, glad
> >> to
> >> >>> see
> >> >>> >>>>> it
> >> >>> >>>>>>> back)
> >> >>> >>>>>>>>>> and it will be used for auth management, configuration
> >> >>> >>>>> specification,
> >> >>> >>>>>>> etc
> >> >>> >>>>>>>>>> but will be carried by Bundle instead of the dag model.
> >> Which
> >> >>> as
> >> >>> >>>>> you
> >> >>> >>>>>>> say
> >> >>> >>>>>>>>>> “For that we will need to make sure that both api-server,
> >> >>> >>>>> scheduler
> >> >>> >>>>>> and
> >> >>> >>>>>>>>>> triggerer have access to the "bundle definition" (to
> >> perform
> >> >>> the
> >> >>> >>>>>>> mapping)"
> >> >>> >>>>>>>>>> which honestly doesn’t feel too much different from the
> >> >>> original
> >> >>> >>>>>>> proposal
> >> >>> >>>>>>>>>> we had last week of adding it to Dag table and ensuring
> >> it’s
> >> >>> >>>>>> available
> >> >>> >>>>>>>>>> everywhere. but either way I’m happy to meet in the
> middle
> >> and
> >> >>> >>>>> keep
> >> >>> >>>>>> it
> >> >>> >>>>>>> on
> >> >>> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable
> >> location.
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>> I think the big difference is the "ripple effect" that was
> >> >>> >>>>> discussed
> >> >>> >>>>>> in
> >> >>> >>>>>>>>>
> >> >>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
> >> >>> >>>>>> (and I
> >> >>> >>>>>>>>> believe - correct me if I am wrong Ash - important trigger
> >> for
> >> >>> the
> >> >>> >>>>>>>>> discussion) so far what we wanted is to extend the primary
> >> key
> >> >>> and
> >> >>> >>>>> it
> >> >>> >>>>>>> would
> >> >>> >>>>>>>>> ripple through all the pieces of Airflow -> models, API,
> UI
> >> >>> etc.
> >> >>> >>>>> ...
> >> >>> >>>>>>>>> However - we already have `bundle_name" and
> "bundle_version"
> >> >>> in the
> >> >>> >>>>>> Dag
> >> >>> >>>>>>>>> model. So I think when we add a separate table where we
> map
> >> the
> >> >>> >>>>> bundle
> >> >>> >>>>>>> to
> >> >>> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not
> >> want
> >> >>> to
> >> >>> >>>>>> change
> >> >>> >>>>>>>>> primary key, we do not want to change UI in any way
> (except
> >> >>> >>>>> filtering
> >> >>> >>>>>> of
> >> >>> >>>>>>>>> DAGs available based on your team - but that will be
> >> handled in
> >> >>> >>>>> Auth
> >> >>> >>>>>>>>> Manager and will not impact UI in any way, I think that's
> a
> >> >>> huge
> >> >>> >>>>>>>>> simplification of the implementation, and if we agree to
> it
> >> - i
> >> >>> >>>>> think
> >> >>> >>>>>> it
> >> >>> >>>>>>>>> should speed up the implementation significantly. There
> are
> >> >>> only a
> >> >>> >>>>>>> limited
> >> >>> >>>>>>>>> number of times where you need to look up the team_id - so
> >> >>> having
> >> >>> >>>>> the
> >> >>> >>>>>>>>> bundle -> team mapping in a separate table and having to
> >> look
> >> >>> them
> >> >>> >>>>> up
> >> >>> >>>>>>>>> should not be a problem. And it has much less complexity
> and
> >> >>> >>>>>>>>> "ripple-effect" through the codebase (for example I could
> >> >>> imagine
> >> >>> >>>>> 100s
> >> >>> >>>>>>> or
> >> >>> >>>>>>>>> thousands already written tests that would have to be
> >> adapted
> >> >>> if we
> >> >>> >>>>>>> changed
> >> >>> >>>>>>>>> the primary key - where there will be pretty much zero
> >> impact
> >> >>> on
> >> >>> >>>>>>> existing
> >> >>> >>>>>>>>> tests if we just add bundle -> team lookup table.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>> One other thing I’d point out is that I think including
> >> >>> executors
> >> >>> >>>>> per
> >> >>> >>>>>>>>>> team is a very easy win and quite possible without much
> >> work.
> >> >>> I
> >> >>> >>>>>> already
> >> >>> >>>>>>>>>> have much of the code written. Executors are already
> aware
> >> of
> >> >>> >>>>> Teams
> >> >>> >>>>>>> that
> >> >>> >>>>>>>>>> own them (merged), I have a PR open to have configuration
> >> per
> >> >>> team
> >> >>> >>>>>>> (with a
> >> >>> >>>>>>>>>> quite simple and isolated approach, I believe you
> approved
> >> >>> Jarek).
> >> >>> >>>>>> The
> >> >>> >>>>>>> last
> >> >>> >>>>>>>>>> piece is updating the scheduling logic to route tasks
> from
> >> a
> >> >>> >>>>>> particular
> >> >>> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much
> >> work
> >> >>> >>>>> (though
> >> >>> >>>>>> it
> >> >>> >>>>>>>>>> would be easier if the Task models had a column for the
> >> team
> >> >>> they
> >> >>> >>>>>>> belong
> >> >>> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to
> get
> >> >>> the
> >> >>> >>>>>> team) I
> >> >>> >>>>>>>>>> have a branch where I was experimenting with this logic
> >> >>> already.
> >> >>> >>>>>>>>>> Any who, long story short, I don’t think we necessarily
> >> need
> >> >>> to
> >> >>> >>>>>> remove
> >> >>> >>>>>>>>>> this piece from the project's scope if it is already
> partly
> >> >>> done
> >> >>> >>>>> and
> >> >>> >>>>>>> not
> >> >>> >>>>>>>>>> too difficult.
> >> >>> >>>>>>>>>>
> >> >>> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want to
> >> just
> >> >>> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we
> >> have
> >> >>> it
> >> >>> >>>>> in.
> >> >>> >>>>>>> And
> >> >>> >>>>>>>>> if we can make it in, easily (which it seems we can) - we
> >> can
> >> >>> also
> >> >>> >>>>>>> include
> >> >>> >>>>>>>>> it in the first iteration. What I wanted to avoid really
> >> (from
> >> >>> the
> >> >>> >>>>>>> original
> >> >>> >>>>>>>>> design) - again trying to simplify it, limit the changes,
> >> and
> >> >>> >>>>> speed up
> >> >>> >>>>>>>>> implementation. And there is one "complexity" that I
> wanted
> >> to
> >> >>> >>>>> avoid
> >> >>> >>>>>>>>> specifically - having to have separate , additional
> >> >>> configuration
> >> >>> >>>>> per
> >> >>> >>>>>>> team.
> >> >>> >>>>>>>>> Not only because it complicates already complex
> >> configuration
> >> >>> >>>>> handling
> >> >>> >>>>>>> (I
> >> >>> >>>>>>>>> know we have PR for that) but mostly because if it is not
> >> >>> needed,
> >> >>> >>>>> we
> >> >>> >>>>>> can
> >> >>> >>>>>>>>> simplify documentation and explain to our users easier
> what
> >> >>> they
> >> >>> >>>>> need
> >> >>> >>>>>>> to do
> >> >>> >>>>>>>>> to have their own multi-team setup. And I am quite open to
> >> >>> keeping
> >> >>> >>>>>>>>> multiple-executors if we can avoid complicating
> >> configuration.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> But I think some details of that and whether we really
> need
> >> >>> >>>>> separate
> >> >>> >>>>>>>>> configuration might also come as a result of updating the
> >> AIP
> >> >>> - I
> >> >>> >>>>> am
> >> >>> >>>>>> not
> >> >>> >>>>>>>>> quite sure now if we need it, but we can discuss it when
> we
> >> >>> >>>>> iterate on
> >> >>> >>>>>>> the
> >> >>> >>>>>>>>> AIP.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> J.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>>
> ---------------------------------------------------------------------
> >> >>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> >>> >>>>>>> For additional commands, e-mail:
> dev-help@airflow.apache.org
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>
> >> >>> >>>>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> ---------------------------------------------------------------------
> >> >>> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> >>> >> For additional commands, e-mail: dev-help@airflow.apache.org
> >> >>>
> >> >>>
> >>
> >
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

["Oliveira, Niko" <on...@amazon.com.INVALID>]

Thanks for getting back to me and sharing a concrete example Jarek!

I think overloading the executor field, yet again, with another dimension is going to get quite messy. If you have many teams and many executors per team you now have colons (serving two purposes), semi-colons and commas in one config that will be tremendously long. I think this is a poor user experience. Same goes for the executor configurations themselves.
And for someone who's looked through the code, I honestly and respectfully, disagree that this is going to be simpler code to write and maintain. I think it's actually cleaner and easier for both the users and our code base to implement this with first class support, rather than embedding team-ness into configs one-by-one and hard coding that parsing/support across the code base in specific locations where those particular configs are used. It's messy and does not scale well (it smells of executor coupling problem we had years ago, this is going to be team-coupling where random bits of our code base are going to have to have these hacks instead of a first class solution flowing through more transparently).

Cheers,
Niko

________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Monday, July 7, 2025 11:15:16 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.



I realized that I owe Niko an explanation of configuration changes. Again -
following the philosophy above - minimal set of changes to "airflow
internals". the "minimum" set of changes that will work. I propose the
change below that has **no** changes to the way how the
current configuration "shared" feature works - it will change the way
executors will retrieve their configuration if they are configured
"per-team" - and we can 100% bank on existing multi-executors.
I believe that will absolutely minimise the set of changes needed to
implement multi-team and we will be able to get it "faster" and with "far
lower risk" of impacting airflow code and say - 3.1 or 3.2 delivery.

Existing multi-executor configuration will be extended to include team
prefix. The prefix will be separated with ":", entries for different teams
will be separated with ";"

[core]

executor = team1:KubernetesExecutor,my.custom.module.Executor
Class;team2:CeleryExecutor

The configuration of executors will also be prefixed with the same team:

[team1:kubernetes_executor]

api_client_retry_configuration = { "total": 3, "backoff_factor": 0.5 }

The environment variables keeping configuration will use ___  (three
underscores) to replace ":". For example:

AIRFLOW__TEAM1___KUBERNETES_EXECUTOR__API_CLIENT_RETRY_CONFIGURATION`


J.

On Thu, Jul 3, 2025 at 8:47 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> > The direction this one is taking is interesting. If you're really just
> trying to make the feature barely possible and mostly targeted towards
> managed providers to implement the rest, then I suppose this hits the mark.
>
> Well actually by taking the direction I took, it's not "mostly for managed
> providers" - i see it as it is equally, for managed providers and on-prem
> users, but also, following the open-source spirit, philosophically, I think
> in Airflow, any such change should be done with those things in mind,
> because we are at the stage where we are already "established' and by
> innovating on top what we have we have sometimes more to lose than to gain
> - so I feel with "deployment' features we should be very careful to
> distinguish 'enabling things" vs. 'doing things". My focus with this
> iteration was to remove all the roadblocks that make it impossible (or
> extremely difficult) to implement "real" multi-team and separation without
> modifying airflow core. I though "what is the minimal set of features that
> will make it "possible" for someone motivated to deploy a single airflow
> for multiple teams.
>
> * minimise maintenance effort increase
> * do not "spoil" the "simple case" - we do not want to add features that
> make "simple" implementation more complex the current `docker run -it
> apache/airflow standalone` - should be simple and straightforward to run
> * if there is anything that involves complex deployment, we should not aim
> to make a "turn-key" solution that we will have to support - similarly like
> we do with our configuration parameters, we have 100s knobs to turn, and as
> long as default settings are reasonable and someone "motivated" can
> configure and fine-tune - this configuration and fine-tuning should be left
> to them - regardless if they are on-prem or managed. And both should be
> able to do it.
>
> I think it's not only smart technically (we support the low-level basic
> features and when someone puts them together and makes it more of a
> turn-key solution they are responsible for designing and implementing it -
> so we have less maintenance effort. But also it's good from a simple
> "open-source business model point of view" - i.e. it's a smart product
> decision we should make.
>
> Why airflow is #18 in OSS rank - of course we have a huge community and
> people contributing in their free time, completely voluntarily. And we
> cherish, support and encourage it. But let's be honest - if not all those
> that make business on top of airflow did not invest literally millions of
> dollars (in terms of engineering salary, sponsoring Airflow Summit,
> supporting people like me (some smart stakeholders at least who understand
> the value of it) who can be good "community spirit" - Airflow would have
> order of magnitude less activity, reach, Airflow 3 would not be simply
> possible. And this is a good thing that we have those stakeholders that are
> interested and make money by turning Airflow into a "turn-key" solution.
> This is a fantastic, symbiotic relationship.
>
> So - what my thinking is - we should NOT make things that make airflow
> more turn-key for those complex cases. We should leave it up to those who
> want to make it and want to charge money for it. This is cool and great
> that they can do - and we should not do it "for them" - but on the other
> hand - we should make it possible that those who want to turn airflow into
> more complex (say multi-team solution) to make it happen - by providing
> them with minimal set of features that make it possible.
>
> And that also - in a way - keeps the balance between on-prem and managed
> implementation.
>
> Something that I've learned as a rule of thumb is that making a feature
> "generic" compared to custom implementation is 3x-10x more expensive (both
> in implementation and maintenance). And it means that if an on-prem user
> wants to implement something for them (say turn-key multi-team solution for
> their case) it will cost `x` , but when a managed provider wants to
> implement a generic multi-team it will cost `10x`. But also managed
> providers can spread the cost over the premium they will charge to their
> users so that they don't have to manage Airflow on their own and pay `x`
> for this mult-team feature to develop on their own. And this is a "fair"
> choice to make by on-prem users. They might choose what they want to do
> then. Also it's fair for managed provider - yes they need to invest more,
> but also they have a chance to shine on promoting it and making it more
> optimised at scale etc. etc.
>
> That is my line of thinking.
>
>
> J.
>
>
> On Thu, Jul 3, 2025 at 1:41 AM Oliveira, Niko <on...@amazon.com.invalid>
> wrote:
>
>> Hey Jarek,
>>
>>
>> The direction this one is taking is interesting. If you're really just
>> trying to make the feature barely possible and mostly targeted towards
>> managed providers to implement the rest, then I suppose this hits the mark.
>>
>> But this is not something we're asking for at Amazon and personally I
>> think we should make the feature reasonably usable for those running
>> self-managed OSS Airflow as well. There are many users running an on-prem
>> Airflow. Getting too hyper-fixated on an implementation that's so
>> simplified that it's obtuse and difficult to use by most users seems like
>> the wrong approach to me. But you and I have already discussed this at
>> length and I haven't convinced you so far, so if I'm the only one with this
>> thinking then I'm happy to disagree and commit as we say at Amazon :)
>>
>>
>> > So I would be rather strong on **not** touching the current
>> configuration and
>>
>> simply adding configuration for per-team executors in executor config -
>> even if it is uglier and more "low-level".
>>
>> Can you explain what "adding configuration for per-team executors in
>> executor config" would look like? I don't have a concrete sense of what you
>> mean by this.
>>
>> Thanks for your efforts on trying to get this feature agreed to and voted
>> on. Looking forward to working on the project in the coming weeks!
>>
>> Cheers,
>> Niko
>>
>> ________________________________
>> From: Jarek Potiuk <ja...@potiuk.com>
>> Sent: Tuesday, July 1, 2025 10:26:55 PM
>> To: dev@airflow.apache.org
>> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External
>> event driven dags) exists
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>>
>>
>>
>> Any last comments ? There is a long weekend coming up in the US, so I will
>> likely start voting on the updated AIP on Monday 7th.
>>
>> On Fri, Jun 27, 2025 at 12:41 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > I'd really love to finalise discussion and put it up to a vote some time
>> > after the recording from the last dev call is posted - so that more
>> > context, details and the LONG discussion we had on it. There is no
>> *huge*
>> > hurry  - we have strong dependency on Task Isolation and it seems that
>> it
>> > will still take a bit of time to complete, so I'd say I would love to
>> start
>> > voting in about a week time - so that maybe at the next dev call we can
>> > "seal" the subject. Happy to see any more comments - especially from
>> those
>> > who have opinions but they had no opportunity to express them.
>> >
>> > I am personally very happy with the direction it took - simplification
>> and
>> > "MVP" kind of approach - also I invite the stakeholders of ours to take
>> a
>> > close look at the scope and what we really propose - I have a feeling
>> that
>> > we can balance it out - there is something we can make to make it not
>> > "worse" for the offerings they have. I think we have a really good
>> > symbiotic relationship here, and I would love to leverage that. For one
>> -
>> > my goal here is to have a minimum number of changes that are impacting
>> > maintainability of the open-source airflow - but mostly "opening up some
>> > possibilities" - rather than provide turn-key solutions. And mostly
>> because
>> > this is good for all sides - less maintenance and complexity for OSS
>> > maintainers, but more opportunities to make it into "turn-key"
>> solutions by
>> > the stakeholders, while also allowing the "on-prem" users - if they are
>> > highly motivated - to use those features by adding the "turn-key" layer
>> on
>> > their own. Also adding multi-team should not be at the expense of
>> "simple"
>> > installations - they should be virtually unaffected.
>> >
>> > One example of applying this is cutting on "separate config files". I
>> > think it moves us closer to a "turn-key" solution but it is not really
>> > necessary to achieve the three goals above - that's why in the current
>> > proposal this part is completely removed - Sorry Niko, but I still think
>> > it's one of the things that falls into this bucket. We can easily remove
>> > it, they complicate code, documentation and options the users have, and
>> > even if it is a "little" more complex to manage configuration by
>> motivated
>> > users, it's also an opportunity for "turn-key" option that stakeholders
>> can
>> > build in their products - and we do not have to maintain it in the
>> > open-source. So I would be rather strong on **not** touching the current
>> > configuration and simply adding configuration for per-team executors in
>> > executor config - even if it is uglier and more "low-level".
>> >
>> > So if there are some constructive ideas on what can be done to make it
>> > "simpler" and less "turn-key" in that respect - I would highly value
>> such
>> > ideas and comments. If we can cut down something more that is not
>> > "necessary" for the three primary goals I came up with - I am more than
>> > happy to do it.
>> >
>> > Just to remind - those are the "extracted" goals. I slightly updated
>> them
>> > and added to the preamble of the AIP:
>> >
>> > * less operational overhead for managing multi-team (once AIP-72 is
>> > complete) where separate execution environments are important
>> > * virtual assets sharing between teams
>> > * ability of having "admin" and "team sharing" capability where dags
>> from
>> > multiple teams can be seen in a single Airflow UI  (requires custom
>> RBAC an
>> > AIP-56 implementation of Auth Manager - with KeyCloak Auth Manager
>> being a
>> > reference implementation)
>> >
>> > J.
>> >
>> >
>> > On Thu, Jun 26, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> >>
>> >>> One technical observation: Now that the dag table no longer has a
>> >>> team_id in it, what would the behaviour be when a DAG is attempted to
>> move
>> >>> between bundles? How do we detect this? (I’m not all convinced that we
>> >>> correctly detect duplicate dag ids across bundles today, so I wouldn’t
>> >>> assume or rely on the current behaviour.)
>> >>>
>> >>
>> >> Of course - yes, I realise that - that problem was also not handled in
>> >> the previous iteration to be honest. That is something that dag bundle
>> >> solution allows to solve eventually - but I do not think it's a
>> blocker for
>> >> the proposed implementation. We will have to eventually add some way of
>> >> blocking dags to jump between bundles, we might tackle this
>> separately. I
>> >> already wanted to propose a separate update to that - but I did not
>> want to
>> >> complicate the current proposal. One thing at a time. I can, however -
>> if
>> >> you consider that as a blocker, extend the current AIP with it. Not a
>> big
>> >> problem. This is however a bit independent from the team_id
>> introduction.
>> >>
>> >> Overall, I am still unconvinced this proposal has enough real user
>> >>> benefit over actually separate deployments, and on balance of the
>> added
>> >>> complexity and maintenance burden I do not think it is worth it.
>> >>>
>> >>
>> >> That makes me sad, I thought that over the course of the discussion I
>> >> addressed all the concerns (in this case the concern was "is it worth
>> with
>> >> the cost and little benefit", but when I did it and heavily limited the
>> >> impact, now the concern is "is it worth at all as changes are really
>> >> minimal" - and surely, anyone can change and adapt their concerns, over
>> >> time,  but that one seems like ever-moving target. I hoped at least for
>> >> some acknowledgment of some concerns (complexity in this case) is
>> >> addressed, but it seems that you are deeply convinced that we do not
>> need
>> >> multi-team at all (which is in stark contrast with at least a dozen of
>> >> bigger and smaller users of Airflow who submitted talks to Airflow
>> summit
>> >> (including about 5 or 6 submissions for Airflow 2025) on how they spent
>> >> their engineering effort, time and money on trying to achieve something
>> >> similar - they assessed that it's worth, you  assess that it's not
>> worth.
>> >> Somehow I trust our users that they were not spending the money, time
>> and
>> >> engineering effort to achieve this because they wanted to spend more
>> money.
>> >> I think they assessed it's worth it. So I want to make it a bit easier
>> and
>> >> more "proper" way for them to do that.
>> >>
>> >>>
>> >>> Upgrades: it is not easier to upgrade under this multi team proposal,
>> >>> but much much harder. This is based on hard earned experience from
>> helping
>> >>> Astronomer users — having to coordinate upgrades between multiple
>> teams
>> >>> turns in to a months long slog of the hardest kind of work —  people
>> work:
>> >>> getting other teams to agree to do things that they don’t directly
>> care
>> >>> about — “It’s working for me, I don’t care about upgrading, we’ll get
>> to it
>> >>> next quarter” is a refrain I’ve heard many times.
>> >>>
>> >>
>> >> Yes. absolutely - this is why we deferred it until we knew what shape
>> >> task isolation and other AIPs we depend on take on. Because it is clear
>> >> that pretty much all the problem you explain above are going to be
>> solved
>> >> with task isolation. And it's not just my opinion. If you want to argue
>> >> with it, you likely need to argue with yourself:
>> >> https://github.com/apache/airflow/issues/51545#issuecomment-2980038478
>> .
>> >> Let me quote what you wrote there last week:
>> >>
>> >> Ash Berlin Taylor wrote:
>> >>
>> >> > A tight coupling between task-sdk and any "server side" component is
>> >> the opposite to one of the goals of AIP-72 (I'm not sure we ever
>> explicitly
>> >> said this, but the first point of motivation for the AIP says
>> "Dependency
>> >> conflicts for administrators supporting data teams using different
>> versions
>> >> of providers, libraries, or python packages")
>> >> > In short, my goal with TaskSDK, and the reason for introducing CalVer
>> >> and Cadwyn with the execution API is to end up in a world where you can
>> >> upgrade the Airflow Scheduler/API server interdependently of any worker
>> >> nodes (with the exception that the server must be at least as new as
>> the
>> >> clients)
>> >> > This ability to have version-skew is pretty much non-negotiable to me
>> >> and is (other than other languages) one of primary benefits of AIP-72
>> >>
>> >> If you read yourself from that quote it basically means "it will be
>> easy
>> >> to upgrade airflow independently of workers". So I am a bit confused
>> here.
>> >> Yes, I agree it was difficult, but you yourself explain that when
>> AIP-72
>> >> (which since API-67 has been accepted has always beem prerequisite of
>> it)
>> >> wrote it will be "easy". So I am not sure why you are bringing it now.
>> We
>> >> assume AIP-72 will be completed and this problem will be gone. Let's
>> not
>> >> mention it any more please.
>> >>
>> >> The true separation from TaskSDK will likely only land in about 3.2
>> time
>> >>> frame. We are actively working on it, but it’s a slow process of
>> untangling
>> >>> lots of assumptions made in the code base over the years. Maybe once
>> we
>> >>> have that my view would be different, but right now I think this
>> makes the
>> >>> proposal a non-starter. Especially as you are saying that most teams
>> will
>> >>> have unique connections. If they’ve got those already, then having an
>> asset
>> >>> trigger use those conns to watch/poll for activity is a much easier
>> >>> solution to operate and crucially, to scale and upgrade.
>> >>>
>> >>
>> >> Yes. I perfectly understand that and I am fully aware of potentially
>> 3.2
>> >> time-frame. And that's fine. Actually I heartily invite you to listen
>> to
>> >> the part of my talk from Berlin Buzzwords when I was asked for the
>> timeline
>> >> - https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the exact
>> >> timeline in my talk . My answer was basically - "3.1" or "3.2", and I
>> >> sincerely hope "3.1" but we might not be able to complete it because we
>> >> have other things to do (other - is indeed the Task Isolation work
>> that you
>> >> are leading). And that's perfectly fine. And it absolutely does not
>> prevent
>> >> us from voting on the AIP now - similarly as we voted on the previous
>> >> version of the AIP - knowing that it has some prerequisites a few
>> months
>> >> ago. Especially that we know that the feature we need from task
>> isolation
>> >> is "non-negotiable". I.e. it WILL happen. We don't hope for it, we
>> know it
>> >> will be there. Those are your own words.
>> >>
>> >>
>> >>> >  I think we can’t compare AIP-82 to sharing virtual assets due to
>> >>> complexity of it.
>> >>>
>> >>> Virtual Assets was a mistake, and not how users actually want to use
>> >>> them. Mea culpa
>> >>>
>> >>
>> >> This is the first time I hear this - certainly you never raised this
>> >> concern on the devlist. So if you have some concerns about virtual
>> assets I
>> >> think you should raise it on the devlist, because I think everyone
>> here is
>> >> missing some conversation (or maybe it's just your private opinion that
>> >> you never shared with anyone, but maybe it's worth). I would be
>> >> interested to hear how the feature that was absolutely most successful
>> >> feature of airflow 2 was a mistake. According to the 2024 survey
>> >> https://airflow.apache.org/blog/airflow-survey-2024/  - 48% of Airflow
>> >> users have been using it, even if it was added as one of the last
>> >> big features of Airflow 2. It's the MOST used feature out of all the
>> >> features out there. I would be really curious to see how it was a
>> mistake
>> >> (but  please start a separate thread explaining why you think it was a
>> >> mistake, what are your data points and what do you think should be
>> fixed.
>> >> Just dropping "virtual assets were a mistake" in the middle of
>> multi-team
>> >> conversation seems completely unjustified without knowing what you are
>> >> talking about. So I think, until we know more, this argument has no
>> base.
>> >>
>> >>
>> >>>
>> >>> S
>> >>> To restate my points:
>> >>>
>> >>> - Sharing a deployment between teams today/in 3.1 is operationally
>> more
>> >>> complex (both scaling, and upgrades) — this is a con, not a plus.
>> >>>
>> >>
>> >> Surely. But it will be easier when AIP-72 is complete (which I am
>> >> definitely looking forward to and as clearly explained in AIP-82, is a
>> >> prerequisite of it). Nothing changed here.
>> >>
>> >>
>> >>> - The main user benefit appears to be “allow teams’ DAGs to
>> communicate
>> >>> via Assets”, in which case we can do that today by putting more work
>> in to
>> >>> AIP-82’s Asset triggers
>> >>>
>> >>
>> >> No. Lower operational complexity for multi-teams (providing that we
>> >> deliver AIP-72) is another benefit. Virtual assets is another, and
>> since
>> >> there is no ground in "virtual assets is a mistake" statement (not
>> until
>> >> you explain what you mean by that in a separate discussion) - this is
>> also
>> >> still a very valid point.
>> >>
>> >>
>> >>> Soon, we will have then be asked about cross-team governance, policy
>> >>> enforcement, and potentially unbounded edge cases (e.g., team-specific
>> >>> secrets, roles, quotas). ain, you get this for free with truely
>> separate
>> >>> deployments already
>> >>> allow different teams to use different executors (including multiple
>> >>> executors per-team following AIP-61)
>> >>>
>> >>
>> >> Not really. We very explicitly say in the AIP that his is not a goal
>> and
>> >> that we have no plans for. And yes, using separate executors per team
>> is
>> >> actually back in the AIP-82 in case you did not notice (and the code
>> needed
>> >> for it's even implemented and merged already in main by Vincent).
>> >>
>> >>
>> >>> Provably not true right now, and until ~3.2 delivers the full Task
>> >>> SDK/Core dependency separation this would be _more_ work to upgrade,
>> not
>> >>> less, and that work is not shared but still on a central team.
>> >>>
>> >>
>> >> Absolutely - we will wait for AIP-72 completion. I do not want to say
>> 3.1
>> >> or 3.2 directly - because there are - as you said - a lot of moving
>> pieces.
>> >> So my target for multi-team is "After AIP-72 is completed". Full stop.
>> But
>> >> there is nothing wrong with accepting the AIP now and doing preparatory
>> >> work in parallel. Similarly as there is no way to have a baby in 1
>> month by
>> >> 9 women, there is no way adding more effort to task-sdk isolation will
>> >> speed it up - we alredy have not only 3 people (you leading it, Kaxil
>> and
>> >> Amog) but also all the help from me and even 10s of different
>> contributors
>> >> (for example with the recent db_test cleanup that I took leadership
>> on) -
>> >> and there are people who wish to work on adding multi-team features.
>> Since
>> >> the design heavily limits impact on airflow codebase and interactions
>> with
>> >> task-sdk implementation, there is nothing wrong with starting
>> >> implementation in parallel either- amazon team is keen to move it
>> forward -
>> >> they even already implemented SQS trigger for assets, and we are
>> working
>> >> together on FAB removal, Keycloak authentication manager - and they
>> seem to
>> >> still have capacity and drive to progress multi-team. So I am not sure
>> if
>> >> we are trading off something. There is no "if we work on more on task
>> sdk
>> >> and drop multi-team things will be faster". Generally in open source
>> people
>> >> work in the area where they feel they can provide best value - such as
>> you
>> >> working on task-sdk, me on CI,dev env, they will deliver more value on
>> >> multi-team
>> >>
>> >>
>> >>>
>> >>> So please, as succinctly as possible, please tell me what the direct
>> >>> benefit to users this proposal is over us putting this effort in to
>> writing
>> >>> better Asset triggers instead?
>> >>>
>> >>
>> >>
>> >> * less operational overhead for managing multi-team (once AIP-72 is
>> >> complete) where separate execution environments are important
>> >> * virtual assets sharing
>> >> * ability of having "admin" and "team sharing" capability where dags
>> from
>> >> multiple teams can be seen in a single Airflow UI  (requires custom
>> RBAC)
>> >>
>> >> None of this can be done via beter asset triggers
>> >>
>> >>
>> >>>
>> >>> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>> >
>> >>> > My counter-points:
>> >>> >
>> >>> >
>> >>> >> 1. Managing a multi team deployment is not materially different
>> from
>> >>> >> managing a deployment per team
>> >>> >>
>> >>> >
>> >>> > It's a bit easier - especially when it comes to upgrades (especially
>> >>> in the
>> >>> > case we are targetting when we are not targetting multi-tenant, but
>> >>> several
>> >>> > relatively closely cooperating teams with different dependncy
>> >>> requiremens
>> >>> > and isolation need.
>> >>> >
>> >>> > 2. The database changes were quite wide-reaching
>> >>> >>
>> >>> >
>> >>> > Yes. that is addressed.
>> >>> >
>> >>> >
>> >>> >> 3. I don’t believe the original AIP (again, I haven’t read the
>> updated
>> >>> >> proposal or recent messages on the thread. yet) will meet what many
>> >>> users
>> >>> >> want out of a multiteam solution
>> >>> >>
>> >>> >
>> >>> > I think we will only see when we try. A lot of people thing they
>> would,
>> >>> > even if they are warned. I know at least one user (Wealthsimple) who
>> >>> > definitely want to use it and they got a very detailed explanation
>> of
>> >>> the
>> >>> > idea and understand it well. So I am sure that **some** users would.
>> >>> But we
>> >>> > do not know how many.
>> >>> >
>> >>> >
>> >>> >> To expand on those points a bit more
>> >>> >>
>> >>> >> On 1. The only components that are shared are, I think, the
>> scheduler
>> >>> and
>> >>> >> the API server, and it’s arguable if that is actually a good idea
>> >>> given
>> >>> >> those are likely to be the most performance sensitive components
>> >>> anyway.
>> >>> >>
>> >>> >> Additionally the fact that the scheduler is a shared component
>> makes
>> >>> >> upgrading it almost a non starter as you would likely need buy-in,
>> >>> changes,
>> >>> >> and testing form ALL teams using it. I’d argue that this is a huge
>> >>> negative
>> >>> >> until we finish off the version indepence work of AIP-72.
>> >>> >>
>> >>> >
>> >>> > Quite disagree here - especially that our target is that task-sdk is
>> >>> > supposed to provide all isolation that is needed. There should be 0
>> >>> changes
>> >>> > in the dags needed to upgrade scheduler, api_server, triggerer -
>> >>> precisely
>> >>> > because we introduced backwards-compatible task-sdk.
>> >>> >
>> >>> > On 3 my complaint is essentially that this doesn’t go nearly far
>> >>> enough. It
>> >>> >> doesn’t allow read only views to other teams dags. I don’t think it
>> >>> allows
>> >>> >> you to be in multiple teams at once. You can’t share a connection
>> >>> between
>> >>> >> teams but only allow certain specified dags to access it, but would
>> >>> have to
>> >>> >> either be globally usable, or duplicated-and-kept-in-sync between
>> >>> teams. In
>> >>> >> short I think it fall short of being useful..
>> >>> >>
>> >>> >
>> >>> > Oh absolutely all that is possible (except sharing single
>> connections
>> >>> > between multiple teams - which is a very niche use cases and
>> >>> duplication
>> >>> > here is perfectly ok as first approximation - and if we need more we
>> >>> can
>> >>> > add it later).
>> >>> >
>> >>> > Auth manager RBAC and access is abstracted away, and the Keyclock
>> >>> Manager
>> >>> > implemented by Vincent allows to manage completely independent and
>> >>> separate
>> >>> > RBAC based on arguments and resources provided by Airflow. There is
>> >>> nothing
>> >>> > to prevent the user who configures KeyCloak RBAC to define it in the
>> >>> way:
>> >>> >
>> >>> > if group a > allow to read a and write b
>> >>> > if group b > alllow to write b but not a
>> >>> >
>> >>> > and any other combinations. KeyCloak implementation - pretty
>> advanced
>> >>> > already - (and design of auth manager) completely abstracts away
>> both
>> >>> > authentication and authorization to KeyCloak and KeyCloak has RBAC
>> >>> > management built in. Also any of the users can write their own -
>> even
>> >>> > hard-coded authentication manager to do the same if they do not
>> want to
>> >>> > have configurable KeyCloak. Even SimpleAuthManager could be
>> hard-coded
>> >>> to
>> >>> > provide thiose features.
>> >>> >
>> >>> >
>> >>> >>
>> >>> >> So on the surface, I’m no more in favour of using dag bundle as a
>> >>> >> replacement for team id as I think most of the above points still
>> >>> stand.
>> >>> >>
>> >>> >
>> >>> > We disagree here.
>> >>> >
>> >>> >>
>> >>> >> My counter proposal: We do _nothing_ to core airflow. We work on
>> >>> improving
>> >>> >> the event-based trigger o fdags (write more triggers for read/check
>> >>> remote
>> >>> >> Assets etc) so that teams can have 100% isolated deployments but
>> still
>> >>> >> trigger dags based on asset events from other teams.
>> >>> >>
>> >>> >
>> >>> > That does not solve any of the other design goals - only allows to
>> >>> trigger
>> >>> > assets a bit more easily (but also it's not entirely solved by
>> AIP-82
>> >>> > because it does not solve virtual assets - only ones that have
>> defined
>> >>> > triggerer and "something" to listen on - which is way more complex
>> than
>> >>> > just defining asset in a Dag and using it in another). I think we
>> can't
>> >>> > compare AIP-82 to sharing virtual assets due to complexity of it. I
>> >>> > explained it in the doc.
>> >>> >
>> >>> >
>> >>> > I will now go and catch up with the long thread and updated proposal
>> >>> and
>> >>> >> come back.
>> >>> >>
>> >>> >
>> >>> > Please. I hope the above explaination will help in better
>> >>> understanding of
>> >>> > the proposal, because I think you had some assumptions that do not
>> >>> hold any
>> >>> > more with the new proposal.
>> >>> >
>> >>> > J.
>> >>> >
>> >>> >
>> >>> >>
>> >>> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>> >>>
>> >>> >>> Just to clarify the relation - I updated the AIP now to refer to
>> >>> AIP-82
>> >>> >> and
>> >>> >>> to explain relation between the "cross-team" and "cross-airflow"
>> >>> asset
>> >>> >>> triggering - this is what I added:
>> >>> >>>
>> >>> >>> Note that there is a relation between AIP-82 ("External Driven
>> >>> >> Scheduling")
>> >>> >>> and this part of the functionality. When you have multiple
>> instances
>> >>> of
>> >>> >>> Airflow, you can use shared datasets - "Physical datasets" - that
>> >>> several
>> >>> >>> Airflow Instances can use - for example there could be an S3
>> object
>> >>> that
>> >>> >> is
>> >>> >>> produced by one airflow instance, and consumed by another. That
>> >>> requires
>> >>> >>> deferred trigger to monitor for such datasets, and appropriate
>> >>> >> permissions
>> >>> >>> to the external dataset, and you could achive similar result to
>> >>> >> cross-team
>> >>> >>> dataset triggering (but cross airflow). However the feature of
>> >>> sharing
>> >>> >>> datasets between the teams also works for virtual assets, that do
>> not
>> >>> >> have
>> >>> >>> physically shared "objects" and trigger that is monitoring for
>> >>> changes in
>> >>> >>> such asset.
>> >>> >>>
>> >>> >>> J.
>> >>> >>>
>> >>> >>>
>> >>> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com>
>> >>> wrote:
>> >>> >>>
>> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>> >>> reference
>> >>> >> to
>> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>> >>> through.
>> >>> >>>>
>> >>> >>>> Yep. It did not - because I did not think it was needed or even
>> very
>> >>> >>>> important after the simplifications. AIP-82 has a different
>> scope,
>> >>> >> really.
>> >>> >>>> It only helps when the Assets are "real" data files which we have
>> >>> >> physical
>> >>> >>>> triggers for, it's slightly related - sharing datasets between
>> teams
>> >>> >>>> (including those that do not require physical files and
>> triggers) is
>> >>> >> still
>> >>> >>>> possible in the design we have now, but it's not (and never was)
>> the
>> >>> >>>> **only** reason for having multi-team. There always was (and
>> still
>> >>> is)
>> >>> >> the
>> >>> >>>> possibility of having a common, distinct environments (i.e.
>> >>> dependencies
>> >>> >>>> and providers) per team, the possibility of having connections
>> and
>> >>> >>>> variables that are only accessible to one team and not the other,
>> >>> and
>> >>> >>>> isolating workload execution (all that while allowing to manage
>> >>> multiple
>> >>> >>>> team and schedule things with single deployment). That did not
>> >>> change.
>> >>> >> What
>> >>> >>>> changed a lot is that it is now way simpler, something that we
>> can
>> >>> >>>> implement without heavy changes to the codebase - and give it to
>> our
>> >>> >> users,
>> >>> >>>> so that they can assess if this is something they need without
>> too
>> >>> much
>> >>> >>>> risk and effort.
>> >>> >>>>
>> >>> >>>> This was - I believe the main concern, that the value we get from
>> >>> it is
>> >>> >>>> not dramatic, but the required changes are huge. This "redesign"
>> >>> changes
>> >>> >>>> the equation - the value is still unchanged, but the cost of
>> >>> >> implementing
>> >>> >>>> it and impact on the Airflow codebase is much smaller. I still
>> have
>> >>> not
>> >>> >>>> heard back from Ash if my proposal responds to his original
>> concern
>> >>> >> though,
>> >>> >>>> so I am mostly guessing (also based on the positive impact of
>> >>> others)
>> >>> >> that
>> >>> >>>> yes it does. But to be honest I am not sure and I would love to
>> hear
>> >>> >> back,
>> >>> >>>> I decided to update the AIP to reflect it - regardless, because I
>> >>> think
>> >>> >> the
>> >>> >>>> simplification I proposed keeps the original goals, but is indeed
>> >>> way
>> >>> >>>> simpler.
>> >>> >>>>
>> >>> >>>>> This is a very difficult thread to catch up on.
>> >>> >>>>
>> >>> >>>> Valid point. Let me summarize what is the result:
>> >>> >>>>
>> >>> >>>> * I significantly simplified the implementation proposal
>> comparing
>> >>> to
>> >>> >> the
>> >>> >>>> original version
>> >>> >>>> * main simplification is very limited impact on existing
>> database -
>> >>> >>>> without "ripple effect" that would require us to change a lot of
>> >>> tables,
>> >>> >>>> including their primary keys, and heavily impact the UI
>> >>> >>>> * this is now more of an incremental change that can be
>> implemented
>> >>> way
>> >>> >>>> faster and with far less risk
>> >>> >>>> * updated idea is based on leveraging bundles (already part of
>> our
>> >>> data
>> >>> >>>> model) to map them (many-to-one) to a team - which requires to
>> just
>> >>> >> extend
>> >>> >>>> the data model with bundle mapping and add team_id to connections
>> >>> and
>> >>> >>>> variables. Those are all needed DB changes.
>> >>> >>>>
>> >>> >>>> The AIP is updated - in a one single big change so It should be
>> >>> easy to
>> >>> >>>> compare the changes:
>> >>> >>>>
>> >>> >>
>> >>>
>> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
>> >>> >>>> -> I even named the version appropriately "Simplified multi-team
>> >>> AIP" -
>> >>> >> you
>> >>> >>>> can select and compare v.65 with v.66 to see the exact
>> differences I
>> >>> >>>> proposed.
>> >>> >>>>
>> >>> >>>> I hope it will be helpful to catch up and for those who did not
>> >>> follow,
>> >>> >> to
>> >>> >>>> be able to make up their minds about it.
>> >>> >>>>
>> >>> >>>> J.
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
>> >>> >> <vi...@astronomer.io.invalid>
>> >>> >>>> wrote:
>> >>> >>>>
>> >>> >>>>> This is a very difficult thread to catch up on.
>> >>> >>>>> I will take a detailed look at the AIP update to try to figure
>> out
>> >>> the
>> >>> >>>>> changes in the proposal.
>> >>> >>>>>
>> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>> >>> reference
>> >>> >> to
>> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>> >>> through.
>> >>> >>>>>
>> >>> >>>>> Vikram
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>> >>> >>>>> gopidesupavan@gmail.com>
>> >>> >>>>> wrote:
>> >>> >>>>>
>> >>> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's much
>> >>> more
>> >>> >> slim
>> >>> >>>>>> down.
>> >>> >>>>>>
>> >>> >>>>>> left a minor comment. :) Overall looking great.
>> >>> >>>>>>
>> >>> >>>>>> Pavan
>> >>> >>>>>>
>> >>> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>> >>> >>>>> <j_scheffler@gmx.de.invalid
>> >>> >>>>>>>
>> >>> >>>>>> wrote:
>> >>> >>>>>>
>> >>> >>>>>>> Thanks for the rework/update of the AIP-72!
>> >>> >>>>>>>
>> >>> >>>>>>> Just a few small comments but overall I like it as it is much
>> >>> leaner
>> >>> >>>>>>> than originally planned and is in a level of complexity that
>> it
>> >>> >> really
>> >>> >>>>>>> seems to be a benefit to close the gap as described.
>> >>> >>>>>>>
>> >>> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
>> >>> >>>>>>>> I updated the AIP - including architecture images and
>> reviewed
>> >>> it
>> >>> >>>>>> (again)
>> >>> >>>>>>>> and corrected any ambiguities and places where it needed to
>> be
>> >>> >>>>> changed.
>> >>> >>>>>>>>
>> >>> >>>>>>>> I think the current state
>> >>> >>>>>>>>
>> >>> >>>>>>>
>> >>> >>>>>>
>> >>> >>>>>
>> >>> >>
>> >>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>> >>> >>>>>>>> - nicely describes the proposal.
>> >>> >>>>>>>>
>> >>> >>>>>>>> Comparing to the previous one:
>> >>> >>>>>>>>
>> >>> >>>>>>>> 1. The DB changes are far less intrusive - no ripple effect
>> on
>> >>> >>>>> Airflow
>> >>> >>>>>>>> 2. There is no need to merge configurations and provide
>> >>> different
>> >>> >>>>> set
>> >>> >>>>>> of
>> >>> >>>>>>>> configs per team - we can add it later but I do not see why
>> we
>> >>> need
>> >>> >>>>> it
>> >>> >>>>>> in
>> >>> >>>>>>>> this simplified version
>> >>> >>>>>>>> 3. We can still configure a different set of executors per
>> team
>> >>> -
>> >>> >>>>> that
>> >>> >>>>>> is
>> >>> >>>>>>>> already implemented (we just need to wire it to the bundle ->
>> >>> team
>> >>> >>>>>>> mapping).
>> >>> >>>>>>>>
>> >>> >>>>>>>> I think it will be way simpler and faster to implement this
>> way
>> >>> and
>> >>> >>>>> it
>> >>> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can
>> >>> give
>> >>> >>>>> our
>> >>> >>>>>>>> users so that they can provide feedback.
>> >>> >>>>>>>>
>> >>> >>>>>>>> J.
>> >>> >>>>>>>>
>> >>> >>>>>>>>
>> >>> >>>>>>>>
>> >>> >>>>>>>>
>> >>> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <
>> jarek@potiuk.com>
>> >>> >>>>> wrote:
>> >>> >>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>> I like this iteration a bit more now for sure, thanks for
>> >>> being
>> >>> >>>>>>> receptive
>> >>> >>>>>>>>>> to feedback! :)
>> >>> >>>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>> This now becomes quite close to what was proposing before,
>> we
>> >>> now
>> >>> >>>>>> again
>> >>> >>>>>>>>>> have a team ID (which I think is really needed here, glad
>> to
>> >>> see
>> >>> >>>>> it
>> >>> >>>>>>> back)
>> >>> >>>>>>>>>> and it will be used for auth management, configuration
>> >>> >>>>> specification,
>> >>> >>>>>>> etc
>> >>> >>>>>>>>>> but will be carried by Bundle instead of the dag model.
>> Which
>> >>> as
>> >>> >>>>> you
>> >>> >>>>>>> say
>> >>> >>>>>>>>>> “For that we will need to make sure that both api-server,
>> >>> >>>>> scheduler
>> >>> >>>>>> and
>> >>> >>>>>>>>>> triggerer have access to the "bundle definition" (to
>> perform
>> >>> the
>> >>> >>>>>>> mapping)"
>> >>> >>>>>>>>>> which honestly doesn’t feel too much different from the
>> >>> original
>> >>> >>>>>>> proposal
>> >>> >>>>>>>>>> we had last week of adding it to Dag table and ensuring
>> it’s
>> >>> >>>>>> available
>> >>> >>>>>>>>>> everywhere. but either way I’m happy to meet in the middle
>> and
>> >>> >>>>> keep
>> >>> >>>>>> it
>> >>> >>>>>>> on
>> >>> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable
>> location.
>> >>> >>>>>>>>>>
>> >>> >>>>>>>>> I think the big difference is the "ripple effect" that was
>> >>> >>>>> discussed
>> >>> >>>>>> in
>> >>> >>>>>>>>>
>> >>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>> >>> >>>>>> (and I
>> >>> >>>>>>>>> believe - correct me if I am wrong Ash - important trigger
>> for
>> >>> the
>> >>> >>>>>>>>> discussion) so far what we wanted is to extend the primary
>> key
>> >>> and
>> >>> >>>>> it
>> >>> >>>>>>> would
>> >>> >>>>>>>>> ripple through all the pieces of Airflow -> models, API, UI
>> >>> etc.
>> >>> >>>>> ...
>> >>> >>>>>>>>> However - we already have `bundle_name" and "bundle_version"
>> >>> in the
>> >>> >>>>>> Dag
>> >>> >>>>>>>>> model. So I think when we add a separate table where we map
>> the
>> >>> >>>>> bundle
>> >>> >>>>>>> to
>> >>> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not
>> want
>> >>> to
>> >>> >>>>>> change
>> >>> >>>>>>>>> primary key, we do not want to change UI in any way (except
>> >>> >>>>> filtering
>> >>> >>>>>> of
>> >>> >>>>>>>>> DAGs available based on your team - but that will be
>> handled in
>> >>> >>>>> Auth
>> >>> >>>>>>>>> Manager and will not impact UI in any way, I think that's a
>> >>> huge
>> >>> >>>>>>>>> simplification of the implementation, and if we agree to it
>> - i
>> >>> >>>>> think
>> >>> >>>>>> it
>> >>> >>>>>>>>> should speed up the implementation significantly. There are
>> >>> only a
>> >>> >>>>>>> limited
>> >>> >>>>>>>>> number of times where you need to look up the team_id - so
>> >>> having
>> >>> >>>>> the
>> >>> >>>>>>>>> bundle -> team mapping in a separate table and having to
>> look
>> >>> them
>> >>> >>>>> up
>> >>> >>>>>>>>> should not be a problem. And it has much less complexity and
>> >>> >>>>>>>>> "ripple-effect" through the codebase (for example I could
>> >>> imagine
>> >>> >>>>> 100s
>> >>> >>>>>>> or
>> >>> >>>>>>>>> thousands already written tests that would have to be
>> adapted
>> >>> if we
>> >>> >>>>>>> changed
>> >>> >>>>>>>>> the primary key - where there will be pretty much zero
>> impact
>> >>> on
>> >>> >>>>>>> existing
>> >>> >>>>>>>>> tests if we just add bundle -> team lookup table.
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>> One other thing I’d point out is that I think including
>> >>> executors
>> >>> >>>>> per
>> >>> >>>>>>>>>> team is a very easy win and quite possible without much
>> work.
>> >>> I
>> >>> >>>>>> already
>> >>> >>>>>>>>>> have much of the code written. Executors are already aware
>> of
>> >>> >>>>> Teams
>> >>> >>>>>>> that
>> >>> >>>>>>>>>> own them (merged), I have a PR open to have configuration
>> per
>> >>> team
>> >>> >>>>>>> (with a
>> >>> >>>>>>>>>> quite simple and isolated approach, I believe you approved
>> >>> Jarek).
>> >>> >>>>>> The
>> >>> >>>>>>> last
>> >>> >>>>>>>>>> piece is updating the scheduling logic to route tasks from
>> a
>> >>> >>>>>> particular
>> >>> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much
>> work
>> >>> >>>>> (though
>> >>> >>>>>> it
>> >>> >>>>>>>>>> would be easier if the Task models had a column for the
>> team
>> >>> they
>> >>> >>>>>>> belong
>> >>> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to get
>> >>> the
>> >>> >>>>>> team) I
>> >>> >>>>>>>>>> have a branch where I was experimenting with this logic
>> >>> already.
>> >>> >>>>>>>>>> Any who, long story short, I don’t think we necessarily
>> need
>> >>> to
>> >>> >>>>>> remove
>> >>> >>>>>>>>>> this piece from the project's scope if it is already partly
>> >>> done
>> >>> >>>>> and
>> >>> >>>>>>> not
>> >>> >>>>>>>>>> too difficult.
>> >>> >>>>>>>>>>
>> >>> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want to
>> just
>> >>> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we
>> have
>> >>> it
>> >>> >>>>> in.
>> >>> >>>>>>> And
>> >>> >>>>>>>>> if we can make it in, easily (which it seems we can) - we
>> can
>> >>> also
>> >>> >>>>>>> include
>> >>> >>>>>>>>> it in the first iteration. What I wanted to avoid really
>> (from
>> >>> the
>> >>> >>>>>>> original
>> >>> >>>>>>>>> design) - again trying to simplify it, limit the changes,
>> and
>> >>> >>>>> speed up
>> >>> >>>>>>>>> implementation. And there is one "complexity" that I wanted
>> to
>> >>> >>>>> avoid
>> >>> >>>>>>>>> specifically - having to have separate , additional
>> >>> configuration
>> >>> >>>>> per
>> >>> >>>>>>> team.
>> >>> >>>>>>>>> Not only because it complicates already complex
>> configuration
>> >>> >>>>> handling
>> >>> >>>>>>> (I
>> >>> >>>>>>>>> know we have PR for that) but mostly because if it is not
>> >>> needed,
>> >>> >>>>> we
>> >>> >>>>>> can
>> >>> >>>>>>>>> simplify documentation and explain to our users easier what
>> >>> they
>> >>> >>>>> need
>> >>> >>>>>>> to do
>> >>> >>>>>>>>> to have their own multi-team setup. And I am quite open to
>> >>> keeping
>> >>> >>>>>>>>> multiple-executors if we can avoid complicating
>> configuration.
>> >>> >>>>>>>>>
>> >>> >>>>>>>>> But I think some details of that and whether we really need
>> >>> >>>>> separate
>> >>> >>>>>>>>> configuration might also come as a result of updating the
>> AIP
>> >>> - I
>> >>> >>>>> am
>> >>> >>>>>> not
>> >>> >>>>>>>>> quite sure now if we need it, but we can discuss it when we
>> >>> >>>>> iterate on
>> >>> >>>>>>> the
>> >>> >>>>>>>>> AIP.
>> >>> >>>>>>>>>
>> >>> >>>>>>>>> J.
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>
>> >>> >>>>>>>
>> >>> ---------------------------------------------------------------------
>> >>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >>> >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>> >>> >>>>>>>
>> >>> >>>>>>>
>> >>> >>>>>>
>> >>> >>>>>
>> >>> >>>>
>> >>> >>
>> >>> >>
>> >>> >>
>> ---------------------------------------------------------------------
>> >>> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >>> >> For additional commands, e-mail: dev-help@airflow.apache.org
>> >>>
>> >>>
>>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

I realized that I owe Niko an explanation of configuration changes. Again -
following the philosophy above - minimal set of changes to "airflow
internals". the "minimum" set of changes that will work. I propose the
change below that has **no** changes to the way how the
current configuration "shared" feature works - it will change the way
executors will retrieve their configuration if they are configured
"per-team" - and we can 100% bank on existing multi-executors.
I believe that will absolutely minimise the set of changes needed to
implement multi-team and we will be able to get it "faster" and with "far
lower risk" of impacting airflow code and say - 3.1 or 3.2 delivery.

Existing multi-executor configuration will be extended to include team
prefix. The prefix will be separated with ":", entries for different teams
will be separated with ";"

[core]

executor = team1:KubernetesExecutor,my.custom.module.Executor
Class;team2:CeleryExecutor

The configuration of executors will also be prefixed with the same team:

[team1:kubernetes_executor]

api_client_retry_configuration = { "total": 3, "backoff_factor": 0.5 }

The environment variables keeping configuration will use ___  (three
underscores) to replace ":". For example:

AIRFLOW__TEAM1___KUBERNETES_EXECUTOR__API_CLIENT_RETRY_CONFIGURATION`


J.

On Thu, Jul 3, 2025 at 8:47 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> > The direction this one is taking is interesting. If you're really just
> trying to make the feature barely possible and mostly targeted towards
> managed providers to implement the rest, then I suppose this hits the mark.
>
> Well actually by taking the direction I took, it's not "mostly for managed
> providers" - i see it as it is equally, for managed providers and on-prem
> users, but also, following the open-source spirit, philosophically, I think
> in Airflow, any such change should be done with those things in mind,
> because we are at the stage where we are already "established' and by
> innovating on top what we have we have sometimes more to lose than to gain
> - so I feel with "deployment' features we should be very careful to
> distinguish 'enabling things" vs. 'doing things". My focus with this
> iteration was to remove all the roadblocks that make it impossible (or
> extremely difficult) to implement "real" multi-team and separation without
> modifying airflow core. I though "what is the minimal set of features that
> will make it "possible" for someone motivated to deploy a single airflow
> for multiple teams.
>
> * minimise maintenance effort increase
> * do not "spoil" the "simple case" - we do not want to add features that
> make "simple" implementation more complex the current `docker run -it
> apache/airflow standalone` - should be simple and straightforward to run
> * if there is anything that involves complex deployment, we should not aim
> to make a "turn-key" solution that we will have to support - similarly like
> we do with our configuration parameters, we have 100s knobs to turn, and as
> long as default settings are reasonable and someone "motivated" can
> configure and fine-tune - this configuration and fine-tuning should be left
> to them - regardless if they are on-prem or managed. And both should be
> able to do it.
>
> I think it's not only smart technically (we support the low-level basic
> features and when someone puts them together and makes it more of a
> turn-key solution they are responsible for designing and implementing it -
> so we have less maintenance effort. But also it's good from a simple
> "open-source business model point of view" - i.e. it's a smart product
> decision we should make.
>
> Why airflow is #18 in OSS rank - of course we have a huge community and
> people contributing in their free time, completely voluntarily. And we
> cherish, support and encourage it. But let's be honest - if not all those
> that make business on top of airflow did not invest literally millions of
> dollars (in terms of engineering salary, sponsoring Airflow Summit,
> supporting people like me (some smart stakeholders at least who understand
> the value of it) who can be good "community spirit" - Airflow would have
> order of magnitude less activity, reach, Airflow 3 would not be simply
> possible. And this is a good thing that we have those stakeholders that are
> interested and make money by turning Airflow into a "turn-key" solution.
> This is a fantastic, symbiotic relationship.
>
> So - what my thinking is - we should NOT make things that make airflow
> more turn-key for those complex cases. We should leave it up to those who
> want to make it and want to charge money for it. This is cool and great
> that they can do - and we should not do it "for them" - but on the other
> hand - we should make it possible that those who want to turn airflow into
> more complex (say multi-team solution) to make it happen - by providing
> them with minimal set of features that make it possible.
>
> And that also - in a way - keeps the balance between on-prem and managed
> implementation.
>
> Something that I've learned as a rule of thumb is that making a feature
> "generic" compared to custom implementation is 3x-10x more expensive (both
> in implementation and maintenance). And it means that if an on-prem user
> wants to implement something for them (say turn-key multi-team solution for
> their case) it will cost `x` , but when a managed provider wants to
> implement a generic multi-team it will cost `10x`. But also managed
> providers can spread the cost over the premium they will charge to their
> users so that they don't have to manage Airflow on their own and pay `x`
> for this mult-team feature to develop on their own. And this is a "fair"
> choice to make by on-prem users. They might choose what they want to do
> then. Also it's fair for managed provider - yes they need to invest more,
> but also they have a chance to shine on promoting it and making it more
> optimised at scale etc. etc.
>
> That is my line of thinking.
>
>
> J.
>
>
> On Thu, Jul 3, 2025 at 1:41 AM Oliveira, Niko <on...@amazon.com.invalid>
> wrote:
>
>> Hey Jarek,
>>
>>
>> The direction this one is taking is interesting. If you're really just
>> trying to make the feature barely possible and mostly targeted towards
>> managed providers to implement the rest, then I suppose this hits the mark.
>>
>> But this is not something we're asking for at Amazon and personally I
>> think we should make the feature reasonably usable for those running
>> self-managed OSS Airflow as well. There are many users running an on-prem
>> Airflow. Getting too hyper-fixated on an implementation that's so
>> simplified that it's obtuse and difficult to use by most users seems like
>> the wrong approach to me. But you and I have already discussed this at
>> length and I haven't convinced you so far, so if I'm the only one with this
>> thinking then I'm happy to disagree and commit as we say at Amazon :)
>>
>>
>> > So I would be rather strong on **not** touching the current
>> configuration and
>>
>> simply adding configuration for per-team executors in executor config -
>> even if it is uglier and more "low-level".
>>
>> Can you explain what "adding configuration for per-team executors in
>> executor config" would look like? I don't have a concrete sense of what you
>> mean by this.
>>
>> Thanks for your efforts on trying to get this feature agreed to and voted
>> on. Looking forward to working on the project in the coming weeks!
>>
>> Cheers,
>> Niko
>>
>> ________________________________
>> From: Jarek Potiuk <ja...@potiuk.com>
>> Sent: Tuesday, July 1, 2025 10:26:55 PM
>> To: dev@airflow.apache.org
>> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External
>> event driven dags) exists
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>>
>>
>>
>> Any last comments ? There is a long weekend coming up in the US, so I will
>> likely start voting on the updated AIP on Monday 7th.
>>
>> On Fri, Jun 27, 2025 at 12:41 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > I'd really love to finalise discussion and put it up to a vote some time
>> > after the recording from the last dev call is posted - so that more
>> > context, details and the LONG discussion we had on it. There is no
>> *huge*
>> > hurry  - we have strong dependency on Task Isolation and it seems that
>> it
>> > will still take a bit of time to complete, so I'd say I would love to
>> start
>> > voting in about a week time - so that maybe at the next dev call we can
>> > "seal" the subject. Happy to see any more comments - especially from
>> those
>> > who have opinions but they had no opportunity to express them.
>> >
>> > I am personally very happy with the direction it took - simplification
>> and
>> > "MVP" kind of approach - also I invite the stakeholders of ours to take
>> a
>> > close look at the scope and what we really propose - I have a feeling
>> that
>> > we can balance it out - there is something we can make to make it not
>> > "worse" for the offerings they have. I think we have a really good
>> > symbiotic relationship here, and I would love to leverage that. For one
>> -
>> > my goal here is to have a minimum number of changes that are impacting
>> > maintainability of the open-source airflow - but mostly "opening up some
>> > possibilities" - rather than provide turn-key solutions. And mostly
>> because
>> > this is good for all sides - less maintenance and complexity for OSS
>> > maintainers, but more opportunities to make it into "turn-key"
>> solutions by
>> > the stakeholders, while also allowing the "on-prem" users - if they are
>> > highly motivated - to use those features by adding the "turn-key" layer
>> on
>> > their own. Also adding multi-team should not be at the expense of
>> "simple"
>> > installations - they should be virtually unaffected.
>> >
>> > One example of applying this is cutting on "separate config files". I
>> > think it moves us closer to a "turn-key" solution but it is not really
>> > necessary to achieve the three goals above - that's why in the current
>> > proposal this part is completely removed - Sorry Niko, but I still think
>> > it's one of the things that falls into this bucket. We can easily remove
>> > it, they complicate code, documentation and options the users have, and
>> > even if it is a "little" more complex to manage configuration by
>> motivated
>> > users, it's also an opportunity for "turn-key" option that stakeholders
>> can
>> > build in their products - and we do not have to maintain it in the
>> > open-source. So I would be rather strong on **not** touching the current
>> > configuration and simply adding configuration for per-team executors in
>> > executor config - even if it is uglier and more "low-level".
>> >
>> > So if there are some constructive ideas on what can be done to make it
>> > "simpler" and less "turn-key" in that respect - I would highly value
>> such
>> > ideas and comments. If we can cut down something more that is not
>> > "necessary" for the three primary goals I came up with - I am more than
>> > happy to do it.
>> >
>> > Just to remind - those are the "extracted" goals. I slightly updated
>> them
>> > and added to the preamble of the AIP:
>> >
>> > * less operational overhead for managing multi-team (once AIP-72 is
>> > complete) where separate execution environments are important
>> > * virtual assets sharing between teams
>> > * ability of having "admin" and "team sharing" capability where dags
>> from
>> > multiple teams can be seen in a single Airflow UI  (requires custom
>> RBAC an
>> > AIP-56 implementation of Auth Manager - with KeyCloak Auth Manager
>> being a
>> > reference implementation)
>> >
>> > J.
>> >
>> >
>> > On Thu, Jun 26, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> >>
>> >>> One technical observation: Now that the dag table no longer has a
>> >>> team_id in it, what would the behaviour be when a DAG is attempted to
>> move
>> >>> between bundles? How do we detect this? (I’m not all convinced that we
>> >>> correctly detect duplicate dag ids across bundles today, so I wouldn’t
>> >>> assume or rely on the current behaviour.)
>> >>>
>> >>
>> >> Of course - yes, I realise that - that problem was also not handled in
>> >> the previous iteration to be honest. That is something that dag bundle
>> >> solution allows to solve eventually - but I do not think it's a
>> blocker for
>> >> the proposed implementation. We will have to eventually add some way of
>> >> blocking dags to jump between bundles, we might tackle this
>> separately. I
>> >> already wanted to propose a separate update to that - but I did not
>> want to
>> >> complicate the current proposal. One thing at a time. I can, however -
>> if
>> >> you consider that as a blocker, extend the current AIP with it. Not a
>> big
>> >> problem. This is however a bit independent from the team_id
>> introduction.
>> >>
>> >> Overall, I am still unconvinced this proposal has enough real user
>> >>> benefit over actually separate deployments, and on balance of the
>> added
>> >>> complexity and maintenance burden I do not think it is worth it.
>> >>>
>> >>
>> >> That makes me sad, I thought that over the course of the discussion I
>> >> addressed all the concerns (in this case the concern was "is it worth
>> with
>> >> the cost and little benefit", but when I did it and heavily limited the
>> >> impact, now the concern is "is it worth at all as changes are really
>> >> minimal" - and surely, anyone can change and adapt their concerns, over
>> >> time,  but that one seems like ever-moving target. I hoped at least for
>> >> some acknowledgment of some concerns (complexity in this case) is
>> >> addressed, but it seems that you are deeply convinced that we do not
>> need
>> >> multi-team at all (which is in stark contrast with at least a dozen of
>> >> bigger and smaller users of Airflow who submitted talks to Airflow
>> summit
>> >> (including about 5 or 6 submissions for Airflow 2025) on how they spent
>> >> their engineering effort, time and money on trying to achieve something
>> >> similar - they assessed that it's worth, you  assess that it's not
>> worth.
>> >> Somehow I trust our users that they were not spending the money, time
>> and
>> >> engineering effort to achieve this because they wanted to spend more
>> money.
>> >> I think they assessed it's worth it. So I want to make it a bit easier
>> and
>> >> more "proper" way for them to do that.
>> >>
>> >>>
>> >>> Upgrades: it is not easier to upgrade under this multi team proposal,
>> >>> but much much harder. This is based on hard earned experience from
>> helping
>> >>> Astronomer users — having to coordinate upgrades between multiple
>> teams
>> >>> turns in to a months long slog of the hardest kind of work —  people
>> work:
>> >>> getting other teams to agree to do things that they don’t directly
>> care
>> >>> about — “It’s working for me, I don’t care about upgrading, we’ll get
>> to it
>> >>> next quarter” is a refrain I’ve heard many times.
>> >>>
>> >>
>> >> Yes. absolutely - this is why we deferred it until we knew what shape
>> >> task isolation and other AIPs we depend on take on. Because it is clear
>> >> that pretty much all the problem you explain above are going to be
>> solved
>> >> with task isolation. And it's not just my opinion. If you want to argue
>> >> with it, you likely need to argue with yourself:
>> >> https://github.com/apache/airflow/issues/51545#issuecomment-2980038478
>> .
>> >> Let me quote what you wrote there last week:
>> >>
>> >> Ash Berlin Taylor wrote:
>> >>
>> >> > A tight coupling between task-sdk and any "server side" component is
>> >> the opposite to one of the goals of AIP-72 (I'm not sure we ever
>> explicitly
>> >> said this, but the first point of motivation for the AIP says
>> "Dependency
>> >> conflicts for administrators supporting data teams using different
>> versions
>> >> of providers, libraries, or python packages")
>> >> > In short, my goal with TaskSDK, and the reason for introducing CalVer
>> >> and Cadwyn with the execution API is to end up in a world where you can
>> >> upgrade the Airflow Scheduler/API server interdependently of any worker
>> >> nodes (with the exception that the server must be at least as new as
>> the
>> >> clients)
>> >> > This ability to have version-skew is pretty much non-negotiable to me
>> >> and is (other than other languages) one of primary benefits of AIP-72
>> >>
>> >> If you read yourself from that quote it basically means "it will be
>> easy
>> >> to upgrade airflow independently of workers". So I am a bit confused
>> here.
>> >> Yes, I agree it was difficult, but you yourself explain that when
>> AIP-72
>> >> (which since API-67 has been accepted has always beem prerequisite of
>> it)
>> >> wrote it will be "easy". So I am not sure why you are bringing it now.
>> We
>> >> assume AIP-72 will be completed and this problem will be gone. Let's
>> not
>> >> mention it any more please.
>> >>
>> >> The true separation from TaskSDK will likely only land in about 3.2
>> time
>> >>> frame. We are actively working on it, but it’s a slow process of
>> untangling
>> >>> lots of assumptions made in the code base over the years. Maybe once
>> we
>> >>> have that my view would be different, but right now I think this
>> makes the
>> >>> proposal a non-starter. Especially as you are saying that most teams
>> will
>> >>> have unique connections. If they’ve got those already, then having an
>> asset
>> >>> trigger use those conns to watch/poll for activity is a much easier
>> >>> solution to operate and crucially, to scale and upgrade.
>> >>>
>> >>
>> >> Yes. I perfectly understand that and I am fully aware of potentially
>> 3.2
>> >> time-frame. And that's fine. Actually I heartily invite you to listen
>> to
>> >> the part of my talk from Berlin Buzzwords when I was asked for the
>> timeline
>> >> - https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the exact
>> >> timeline in my talk . My answer was basically - "3.1" or "3.2", and I
>> >> sincerely hope "3.1" but we might not be able to complete it because we
>> >> have other things to do (other - is indeed the Task Isolation work
>> that you
>> >> are leading). And that's perfectly fine. And it absolutely does not
>> prevent
>> >> us from voting on the AIP now - similarly as we voted on the previous
>> >> version of the AIP - knowing that it has some prerequisites a few
>> months
>> >> ago. Especially that we know that the feature we need from task
>> isolation
>> >> is "non-negotiable". I.e. it WILL happen. We don't hope for it, we
>> know it
>> >> will be there. Those are your own words.
>> >>
>> >>
>> >>> >  I think we can’t compare AIP-82 to sharing virtual assets due to
>> >>> complexity of it.
>> >>>
>> >>> Virtual Assets was a mistake, and not how users actually want to use
>> >>> them. Mea culpa
>> >>>
>> >>
>> >> This is the first time I hear this - certainly you never raised this
>> >> concern on the devlist. So if you have some concerns about virtual
>> assets I
>> >> think you should raise it on the devlist, because I think everyone
>> here is
>> >> missing some conversation (or maybe it's just your private opinion that
>> >> you never shared with anyone, but maybe it's worth). I would be
>> >> interested to hear how the feature that was absolutely most successful
>> >> feature of airflow 2 was a mistake. According to the 2024 survey
>> >> https://airflow.apache.org/blog/airflow-survey-2024/  - 48% of Airflow
>> >> users have been using it, even if it was added as one of the last
>> >> big features of Airflow 2. It's the MOST used feature out of all the
>> >> features out there. I would be really curious to see how it was a
>> mistake
>> >> (but  please start a separate thread explaining why you think it was a
>> >> mistake, what are your data points and what do you think should be
>> fixed.
>> >> Just dropping "virtual assets were a mistake" in the middle of
>> multi-team
>> >> conversation seems completely unjustified without knowing what you are
>> >> talking about. So I think, until we know more, this argument has no
>> base.
>> >>
>> >>
>> >>>
>> >>> S
>> >>> To restate my points:
>> >>>
>> >>> - Sharing a deployment between teams today/in 3.1 is operationally
>> more
>> >>> complex (both scaling, and upgrades) — this is a con, not a plus.
>> >>>
>> >>
>> >> Surely. But it will be easier when AIP-72 is complete (which I am
>> >> definitely looking forward to and as clearly explained in AIP-82, is a
>> >> prerequisite of it). Nothing changed here.
>> >>
>> >>
>> >>> - The main user benefit appears to be “allow teams’ DAGs to
>> communicate
>> >>> via Assets”, in which case we can do that today by putting more work
>> in to
>> >>> AIP-82’s Asset triggers
>> >>>
>> >>
>> >> No. Lower operational complexity for multi-teams (providing that we
>> >> deliver AIP-72) is another benefit. Virtual assets is another, and
>> since
>> >> there is no ground in "virtual assets is a mistake" statement (not
>> until
>> >> you explain what you mean by that in a separate discussion) - this is
>> also
>> >> still a very valid point.
>> >>
>> >>
>> >>> Soon, we will have then be asked about cross-team governance, policy
>> >>> enforcement, and potentially unbounded edge cases (e.g., team-specific
>> >>> secrets, roles, quotas). ain, you get this for free with truely
>> separate
>> >>> deployments already
>> >>> allow different teams to use different executors (including multiple
>> >>> executors per-team following AIP-61)
>> >>>
>> >>
>> >> Not really. We very explicitly say in the AIP that his is not a goal
>> and
>> >> that we have no plans for. And yes, using separate executors per team
>> is
>> >> actually back in the AIP-82 in case you did not notice (and the code
>> needed
>> >> for it's even implemented and merged already in main by Vincent).
>> >>
>> >>
>> >>> Provably not true right now, and until ~3.2 delivers the full Task
>> >>> SDK/Core dependency separation this would be _more_ work to upgrade,
>> not
>> >>> less, and that work is not shared but still on a central team.
>> >>>
>> >>
>> >> Absolutely - we will wait for AIP-72 completion. I do not want to say
>> 3.1
>> >> or 3.2 directly - because there are - as you said - a lot of moving
>> pieces.
>> >> So my target for multi-team is "After AIP-72 is completed". Full stop.
>> But
>> >> there is nothing wrong with accepting the AIP now and doing preparatory
>> >> work in parallel. Similarly as there is no way to have a baby in 1
>> month by
>> >> 9 women, there is no way adding more effort to task-sdk isolation will
>> >> speed it up - we alredy have not only 3 people (you leading it, Kaxil
>> and
>> >> Amog) but also all the help from me and even 10s of different
>> contributors
>> >> (for example with the recent db_test cleanup that I took leadership
>> on) -
>> >> and there are people who wish to work on adding multi-team features.
>> Since
>> >> the design heavily limits impact on airflow codebase and interactions
>> with
>> >> task-sdk implementation, there is nothing wrong with starting
>> >> implementation in parallel either- amazon team is keen to move it
>> forward -
>> >> they even already implemented SQS trigger for assets, and we are
>> working
>> >> together on FAB removal, Keycloak authentication manager - and they
>> seem to
>> >> still have capacity and drive to progress multi-team. So I am not sure
>> if
>> >> we are trading off something. There is no "if we work on more on task
>> sdk
>> >> and drop multi-team things will be faster". Generally in open source
>> people
>> >> work in the area where they feel they can provide best value - such as
>> you
>> >> working on task-sdk, me on CI,dev env, they will deliver more value on
>> >> multi-team
>> >>
>> >>
>> >>>
>> >>> So please, as succinctly as possible, please tell me what the direct
>> >>> benefit to users this proposal is over us putting this effort in to
>> writing
>> >>> better Asset triggers instead?
>> >>>
>> >>
>> >>
>> >> * less operational overhead for managing multi-team (once AIP-72 is
>> >> complete) where separate execution environments are important
>> >> * virtual assets sharing
>> >> * ability of having "admin" and "team sharing" capability where dags
>> from
>> >> multiple teams can be seen in a single Airflow UI  (requires custom
>> RBAC)
>> >>
>> >> None of this can be done via beter asset triggers
>> >>
>> >>
>> >>>
>> >>> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>> >
>> >>> > My counter-points:
>> >>> >
>> >>> >
>> >>> >> 1. Managing a multi team deployment is not materially different
>> from
>> >>> >> managing a deployment per team
>> >>> >>
>> >>> >
>> >>> > It's a bit easier - especially when it comes to upgrades (especially
>> >>> in the
>> >>> > case we are targetting when we are not targetting multi-tenant, but
>> >>> several
>> >>> > relatively closely cooperating teams with different dependncy
>> >>> requiremens
>> >>> > and isolation need.
>> >>> >
>> >>> > 2. The database changes were quite wide-reaching
>> >>> >>
>> >>> >
>> >>> > Yes. that is addressed.
>> >>> >
>> >>> >
>> >>> >> 3. I don’t believe the original AIP (again, I haven’t read the
>> updated
>> >>> >> proposal or recent messages on the thread. yet) will meet what many
>> >>> users
>> >>> >> want out of a multiteam solution
>> >>> >>
>> >>> >
>> >>> > I think we will only see when we try. A lot of people thing they
>> would,
>> >>> > even if they are warned. I know at least one user (Wealthsimple) who
>> >>> > definitely want to use it and they got a very detailed explanation
>> of
>> >>> the
>> >>> > idea and understand it well. So I am sure that **some** users would.
>> >>> But we
>> >>> > do not know how many.
>> >>> >
>> >>> >
>> >>> >> To expand on those points a bit more
>> >>> >>
>> >>> >> On 1. The only components that are shared are, I think, the
>> scheduler
>> >>> and
>> >>> >> the API server, and it’s arguable if that is actually a good idea
>> >>> given
>> >>> >> those are likely to be the most performance sensitive components
>> >>> anyway.
>> >>> >>
>> >>> >> Additionally the fact that the scheduler is a shared component
>> makes
>> >>> >> upgrading it almost a non starter as you would likely need buy-in,
>> >>> changes,
>> >>> >> and testing form ALL teams using it. I’d argue that this is a huge
>> >>> negative
>> >>> >> until we finish off the version indepence work of AIP-72.
>> >>> >>
>> >>> >
>> >>> > Quite disagree here - especially that our target is that task-sdk is
>> >>> > supposed to provide all isolation that is needed. There should be 0
>> >>> changes
>> >>> > in the dags needed to upgrade scheduler, api_server, triggerer -
>> >>> precisely
>> >>> > because we introduced backwards-compatible task-sdk.
>> >>> >
>> >>> > On 3 my complaint is essentially that this doesn’t go nearly far
>> >>> enough. It
>> >>> >> doesn’t allow read only views to other teams dags. I don’t think it
>> >>> allows
>> >>> >> you to be in multiple teams at once. You can’t share a connection
>> >>> between
>> >>> >> teams but only allow certain specified dags to access it, but would
>> >>> have to
>> >>> >> either be globally usable, or duplicated-and-kept-in-sync between
>> >>> teams. In
>> >>> >> short I think it fall short of being useful..
>> >>> >>
>> >>> >
>> >>> > Oh absolutely all that is possible (except sharing single
>> connections
>> >>> > between multiple teams - which is a very niche use cases and
>> >>> duplication
>> >>> > here is perfectly ok as first approximation - and if we need more we
>> >>> can
>> >>> > add it later).
>> >>> >
>> >>> > Auth manager RBAC and access is abstracted away, and the Keyclock
>> >>> Manager
>> >>> > implemented by Vincent allows to manage completely independent and
>> >>> separate
>> >>> > RBAC based on arguments and resources provided by Airflow. There is
>> >>> nothing
>> >>> > to prevent the user who configures KeyCloak RBAC to define it in the
>> >>> way:
>> >>> >
>> >>> > if group a > allow to read a and write b
>> >>> > if group b > alllow to write b but not a
>> >>> >
>> >>> > and any other combinations. KeyCloak implementation - pretty
>> advanced
>> >>> > already - (and design of auth manager) completely abstracts away
>> both
>> >>> > authentication and authorization to KeyCloak and KeyCloak has RBAC
>> >>> > management built in. Also any of the users can write their own -
>> even
>> >>> > hard-coded authentication manager to do the same if they do not
>> want to
>> >>> > have configurable KeyCloak. Even SimpleAuthManager could be
>> hard-coded
>> >>> to
>> >>> > provide thiose features.
>> >>> >
>> >>> >
>> >>> >>
>> >>> >> So on the surface, I’m no more in favour of using dag bundle as a
>> >>> >> replacement for team id as I think most of the above points still
>> >>> stand.
>> >>> >>
>> >>> >
>> >>> > We disagree here.
>> >>> >
>> >>> >>
>> >>> >> My counter proposal: We do _nothing_ to core airflow. We work on
>> >>> improving
>> >>> >> the event-based trigger o fdags (write more triggers for read/check
>> >>> remote
>> >>> >> Assets etc) so that teams can have 100% isolated deployments but
>> still
>> >>> >> trigger dags based on asset events from other teams.
>> >>> >>
>> >>> >
>> >>> > That does not solve any of the other design goals - only allows to
>> >>> trigger
>> >>> > assets a bit more easily (but also it's not entirely solved by
>> AIP-82
>> >>> > because it does not solve virtual assets - only ones that have
>> defined
>> >>> > triggerer and "something" to listen on - which is way more complex
>> than
>> >>> > just defining asset in a Dag and using it in another). I think we
>> can't
>> >>> > compare AIP-82 to sharing virtual assets due to complexity of it. I
>> >>> > explained it in the doc.
>> >>> >
>> >>> >
>> >>> > I will now go and catch up with the long thread and updated proposal
>> >>> and
>> >>> >> come back.
>> >>> >>
>> >>> >
>> >>> > Please. I hope the above explaination will help in better
>> >>> understanding of
>> >>> > the proposal, because I think you had some assumptions that do not
>> >>> hold any
>> >>> > more with the new proposal.
>> >>> >
>> >>> > J.
>> >>> >
>> >>> >
>> >>> >>
>> >>> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>> >>>
>> >>> >>> Just to clarify the relation - I updated the AIP now to refer to
>> >>> AIP-82
>> >>> >> and
>> >>> >>> to explain relation between the "cross-team" and "cross-airflow"
>> >>> asset
>> >>> >>> triggering - this is what I added:
>> >>> >>>
>> >>> >>> Note that there is a relation between AIP-82 ("External Driven
>> >>> >> Scheduling")
>> >>> >>> and this part of the functionality. When you have multiple
>> instances
>> >>> of
>> >>> >>> Airflow, you can use shared datasets - "Physical datasets" - that
>> >>> several
>> >>> >>> Airflow Instances can use - for example there could be an S3
>> object
>> >>> that
>> >>> >> is
>> >>> >>> produced by one airflow instance, and consumed by another. That
>> >>> requires
>> >>> >>> deferred trigger to monitor for such datasets, and appropriate
>> >>> >> permissions
>> >>> >>> to the external dataset, and you could achive similar result to
>> >>> >> cross-team
>> >>> >>> dataset triggering (but cross airflow). However the feature of
>> >>> sharing
>> >>> >>> datasets between the teams also works for virtual assets, that do
>> not
>> >>> >> have
>> >>> >>> physically shared "objects" and trigger that is monitoring for
>> >>> changes in
>> >>> >>> such asset.
>> >>> >>>
>> >>> >>> J.
>> >>> >>>
>> >>> >>>
>> >>> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com>
>> >>> wrote:
>> >>> >>>
>> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>> >>> reference
>> >>> >> to
>> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>> >>> through.
>> >>> >>>>
>> >>> >>>> Yep. It did not - because I did not think it was needed or even
>> very
>> >>> >>>> important after the simplifications. AIP-82 has a different
>> scope,
>> >>> >> really.
>> >>> >>>> It only helps when the Assets are "real" data files which we have
>> >>> >> physical
>> >>> >>>> triggers for, it's slightly related - sharing datasets between
>> teams
>> >>> >>>> (including those that do not require physical files and
>> triggers) is
>> >>> >> still
>> >>> >>>> possible in the design we have now, but it's not (and never was)
>> the
>> >>> >>>> **only** reason for having multi-team. There always was (and
>> still
>> >>> is)
>> >>> >> the
>> >>> >>>> possibility of having a common, distinct environments (i.e.
>> >>> dependencies
>> >>> >>>> and providers) per team, the possibility of having connections
>> and
>> >>> >>>> variables that are only accessible to one team and not the other,
>> >>> and
>> >>> >>>> isolating workload execution (all that while allowing to manage
>> >>> multiple
>> >>> >>>> team and schedule things with single deployment). That did not
>> >>> change.
>> >>> >> What
>> >>> >>>> changed a lot is that it is now way simpler, something that we
>> can
>> >>> >>>> implement without heavy changes to the codebase - and give it to
>> our
>> >>> >> users,
>> >>> >>>> so that they can assess if this is something they need without
>> too
>> >>> much
>> >>> >>>> risk and effort.
>> >>> >>>>
>> >>> >>>> This was - I believe the main concern, that the value we get from
>> >>> it is
>> >>> >>>> not dramatic, but the required changes are huge. This "redesign"
>> >>> changes
>> >>> >>>> the equation - the value is still unchanged, but the cost of
>> >>> >> implementing
>> >>> >>>> it and impact on the Airflow codebase is much smaller. I still
>> have
>> >>> not
>> >>> >>>> heard back from Ash if my proposal responds to his original
>> concern
>> >>> >> though,
>> >>> >>>> so I am mostly guessing (also based on the positive impact of
>> >>> others)
>> >>> >> that
>> >>> >>>> yes it does. But to be honest I am not sure and I would love to
>> hear
>> >>> >> back,
>> >>> >>>> I decided to update the AIP to reflect it - regardless, because I
>> >>> think
>> >>> >> the
>> >>> >>>> simplification I proposed keeps the original goals, but is indeed
>> >>> way
>> >>> >>>> simpler.
>> >>> >>>>
>> >>> >>>>> This is a very difficult thread to catch up on.
>> >>> >>>>
>> >>> >>>> Valid point. Let me summarize what is the result:
>> >>> >>>>
>> >>> >>>> * I significantly simplified the implementation proposal
>> comparing
>> >>> to
>> >>> >> the
>> >>> >>>> original version
>> >>> >>>> * main simplification is very limited impact on existing
>> database -
>> >>> >>>> without "ripple effect" that would require us to change a lot of
>> >>> tables,
>> >>> >>>> including their primary keys, and heavily impact the UI
>> >>> >>>> * this is now more of an incremental change that can be
>> implemented
>> >>> way
>> >>> >>>> faster and with far less risk
>> >>> >>>> * updated idea is based on leveraging bundles (already part of
>> our
>> >>> data
>> >>> >>>> model) to map them (many-to-one) to a team - which requires to
>> just
>> >>> >> extend
>> >>> >>>> the data model with bundle mapping and add team_id to connections
>> >>> and
>> >>> >>>> variables. Those are all needed DB changes.
>> >>> >>>>
>> >>> >>>> The AIP is updated - in a one single big change so It should be
>> >>> easy to
>> >>> >>>> compare the changes:
>> >>> >>>>
>> >>> >>
>> >>>
>> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
>> >>> >>>> -> I even named the version appropriately "Simplified multi-team
>> >>> AIP" -
>> >>> >> you
>> >>> >>>> can select and compare v.65 with v.66 to see the exact
>> differences I
>> >>> >>>> proposed.
>> >>> >>>>
>> >>> >>>> I hope it will be helpful to catch up and for those who did not
>> >>> follow,
>> >>> >> to
>> >>> >>>> be able to make up their minds about it.
>> >>> >>>>
>> >>> >>>> J.
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
>> >>> >> <vi...@astronomer.io.invalid>
>> >>> >>>> wrote:
>> >>> >>>>
>> >>> >>>>> This is a very difficult thread to catch up on.
>> >>> >>>>> I will take a detailed look at the AIP update to try to figure
>> out
>> >>> the
>> >>> >>>>> changes in the proposal.
>> >>> >>>>>
>> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>> >>> reference
>> >>> >> to
>> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>> >>> through.
>> >>> >>>>>
>> >>> >>>>> Vikram
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>> >>> >>>>> gopidesupavan@gmail.com>
>> >>> >>>>> wrote:
>> >>> >>>>>
>> >>> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's much
>> >>> more
>> >>> >> slim
>> >>> >>>>>> down.
>> >>> >>>>>>
>> >>> >>>>>> left a minor comment. :) Overall looking great.
>> >>> >>>>>>
>> >>> >>>>>> Pavan
>> >>> >>>>>>
>> >>> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>> >>> >>>>> <j_scheffler@gmx.de.invalid
>> >>> >>>>>>>
>> >>> >>>>>> wrote:
>> >>> >>>>>>
>> >>> >>>>>>> Thanks for the rework/update of the AIP-72!
>> >>> >>>>>>>
>> >>> >>>>>>> Just a few small comments but overall I like it as it is much
>> >>> leaner
>> >>> >>>>>>> than originally planned and is in a level of complexity that
>> it
>> >>> >> really
>> >>> >>>>>>> seems to be a benefit to close the gap as described.
>> >>> >>>>>>>
>> >>> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
>> >>> >>>>>>>> I updated the AIP - including architecture images and
>> reviewed
>> >>> it
>> >>> >>>>>> (again)
>> >>> >>>>>>>> and corrected any ambiguities and places where it needed to
>> be
>> >>> >>>>> changed.
>> >>> >>>>>>>>
>> >>> >>>>>>>> I think the current state
>> >>> >>>>>>>>
>> >>> >>>>>>>
>> >>> >>>>>>
>> >>> >>>>>
>> >>> >>
>> >>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>> >>> >>>>>>>> - nicely describes the proposal.
>> >>> >>>>>>>>
>> >>> >>>>>>>> Comparing to the previous one:
>> >>> >>>>>>>>
>> >>> >>>>>>>> 1. The DB changes are far less intrusive - no ripple effect
>> on
>> >>> >>>>> Airflow
>> >>> >>>>>>>> 2. There is no need to merge configurations and provide
>> >>> different
>> >>> >>>>> set
>> >>> >>>>>> of
>> >>> >>>>>>>> configs per team - we can add it later but I do not see why
>> we
>> >>> need
>> >>> >>>>> it
>> >>> >>>>>> in
>> >>> >>>>>>>> this simplified version
>> >>> >>>>>>>> 3. We can still configure a different set of executors per
>> team
>> >>> -
>> >>> >>>>> that
>> >>> >>>>>> is
>> >>> >>>>>>>> already implemented (we just need to wire it to the bundle ->
>> >>> team
>> >>> >>>>>>> mapping).
>> >>> >>>>>>>>
>> >>> >>>>>>>> I think it will be way simpler and faster to implement this
>> way
>> >>> and
>> >>> >>>>> it
>> >>> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can
>> >>> give
>> >>> >>>>> our
>> >>> >>>>>>>> users so that they can provide feedback.
>> >>> >>>>>>>>
>> >>> >>>>>>>> J.
>> >>> >>>>>>>>
>> >>> >>>>>>>>
>> >>> >>>>>>>>
>> >>> >>>>>>>>
>> >>> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <
>> jarek@potiuk.com>
>> >>> >>>>> wrote:
>> >>> >>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>> I like this iteration a bit more now for sure, thanks for
>> >>> being
>> >>> >>>>>>> receptive
>> >>> >>>>>>>>>> to feedback! :)
>> >>> >>>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>> This now becomes quite close to what was proposing before,
>> we
>> >>> now
>> >>> >>>>>> again
>> >>> >>>>>>>>>> have a team ID (which I think is really needed here, glad
>> to
>> >>> see
>> >>> >>>>> it
>> >>> >>>>>>> back)
>> >>> >>>>>>>>>> and it will be used for auth management, configuration
>> >>> >>>>> specification,
>> >>> >>>>>>> etc
>> >>> >>>>>>>>>> but will be carried by Bundle instead of the dag model.
>> Which
>> >>> as
>> >>> >>>>> you
>> >>> >>>>>>> say
>> >>> >>>>>>>>>> “For that we will need to make sure that both api-server,
>> >>> >>>>> scheduler
>> >>> >>>>>> and
>> >>> >>>>>>>>>> triggerer have access to the "bundle definition" (to
>> perform
>> >>> the
>> >>> >>>>>>> mapping)"
>> >>> >>>>>>>>>> which honestly doesn’t feel too much different from the
>> >>> original
>> >>> >>>>>>> proposal
>> >>> >>>>>>>>>> we had last week of adding it to Dag table and ensuring
>> it’s
>> >>> >>>>>> available
>> >>> >>>>>>>>>> everywhere. but either way I’m happy to meet in the middle
>> and
>> >>> >>>>> keep
>> >>> >>>>>> it
>> >>> >>>>>>> on
>> >>> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable
>> location.
>> >>> >>>>>>>>>>
>> >>> >>>>>>>>> I think the big difference is the "ripple effect" that was
>> >>> >>>>> discussed
>> >>> >>>>>> in
>> >>> >>>>>>>>>
>> >>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>> >>> >>>>>> (and I
>> >>> >>>>>>>>> believe - correct me if I am wrong Ash - important trigger
>> for
>> >>> the
>> >>> >>>>>>>>> discussion) so far what we wanted is to extend the primary
>> key
>> >>> and
>> >>> >>>>> it
>> >>> >>>>>>> would
>> >>> >>>>>>>>> ripple through all the pieces of Airflow -> models, API, UI
>> >>> etc.
>> >>> >>>>> ...
>> >>> >>>>>>>>> However - we already have `bundle_name" and "bundle_version"
>> >>> in the
>> >>> >>>>>> Dag
>> >>> >>>>>>>>> model. So I think when we add a separate table where we map
>> the
>> >>> >>>>> bundle
>> >>> >>>>>>> to
>> >>> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not
>> want
>> >>> to
>> >>> >>>>>> change
>> >>> >>>>>>>>> primary key, we do not want to change UI in any way (except
>> >>> >>>>> filtering
>> >>> >>>>>> of
>> >>> >>>>>>>>> DAGs available based on your team - but that will be
>> handled in
>> >>> >>>>> Auth
>> >>> >>>>>>>>> Manager and will not impact UI in any way, I think that's a
>> >>> huge
>> >>> >>>>>>>>> simplification of the implementation, and if we agree to it
>> - i
>> >>> >>>>> think
>> >>> >>>>>> it
>> >>> >>>>>>>>> should speed up the implementation significantly. There are
>> >>> only a
>> >>> >>>>>>> limited
>> >>> >>>>>>>>> number of times where you need to look up the team_id - so
>> >>> having
>> >>> >>>>> the
>> >>> >>>>>>>>> bundle -> team mapping in a separate table and having to
>> look
>> >>> them
>> >>> >>>>> up
>> >>> >>>>>>>>> should not be a problem. And it has much less complexity and
>> >>> >>>>>>>>> "ripple-effect" through the codebase (for example I could
>> >>> imagine
>> >>> >>>>> 100s
>> >>> >>>>>>> or
>> >>> >>>>>>>>> thousands already written tests that would have to be
>> adapted
>> >>> if we
>> >>> >>>>>>> changed
>> >>> >>>>>>>>> the primary key - where there will be pretty much zero
>> impact
>> >>> on
>> >>> >>>>>>> existing
>> >>> >>>>>>>>> tests if we just add bundle -> team lookup table.
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>> One other thing I’d point out is that I think including
>> >>> executors
>> >>> >>>>> per
>> >>> >>>>>>>>>> team is a very easy win and quite possible without much
>> work.
>> >>> I
>> >>> >>>>>> already
>> >>> >>>>>>>>>> have much of the code written. Executors are already aware
>> of
>> >>> >>>>> Teams
>> >>> >>>>>>> that
>> >>> >>>>>>>>>> own them (merged), I have a PR open to have configuration
>> per
>> >>> team
>> >>> >>>>>>> (with a
>> >>> >>>>>>>>>> quite simple and isolated approach, I believe you approved
>> >>> Jarek).
>> >>> >>>>>> The
>> >>> >>>>>>> last
>> >>> >>>>>>>>>> piece is updating the scheduling logic to route tasks from
>> a
>> >>> >>>>>> particular
>> >>> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much
>> work
>> >>> >>>>> (though
>> >>> >>>>>> it
>> >>> >>>>>>>>>> would be easier if the Task models had a column for the
>> team
>> >>> they
>> >>> >>>>>>> belong
>> >>> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to get
>> >>> the
>> >>> >>>>>> team) I
>> >>> >>>>>>>>>> have a branch where I was experimenting with this logic
>> >>> already.
>> >>> >>>>>>>>>> Any who, long story short, I don’t think we necessarily
>> need
>> >>> to
>> >>> >>>>>> remove
>> >>> >>>>>>>>>> this piece from the project's scope if it is already partly
>> >>> done
>> >>> >>>>> and
>> >>> >>>>>>> not
>> >>> >>>>>>>>>> too difficult.
>> >>> >>>>>>>>>>
>> >>> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want to
>> just
>> >>> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we
>> have
>> >>> it
>> >>> >>>>> in.
>> >>> >>>>>>> And
>> >>> >>>>>>>>> if we can make it in, easily (which it seems we can) - we
>> can
>> >>> also
>> >>> >>>>>>> include
>> >>> >>>>>>>>> it in the first iteration. What I wanted to avoid really
>> (from
>> >>> the
>> >>> >>>>>>> original
>> >>> >>>>>>>>> design) - again trying to simplify it, limit the changes,
>> and
>> >>> >>>>> speed up
>> >>> >>>>>>>>> implementation. And there is one "complexity" that I wanted
>> to
>> >>> >>>>> avoid
>> >>> >>>>>>>>> specifically - having to have separate , additional
>> >>> configuration
>> >>> >>>>> per
>> >>> >>>>>>> team.
>> >>> >>>>>>>>> Not only because it complicates already complex
>> configuration
>> >>> >>>>> handling
>> >>> >>>>>>> (I
>> >>> >>>>>>>>> know we have PR for that) but mostly because if it is not
>> >>> needed,
>> >>> >>>>> we
>> >>> >>>>>> can
>> >>> >>>>>>>>> simplify documentation and explain to our users easier what
>> >>> they
>> >>> >>>>> need
>> >>> >>>>>>> to do
>> >>> >>>>>>>>> to have their own multi-team setup. And I am quite open to
>> >>> keeping
>> >>> >>>>>>>>> multiple-executors if we can avoid complicating
>> configuration.
>> >>> >>>>>>>>>
>> >>> >>>>>>>>> But I think some details of that and whether we really need
>> >>> >>>>> separate
>> >>> >>>>>>>>> configuration might also come as a result of updating the
>> AIP
>> >>> - I
>> >>> >>>>> am
>> >>> >>>>>> not
>> >>> >>>>>>>>> quite sure now if we need it, but we can discuss it when we
>> >>> >>>>> iterate on
>> >>> >>>>>>> the
>> >>> >>>>>>>>> AIP.
>> >>> >>>>>>>>>
>> >>> >>>>>>>>> J.
>> >>> >>>>>>>>>
>> >>> >>>>>>>>>
>> >>> >>>>>>>
>> >>> >>>>>>>
>> >>> ---------------------------------------------------------------------
>> >>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >>> >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>> >>> >>>>>>>
>> >>> >>>>>>>
>> >>> >>>>>>
>> >>> >>>>>
>> >>> >>>>
>> >>> >>
>> >>> >>
>> >>> >>
>> ---------------------------------------------------------------------
>> >>> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >>> >> For additional commands, e-mail: dev-help@airflow.apache.org
>> >>>
>> >>>
>>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

> The direction this one is taking is interesting. If you're really just
trying to make the feature barely possible and mostly targeted towards
managed providers to implement the rest, then I suppose this hits the mark.

Well actually by taking the direction I took, it's not "mostly for managed
providers" - i see it as it is equally, for managed providers and on-prem
users, but also, following the open-source spirit, philosophically, I think
in Airflow, any such change should be done with those things in mind,
because we are at the stage where we are already "established' and by
innovating on top what we have we have sometimes more to lose than to gain
- so I feel with "deployment' features we should be very careful to
distinguish 'enabling things" vs. 'doing things". My focus with this
iteration was to remove all the roadblocks that make it impossible (or
extremely difficult) to implement "real" multi-team and separation without
modifying airflow core. I though "what is the minimal set of features that
will make it "possible" for someone motivated to deploy a single airflow
for multiple teams.

* minimise maintenance effort increase
* do not "spoil" the "simple case" - we do not want to add features that
make "simple" implementation more complex the current `docker run -it
apache/airflow standalone` - should be simple and straightforward to run
* if there is anything that involves complex deployment, we should not aim
to make a "turn-key" solution that we will have to support - similarly like
we do with our configuration parameters, we have 100s knobs to turn, and as
long as default settings are reasonable and someone "motivated" can
configure and fine-tune - this configuration and fine-tuning should be left
to them - regardless if they are on-prem or managed. And both should be
able to do it.

I think it's not only smart technically (we support the low-level basic
features and when someone puts them together and makes it more of a
turn-key solution they are responsible for designing and implementing it -
so we have less maintenance effort. But also it's good from a simple
"open-source business model point of view" - i.e. it's a smart product
decision we should make.

Why airflow is #18 in OSS rank - of course we have a huge community and
people contributing in their free time, completely voluntarily. And we
cherish, support and encourage it. But let's be honest - if not all those
that make business on top of airflow did not invest literally millions of
dollars (in terms of engineering salary, sponsoring Airflow Summit,
supporting people like me (some smart stakeholders at least who understand
the value of it) who can be good "community spirit" - Airflow would have
order of magnitude less activity, reach, Airflow 3 would not be simply
possible. And this is a good thing that we have those stakeholders that are
interested and make money by turning Airflow into a "turn-key" solution.
This is a fantastic, symbiotic relationship.

So - what my thinking is - we should NOT make things that make airflow more
turn-key for those complex cases. We should leave it up to those who want
to make it and want to charge money for it. This is cool and great that
they can do - and we should not do it "for them" - but on the other hand -
we should make it possible that those who want to turn airflow into more
complex (say multi-team solution) to make it happen - by providing them
with minimal set of features that make it possible.

And that also - in a way - keeps the balance between on-prem and managed
implementation.

Something that I've learned as a rule of thumb is that making a feature
"generic" compared to custom implementation is 3x-10x more expensive (both
in implementation and maintenance). And it means that if an on-prem user
wants to implement something for them (say turn-key multi-team solution for
their case) it will cost `x` , but when a managed provider wants to
implement a generic multi-team it will cost `10x`. But also managed
providers can spread the cost over the premium they will charge to their
users so that they don't have to manage Airflow on their own and pay `x`
for this mult-team feature to develop on their own. And this is a "fair"
choice to make by on-prem users. They might choose what they want to do
then. Also it's fair for managed provider - yes they need to invest more,
but also they have a chance to shine on promoting it and making it more
optimised at scale etc. etc.

That is my line of thinking.


J.


On Thu, Jul 3, 2025 at 1:41 AM Oliveira, Niko <on...@amazon.com.invalid>
wrote:

> Hey Jarek,
>
>
> The direction this one is taking is interesting. If you're really just
> trying to make the feature barely possible and mostly targeted towards
> managed providers to implement the rest, then I suppose this hits the mark.
>
> But this is not something we're asking for at Amazon and personally I
> think we should make the feature reasonably usable for those running
> self-managed OSS Airflow as well. There are many users running an on-prem
> Airflow. Getting too hyper-fixated on an implementation that's so
> simplified that it's obtuse and difficult to use by most users seems like
> the wrong approach to me. But you and I have already discussed this at
> length and I haven't convinced you so far, so if I'm the only one with this
> thinking then I'm happy to disagree and commit as we say at Amazon :)
>
>
> > So I would be rather strong on **not** touching the current
> configuration and
>
> simply adding configuration for per-team executors in executor config -
> even if it is uglier and more "low-level".
>
> Can you explain what "adding configuration for per-team executors in
> executor config" would look like? I don't have a concrete sense of what you
> mean by this.
>
> Thanks for your efforts on trying to get this feature agreed to and voted
> on. Looking forward to working on the project in the coming weeks!
>
> Cheers,
> Niko
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Tuesday, July 1, 2025 10:26:55 PM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External
> event driven dags) exists
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> Any last comments ? There is a long weekend coming up in the US, so I will
> likely start voting on the updated AIP on Monday 7th.
>
> On Fri, Jun 27, 2025 at 12:41 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I'd really love to finalise discussion and put it up to a vote some time
> > after the recording from the last dev call is posted - so that more
> > context, details and the LONG discussion we had on it. There is no *huge*
> > hurry  - we have strong dependency on Task Isolation and it seems that it
> > will still take a bit of time to complete, so I'd say I would love to
> start
> > voting in about a week time - so that maybe at the next dev call we can
> > "seal" the subject. Happy to see any more comments - especially from
> those
> > who have opinions but they had no opportunity to express them.
> >
> > I am personally very happy with the direction it took - simplification
> and
> > "MVP" kind of approach - also I invite the stakeholders of ours to take a
> > close look at the scope and what we really propose - I have a feeling
> that
> > we can balance it out - there is something we can make to make it not
> > "worse" for the offerings they have. I think we have a really good
> > symbiotic relationship here, and I would love to leverage that. For one -
> > my goal here is to have a minimum number of changes that are impacting
> > maintainability of the open-source airflow - but mostly "opening up some
> > possibilities" - rather than provide turn-key solutions. And mostly
> because
> > this is good for all sides - less maintenance and complexity for OSS
> > maintainers, but more opportunities to make it into "turn-key" solutions
> by
> > the stakeholders, while also allowing the "on-prem" users - if they are
> > highly motivated - to use those features by adding the "turn-key" layer
> on
> > their own. Also adding multi-team should not be at the expense of
> "simple"
> > installations - they should be virtually unaffected.
> >
> > One example of applying this is cutting on "separate config files". I
> > think it moves us closer to a "turn-key" solution but it is not really
> > necessary to achieve the three goals above - that's why in the current
> > proposal this part is completely removed - Sorry Niko, but I still think
> > it's one of the things that falls into this bucket. We can easily remove
> > it, they complicate code, documentation and options the users have, and
> > even if it is a "little" more complex to manage configuration by
> motivated
> > users, it's also an opportunity for "turn-key" option that stakeholders
> can
> > build in their products - and we do not have to maintain it in the
> > open-source. So I would be rather strong on **not** touching the current
> > configuration and simply adding configuration for per-team executors in
> > executor config - even if it is uglier and more "low-level".
> >
> > So if there are some constructive ideas on what can be done to make it
> > "simpler" and less "turn-key" in that respect - I would highly value such
> > ideas and comments. If we can cut down something more that is not
> > "necessary" for the three primary goals I came up with - I am more than
> > happy to do it.
> >
> > Just to remind - those are the "extracted" goals. I slightly updated them
> > and added to the preamble of the AIP:
> >
> > * less operational overhead for managing multi-team (once AIP-72 is
> > complete) where separate execution environments are important
> > * virtual assets sharing between teams
> > * ability of having "admin" and "team sharing" capability where dags from
> > multiple teams can be seen in a single Airflow UI  (requires custom RBAC
> an
> > AIP-56 implementation of Auth Manager - with KeyCloak Auth Manager being
> a
> > reference implementation)
> >
> > J.
> >
> >
> > On Thu, Jun 26, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >>
> >>> One technical observation: Now that the dag table no longer has a
> >>> team_id in it, what would the behaviour be when a DAG is attempted to
> move
> >>> between bundles? How do we detect this? (I’m not all convinced that we
> >>> correctly detect duplicate dag ids across bundles today, so I wouldn’t
> >>> assume or rely on the current behaviour.)
> >>>
> >>
> >> Of course - yes, I realise that - that problem was also not handled in
> >> the previous iteration to be honest. That is something that dag bundle
> >> solution allows to solve eventually - but I do not think it's a blocker
> for
> >> the proposed implementation. We will have to eventually add some way of
> >> blocking dags to jump between bundles, we might tackle this separately.
> I
> >> already wanted to propose a separate update to that - but I did not
> want to
> >> complicate the current proposal. One thing at a time. I can, however -
> if
> >> you consider that as a blocker, extend the current AIP with it. Not a
> big
> >> problem. This is however a bit independent from the team_id
> introduction.
> >>
> >> Overall, I am still unconvinced this proposal has enough real user
> >>> benefit over actually separate deployments, and on balance of the added
> >>> complexity and maintenance burden I do not think it is worth it.
> >>>
> >>
> >> That makes me sad, I thought that over the course of the discussion I
> >> addressed all the concerns (in this case the concern was "is it worth
> with
> >> the cost and little benefit", but when I did it and heavily limited the
> >> impact, now the concern is "is it worth at all as changes are really
> >> minimal" - and surely, anyone can change and adapt their concerns, over
> >> time,  but that one seems like ever-moving target. I hoped at least for
> >> some acknowledgment of some concerns (complexity in this case) is
> >> addressed, but it seems that you are deeply convinced that we do not
> need
> >> multi-team at all (which is in stark contrast with at least a dozen of
> >> bigger and smaller users of Airflow who submitted talks to Airflow
> summit
> >> (including about 5 or 6 submissions for Airflow 2025) on how they spent
> >> their engineering effort, time and money on trying to achieve something
> >> similar - they assessed that it's worth, you  assess that it's not
> worth.
> >> Somehow I trust our users that they were not spending the money, time
> and
> >> engineering effort to achieve this because they wanted to spend more
> money.
> >> I think they assessed it's worth it. So I want to make it a bit easier
> and
> >> more "proper" way for them to do that.
> >>
> >>>
> >>> Upgrades: it is not easier to upgrade under this multi team proposal,
> >>> but much much harder. This is based on hard earned experience from
> helping
> >>> Astronomer users — having to coordinate upgrades between multiple teams
> >>> turns in to a months long slog of the hardest kind of work —  people
> work:
> >>> getting other teams to agree to do things that they don’t directly care
> >>> about — “It’s working for me, I don’t care about upgrading, we’ll get
> to it
> >>> next quarter” is a refrain I’ve heard many times.
> >>>
> >>
> >> Yes. absolutely - this is why we deferred it until we knew what shape
> >> task isolation and other AIPs we depend on take on. Because it is clear
> >> that pretty much all the problem you explain above are going to be
> solved
> >> with task isolation. And it's not just my opinion. If you want to argue
> >> with it, you likely need to argue with yourself:
> >> https://github.com/apache/airflow/issues/51545#issuecomment-2980038478.
> >> Let me quote what you wrote there last week:
> >>
> >> Ash Berlin Taylor wrote:
> >>
> >> > A tight coupling between task-sdk and any "server side" component is
> >> the opposite to one of the goals of AIP-72 (I'm not sure we ever
> explicitly
> >> said this, but the first point of motivation for the AIP says
> "Dependency
> >> conflicts for administrators supporting data teams using different
> versions
> >> of providers, libraries, or python packages")
> >> > In short, my goal with TaskSDK, and the reason for introducing CalVer
> >> and Cadwyn with the execution API is to end up in a world where you can
> >> upgrade the Airflow Scheduler/API server interdependently of any worker
> >> nodes (with the exception that the server must be at least as new as the
> >> clients)
> >> > This ability to have version-skew is pretty much non-negotiable to me
> >> and is (other than other languages) one of primary benefits of AIP-72
> >>
> >> If you read yourself from that quote it basically means "it will be easy
> >> to upgrade airflow independently of workers". So I am a bit confused
> here.
> >> Yes, I agree it was difficult, but you yourself explain that when AIP-72
> >> (which since API-67 has been accepted has always beem prerequisite of
> it)
> >> wrote it will be "easy". So I am not sure why you are bringing it now.
> We
> >> assume AIP-72 will be completed and this problem will be gone. Let's not
> >> mention it any more please.
> >>
> >> The true separation from TaskSDK will likely only land in about 3.2 time
> >>> frame. We are actively working on it, but it’s a slow process of
> untangling
> >>> lots of assumptions made in the code base over the years. Maybe once we
> >>> have that my view would be different, but right now I think this makes
> the
> >>> proposal a non-starter. Especially as you are saying that most teams
> will
> >>> have unique connections. If they’ve got those already, then having an
> asset
> >>> trigger use those conns to watch/poll for activity is a much easier
> >>> solution to operate and crucially, to scale and upgrade.
> >>>
> >>
> >> Yes. I perfectly understand that and I am fully aware of potentially 3.2
> >> time-frame. And that's fine. Actually I heartily invite you to listen to
> >> the part of my talk from Berlin Buzzwords when I was asked for the
> timeline
> >> - https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the exact
> >> timeline in my talk . My answer was basically - "3.1" or "3.2", and I
> >> sincerely hope "3.1" but we might not be able to complete it because we
> >> have other things to do (other - is indeed the Task Isolation work that
> you
> >> are leading). And that's perfectly fine. And it absolutely does not
> prevent
> >> us from voting on the AIP now - similarly as we voted on the previous
> >> version of the AIP - knowing that it has some prerequisites a few months
> >> ago. Especially that we know that the feature we need from task
> isolation
> >> is "non-negotiable". I.e. it WILL happen. We don't hope for it, we know
> it
> >> will be there. Those are your own words.
> >>
> >>
> >>> >  I think we can’t compare AIP-82 to sharing virtual assets due to
> >>> complexity of it.
> >>>
> >>> Virtual Assets was a mistake, and not how users actually want to use
> >>> them. Mea culpa
> >>>
> >>
> >> This is the first time I hear this - certainly you never raised this
> >> concern on the devlist. So if you have some concerns about virtual
> assets I
> >> think you should raise it on the devlist, because I think everyone here
> is
> >> missing some conversation (or maybe it's just your private opinion that
> >> you never shared with anyone, but maybe it's worth). I would be
> >> interested to hear how the feature that was absolutely most successful
> >> feature of airflow 2 was a mistake. According to the 2024 survey
> >> https://airflow.apache.org/blog/airflow-survey-2024/  - 48% of Airflow
> >> users have been using it, even if it was added as one of the last
> >> big features of Airflow 2. It's the MOST used feature out of all the
> >> features out there. I would be really curious to see how it was a
> mistake
> >> (but  please start a separate thread explaining why you think it was a
> >> mistake, what are your data points and what do you think should be
> fixed.
> >> Just dropping "virtual assets were a mistake" in the middle of
> multi-team
> >> conversation seems completely unjustified without knowing what you are
> >> talking about. So I think, until we know more, this argument has no
> base.
> >>
> >>
> >>>
> >>> S
> >>> To restate my points:
> >>>
> >>> - Sharing a deployment between teams today/in 3.1 is operationally more
> >>> complex (both scaling, and upgrades) — this is a con, not a plus.
> >>>
> >>
> >> Surely. But it will be easier when AIP-72 is complete (which I am
> >> definitely looking forward to and as clearly explained in AIP-82, is a
> >> prerequisite of it). Nothing changed here.
> >>
> >>
> >>> - The main user benefit appears to be “allow teams’ DAGs to communicate
> >>> via Assets”, in which case we can do that today by putting more work
> in to
> >>> AIP-82’s Asset triggers
> >>>
> >>
> >> No. Lower operational complexity for multi-teams (providing that we
> >> deliver AIP-72) is another benefit. Virtual assets is another, and since
> >> there is no ground in "virtual assets is a mistake" statement (not until
> >> you explain what you mean by that in a separate discussion) - this is
> also
> >> still a very valid point.
> >>
> >>
> >>> Soon, we will have then be asked about cross-team governance, policy
> >>> enforcement, and potentially unbounded edge cases (e.g., team-specific
> >>> secrets, roles, quotas). ain, you get this for free with truely
> separate
> >>> deployments already
> >>> allow different teams to use different executors (including multiple
> >>> executors per-team following AIP-61)
> >>>
> >>
> >> Not really. We very explicitly say in the AIP that his is not a goal and
> >> that we have no plans for. And yes, using separate executors per team is
> >> actually back in the AIP-82 in case you did not notice (and the code
> needed
> >> for it's even implemented and merged already in main by Vincent).
> >>
> >>
> >>> Provably not true right now, and until ~3.2 delivers the full Task
> >>> SDK/Core dependency separation this would be _more_ work to upgrade,
> not
> >>> less, and that work is not shared but still on a central team.
> >>>
> >>
> >> Absolutely - we will wait for AIP-72 completion. I do not want to say
> 3.1
> >> or 3.2 directly - because there are - as you said - a lot of moving
> pieces.
> >> So my target for multi-team is "After AIP-72 is completed". Full stop.
> But
> >> there is nothing wrong with accepting the AIP now and doing preparatory
> >> work in parallel. Similarly as there is no way to have a baby in 1
> month by
> >> 9 women, there is no way adding more effort to task-sdk isolation will
> >> speed it up - we alredy have not only 3 people (you leading it, Kaxil
> and
> >> Amog) but also all the help from me and even 10s of different
> contributors
> >> (for example with the recent db_test cleanup that I took leadership on)
> -
> >> and there are people who wish to work on adding multi-team features.
> Since
> >> the design heavily limits impact on airflow codebase and interactions
> with
> >> task-sdk implementation, there is nothing wrong with starting
> >> implementation in parallel either- amazon team is keen to move it
> forward -
> >> they even already implemented SQS trigger for assets, and we are working
> >> together on FAB removal, Keycloak authentication manager - and they
> seem to
> >> still have capacity and drive to progress multi-team. So I am not sure
> if
> >> we are trading off something. There is no "if we work on more on task
> sdk
> >> and drop multi-team things will be faster". Generally in open source
> people
> >> work in the area where they feel they can provide best value - such as
> you
> >> working on task-sdk, me on CI,dev env, they will deliver more value on
> >> multi-team
> >>
> >>
> >>>
> >>> So please, as succinctly as possible, please tell me what the direct
> >>> benefit to users this proposal is over us putting this effort in to
> writing
> >>> better Asset triggers instead?
> >>>
> >>
> >>
> >> * less operational overhead for managing multi-team (once AIP-72 is
> >> complete) where separate execution environments are important
> >> * virtual assets sharing
> >> * ability of having "admin" and "team sharing" capability where dags
> from
> >> multiple teams can be seen in a single Airflow UI  (requires custom
> RBAC)
> >>
> >> None of this can be done via beter asset triggers
> >>
> >>
> >>>
> >>> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>> >
> >>> > My counter-points:
> >>> >
> >>> >
> >>> >> 1. Managing a multi team deployment is not materially different from
> >>> >> managing a deployment per team
> >>> >>
> >>> >
> >>> > It's a bit easier - especially when it comes to upgrades (especially
> >>> in the
> >>> > case we are targetting when we are not targetting multi-tenant, but
> >>> several
> >>> > relatively closely cooperating teams with different dependncy
> >>> requiremens
> >>> > and isolation need.
> >>> >
> >>> > 2. The database changes were quite wide-reaching
> >>> >>
> >>> >
> >>> > Yes. that is addressed.
> >>> >
> >>> >
> >>> >> 3. I don’t believe the original AIP (again, I haven’t read the
> updated
> >>> >> proposal or recent messages on the thread. yet) will meet what many
> >>> users
> >>> >> want out of a multiteam solution
> >>> >>
> >>> >
> >>> > I think we will only see when we try. A lot of people thing they
> would,
> >>> > even if they are warned. I know at least one user (Wealthsimple) who
> >>> > definitely want to use it and they got a very detailed explanation of
> >>> the
> >>> > idea and understand it well. So I am sure that **some** users would.
> >>> But we
> >>> > do not know how many.
> >>> >
> >>> >
> >>> >> To expand on those points a bit more
> >>> >>
> >>> >> On 1. The only components that are shared are, I think, the
> scheduler
> >>> and
> >>> >> the API server, and it’s arguable if that is actually a good idea
> >>> given
> >>> >> those are likely to be the most performance sensitive components
> >>> anyway.
> >>> >>
> >>> >> Additionally the fact that the scheduler is a shared component makes
> >>> >> upgrading it almost a non starter as you would likely need buy-in,
> >>> changes,
> >>> >> and testing form ALL teams using it. I’d argue that this is a huge
> >>> negative
> >>> >> until we finish off the version indepence work of AIP-72.
> >>> >>
> >>> >
> >>> > Quite disagree here - especially that our target is that task-sdk is
> >>> > supposed to provide all isolation that is needed. There should be 0
> >>> changes
> >>> > in the dags needed to upgrade scheduler, api_server, triggerer -
> >>> precisely
> >>> > because we introduced backwards-compatible task-sdk.
> >>> >
> >>> > On 3 my complaint is essentially that this doesn’t go nearly far
> >>> enough. It
> >>> >> doesn’t allow read only views to other teams dags. I don’t think it
> >>> allows
> >>> >> you to be in multiple teams at once. You can’t share a connection
> >>> between
> >>> >> teams but only allow certain specified dags to access it, but would
> >>> have to
> >>> >> either be globally usable, or duplicated-and-kept-in-sync between
> >>> teams. In
> >>> >> short I think it fall short of being useful..
> >>> >>
> >>> >
> >>> > Oh absolutely all that is possible (except sharing single connections
> >>> > between multiple teams - which is a very niche use cases and
> >>> duplication
> >>> > here is perfectly ok as first approximation - and if we need more we
> >>> can
> >>> > add it later).
> >>> >
> >>> > Auth manager RBAC and access is abstracted away, and the Keyclock
> >>> Manager
> >>> > implemented by Vincent allows to manage completely independent and
> >>> separate
> >>> > RBAC based on arguments and resources provided by Airflow. There is
> >>> nothing
> >>> > to prevent the user who configures KeyCloak RBAC to define it in the
> >>> way:
> >>> >
> >>> > if group a > allow to read a and write b
> >>> > if group b > alllow to write b but not a
> >>> >
> >>> > and any other combinations. KeyCloak implementation - pretty advanced
> >>> > already - (and design of auth manager) completely abstracts away both
> >>> > authentication and authorization to KeyCloak and KeyCloak has RBAC
> >>> > management built in. Also any of the users can write their own - even
> >>> > hard-coded authentication manager to do the same if they do not want
> to
> >>> > have configurable KeyCloak. Even SimpleAuthManager could be
> hard-coded
> >>> to
> >>> > provide thiose features.
> >>> >
> >>> >
> >>> >>
> >>> >> So on the surface, I’m no more in favour of using dag bundle as a
> >>> >> replacement for team id as I think most of the above points still
> >>> stand.
> >>> >>
> >>> >
> >>> > We disagree here.
> >>> >
> >>> >>
> >>> >> My counter proposal: We do _nothing_ to core airflow. We work on
> >>> improving
> >>> >> the event-based trigger o fdags (write more triggers for read/check
> >>> remote
> >>> >> Assets etc) so that teams can have 100% isolated deployments but
> still
> >>> >> trigger dags based on asset events from other teams.
> >>> >>
> >>> >
> >>> > That does not solve any of the other design goals - only allows to
> >>> trigger
> >>> > assets a bit more easily (but also it's not entirely solved by AIP-82
> >>> > because it does not solve virtual assets - only ones that have
> defined
> >>> > triggerer and "something" to listen on - which is way more complex
> than
> >>> > just defining asset in a Dag and using it in another). I think we
> can't
> >>> > compare AIP-82 to sharing virtual assets due to complexity of it. I
> >>> > explained it in the doc.
> >>> >
> >>> >
> >>> > I will now go and catch up with the long thread and updated proposal
> >>> and
> >>> >> come back.
> >>> >>
> >>> >
> >>> > Please. I hope the above explaination will help in better
> >>> understanding of
> >>> > the proposal, because I think you had some assumptions that do not
> >>> hold any
> >>> > more with the new proposal.
> >>> >
> >>> > J.
> >>> >
> >>> >
> >>> >>
> >>> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>> >>>
> >>> >>> Just to clarify the relation - I updated the AIP now to refer to
> >>> AIP-82
> >>> >> and
> >>> >>> to explain relation between the "cross-team" and "cross-airflow"
> >>> asset
> >>> >>> triggering - this is what I added:
> >>> >>>
> >>> >>> Note that there is a relation between AIP-82 ("External Driven
> >>> >> Scheduling")
> >>> >>> and this part of the functionality. When you have multiple
> instances
> >>> of
> >>> >>> Airflow, you can use shared datasets - "Physical datasets" - that
> >>> several
> >>> >>> Airflow Instances can use - for example there could be an S3 object
> >>> that
> >>> >> is
> >>> >>> produced by one airflow instance, and consumed by another. That
> >>> requires
> >>> >>> deferred trigger to monitor for such datasets, and appropriate
> >>> >> permissions
> >>> >>> to the external dataset, and you could achive similar result to
> >>> >> cross-team
> >>> >>> dataset triggering (but cross airflow). However the feature of
> >>> sharing
> >>> >>> datasets between the teams also works for virtual assets, that do
> not
> >>> >> have
> >>> >>> physically shared "objects" and trigger that is monitoring for
> >>> changes in
> >>> >>> such asset.
> >>> >>>
> >>> >>> J.
> >>> >>>
> >>> >>>
> >>> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com>
> >>> wrote:
> >>> >>>
> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
> >>> reference
> >>> >> to
> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
> >>> through.
> >>> >>>>
> >>> >>>> Yep. It did not - because I did not think it was needed or even
> very
> >>> >>>> important after the simplifications. AIP-82 has a different scope,
> >>> >> really.
> >>> >>>> It only helps when the Assets are "real" data files which we have
> >>> >> physical
> >>> >>>> triggers for, it's slightly related - sharing datasets between
> teams
> >>> >>>> (including those that do not require physical files and triggers)
> is
> >>> >> still
> >>> >>>> possible in the design we have now, but it's not (and never was)
> the
> >>> >>>> **only** reason for having multi-team. There always was (and still
> >>> is)
> >>> >> the
> >>> >>>> possibility of having a common, distinct environments (i.e.
> >>> dependencies
> >>> >>>> and providers) per team, the possibility of having connections and
> >>> >>>> variables that are only accessible to one team and not the other,
> >>> and
> >>> >>>> isolating workload execution (all that while allowing to manage
> >>> multiple
> >>> >>>> team and schedule things with single deployment). That did not
> >>> change.
> >>> >> What
> >>> >>>> changed a lot is that it is now way simpler, something that we can
> >>> >>>> implement without heavy changes to the codebase - and give it to
> our
> >>> >> users,
> >>> >>>> so that they can assess if this is something they need without too
> >>> much
> >>> >>>> risk and effort.
> >>> >>>>
> >>> >>>> This was - I believe the main concern, that the value we get from
> >>> it is
> >>> >>>> not dramatic, but the required changes are huge. This "redesign"
> >>> changes
> >>> >>>> the equation - the value is still unchanged, but the cost of
> >>> >> implementing
> >>> >>>> it and impact on the Airflow codebase is much smaller. I still
> have
> >>> not
> >>> >>>> heard back from Ash if my proposal responds to his original
> concern
> >>> >> though,
> >>> >>>> so I am mostly guessing (also based on the positive impact of
> >>> others)
> >>> >> that
> >>> >>>> yes it does. But to be honest I am not sure and I would love to
> hear
> >>> >> back,
> >>> >>>> I decided to update the AIP to reflect it - regardless, because I
> >>> think
> >>> >> the
> >>> >>>> simplification I proposed keeps the original goals, but is indeed
> >>> way
> >>> >>>> simpler.
> >>> >>>>
> >>> >>>>> This is a very difficult thread to catch up on.
> >>> >>>>
> >>> >>>> Valid point. Let me summarize what is the result:
> >>> >>>>
> >>> >>>> * I significantly simplified the implementation proposal comparing
> >>> to
> >>> >> the
> >>> >>>> original version
> >>> >>>> * main simplification is very limited impact on existing database
> -
> >>> >>>> without "ripple effect" that would require us to change a lot of
> >>> tables,
> >>> >>>> including their primary keys, and heavily impact the UI
> >>> >>>> * this is now more of an incremental change that can be
> implemented
> >>> way
> >>> >>>> faster and with far less risk
> >>> >>>> * updated idea is based on leveraging bundles (already part of our
> >>> data
> >>> >>>> model) to map them (many-to-one) to a team - which requires to
> just
> >>> >> extend
> >>> >>>> the data model with bundle mapping and add team_id to connections
> >>> and
> >>> >>>> variables. Those are all needed DB changes.
> >>> >>>>
> >>> >>>> The AIP is updated - in a one single big change so It should be
> >>> easy to
> >>> >>>> compare the changes:
> >>> >>>>
> >>> >>
> >>>
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
> >>> >>>> -> I even named the version appropriately "Simplified multi-team
> >>> AIP" -
> >>> >> you
> >>> >>>> can select and compare v.65 with v.66 to see the exact
> differences I
> >>> >>>> proposed.
> >>> >>>>
> >>> >>>> I hope it will be helpful to catch up and for those who did not
> >>> follow,
> >>> >> to
> >>> >>>> be able to make up their minds about it.
> >>> >>>>
> >>> >>>> J.
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
> >>> >> <vi...@astronomer.io.invalid>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> This is a very difficult thread to catch up on.
> >>> >>>>> I will take a detailed look at the AIP update to try to figure
> out
> >>> the
> >>> >>>>> changes in the proposal.
> >>> >>>>>
> >>> >>>>> From a quick glance, the updated AIP didn't seem to have any
> >>> reference
> >>> >> to
> >>> >>>>> AIP-82, which surprised me, but will take a more detailed read
> >>> through.
> >>> >>>>>
> >>> >>>>> Vikram
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
> >>> >>>>> gopidesupavan@gmail.com>
> >>> >>>>> wrote:
> >>> >>>>>
> >>> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's much
> >>> more
> >>> >> slim
> >>> >>>>>> down.
> >>> >>>>>>
> >>> >>>>>> left a minor comment. :) Overall looking great.
> >>> >>>>>>
> >>> >>>>>> Pavan
> >>> >>>>>>
> >>> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
> >>> >>>>> <j_scheffler@gmx.de.invalid
> >>> >>>>>>>
> >>> >>>>>> wrote:
> >>> >>>>>>
> >>> >>>>>>> Thanks for the rework/update of the AIP-72!
> >>> >>>>>>>
> >>> >>>>>>> Just a few small comments but overall I like it as it is much
> >>> leaner
> >>> >>>>>>> than originally planned and is in a level of complexity that it
> >>> >> really
> >>> >>>>>>> seems to be a benefit to close the gap as described.
> >>> >>>>>>>
> >>> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
> >>> >>>>>>>> I updated the AIP - including architecture images and reviewed
> >>> it
> >>> >>>>>> (again)
> >>> >>>>>>>> and corrected any ambiguities and places where it needed to be
> >>> >>>>> changed.
> >>> >>>>>>>>
> >>> >>>>>>>> I think the current state
> >>> >>>>>>>>
> >>> >>>>>>>
> >>> >>>>>>
> >>> >>>>>
> >>> >>
> >>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >>> >>>>>>>> - nicely describes the proposal.
> >>> >>>>>>>>
> >>> >>>>>>>> Comparing to the previous one:
> >>> >>>>>>>>
> >>> >>>>>>>> 1. The DB changes are far less intrusive - no ripple effect on
> >>> >>>>> Airflow
> >>> >>>>>>>> 2. There is no need to merge configurations and provide
> >>> different
> >>> >>>>> set
> >>> >>>>>> of
> >>> >>>>>>>> configs per team - we can add it later but I do not see why we
> >>> need
> >>> >>>>> it
> >>> >>>>>> in
> >>> >>>>>>>> this simplified version
> >>> >>>>>>>> 3. We can still configure a different set of executors per
> team
> >>> -
> >>> >>>>> that
> >>> >>>>>> is
> >>> >>>>>>>> already implemented (we just need to wire it to the bundle ->
> >>> team
> >>> >>>>>>> mapping).
> >>> >>>>>>>>
> >>> >>>>>>>> I think it will be way simpler and faster to implement this
> way
> >>> and
> >>> >>>>> it
> >>> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can
> >>> give
> >>> >>>>> our
> >>> >>>>>>>> users so that they can provide feedback.
> >>> >>>>>>>>
> >>> >>>>>>>> J.
> >>> >>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>>
> >>> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <
> jarek@potiuk.com>
> >>> >>>>> wrote:
> >>> >>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>> I like this iteration a bit more now for sure, thanks for
> >>> being
> >>> >>>>>>> receptive
> >>> >>>>>>>>>> to feedback! :)
> >>> >>>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>> This now becomes quite close to what was proposing before,
> we
> >>> now
> >>> >>>>>> again
> >>> >>>>>>>>>> have a team ID (which I think is really needed here, glad to
> >>> see
> >>> >>>>> it
> >>> >>>>>>> back)
> >>> >>>>>>>>>> and it will be used for auth management, configuration
> >>> >>>>> specification,
> >>> >>>>>>> etc
> >>> >>>>>>>>>> but will be carried by Bundle instead of the dag model.
> Which
> >>> as
> >>> >>>>> you
> >>> >>>>>>> say
> >>> >>>>>>>>>> “For that we will need to make sure that both api-server,
> >>> >>>>> scheduler
> >>> >>>>>> and
> >>> >>>>>>>>>> triggerer have access to the "bundle definition" (to perform
> >>> the
> >>> >>>>>>> mapping)"
> >>> >>>>>>>>>> which honestly doesn’t feel too much different from the
> >>> original
> >>> >>>>>>> proposal
> >>> >>>>>>>>>> we had last week of adding it to Dag table and ensuring it’s
> >>> >>>>>> available
> >>> >>>>>>>>>> everywhere. but either way I’m happy to meet in the middle
> and
> >>> >>>>> keep
> >>> >>>>>> it
> >>> >>>>>>> on
> >>> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable
> location.
> >>> >>>>>>>>>>
> >>> >>>>>>>>> I think the big difference is the "ripple effect" that was
> >>> >>>>> discussed
> >>> >>>>>> in
> >>> >>>>>>>>>
> >>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
> >>> >>>>>> (and I
> >>> >>>>>>>>> believe - correct me if I am wrong Ash - important trigger
> for
> >>> the
> >>> >>>>>>>>> discussion) so far what we wanted is to extend the primary
> key
> >>> and
> >>> >>>>> it
> >>> >>>>>>> would
> >>> >>>>>>>>> ripple through all the pieces of Airflow -> models, API, UI
> >>> etc.
> >>> >>>>> ...
> >>> >>>>>>>>> However - we already have `bundle_name" and "bundle_version"
> >>> in the
> >>> >>>>>> Dag
> >>> >>>>>>>>> model. So I think when we add a separate table where we map
> the
> >>> >>>>> bundle
> >>> >>>>>>> to
> >>> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not
> want
> >>> to
> >>> >>>>>> change
> >>> >>>>>>>>> primary key, we do not want to change UI in any way (except
> >>> >>>>> filtering
> >>> >>>>>> of
> >>> >>>>>>>>> DAGs available based on your team - but that will be handled
> in
> >>> >>>>> Auth
> >>> >>>>>>>>> Manager and will not impact UI in any way, I think that's a
> >>> huge
> >>> >>>>>>>>> simplification of the implementation, and if we agree to it
> - i
> >>> >>>>> think
> >>> >>>>>> it
> >>> >>>>>>>>> should speed up the implementation significantly. There are
> >>> only a
> >>> >>>>>>> limited
> >>> >>>>>>>>> number of times where you need to look up the team_id - so
> >>> having
> >>> >>>>> the
> >>> >>>>>>>>> bundle -> team mapping in a separate table and having to look
> >>> them
> >>> >>>>> up
> >>> >>>>>>>>> should not be a problem. And it has much less complexity and
> >>> >>>>>>>>> "ripple-effect" through the codebase (for example I could
> >>> imagine
> >>> >>>>> 100s
> >>> >>>>>>> or
> >>> >>>>>>>>> thousands already written tests that would have to be adapted
> >>> if we
> >>> >>>>>>> changed
> >>> >>>>>>>>> the primary key - where there will be pretty much zero impact
> >>> on
> >>> >>>>>>> existing
> >>> >>>>>>>>> tests if we just add bundle -> team lookup table.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>> One other thing I’d point out is that I think including
> >>> executors
> >>> >>>>> per
> >>> >>>>>>>>>> team is a very easy win and quite possible without much
> work.
> >>> I
> >>> >>>>>> already
> >>> >>>>>>>>>> have much of the code written. Executors are already aware
> of
> >>> >>>>> Teams
> >>> >>>>>>> that
> >>> >>>>>>>>>> own them (merged), I have a PR open to have configuration
> per
> >>> team
> >>> >>>>>>> (with a
> >>> >>>>>>>>>> quite simple and isolated approach, I believe you approved
> >>> Jarek).
> >>> >>>>>> The
> >>> >>>>>>> last
> >>> >>>>>>>>>> piece is updating the scheduling logic to route tasks from a
> >>> >>>>>> particular
> >>> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much work
> >>> >>>>> (though
> >>> >>>>>> it
> >>> >>>>>>>>>> would be easier if the Task models had a column for the team
> >>> they
> >>> >>>>>>> belong
> >>> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to get
> >>> the
> >>> >>>>>> team) I
> >>> >>>>>>>>>> have a branch where I was experimenting with this logic
> >>> already.
> >>> >>>>>>>>>> Any who, long story short, I don’t think we necessarily need
> >>> to
> >>> >>>>>> remove
> >>> >>>>>>>>>> this piece from the project's scope if it is already partly
> >>> done
> >>> >>>>> and
> >>> >>>>>>> not
> >>> >>>>>>>>>> too difficult.
> >>> >>>>>>>>>>
> >>> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want to
> just
> >>> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we
> have
> >>> it
> >>> >>>>> in.
> >>> >>>>>>> And
> >>> >>>>>>>>> if we can make it in, easily (which it seems we can) - we can
> >>> also
> >>> >>>>>>> include
> >>> >>>>>>>>> it in the first iteration. What I wanted to avoid really
> (from
> >>> the
> >>> >>>>>>> original
> >>> >>>>>>>>> design) - again trying to simplify it, limit the changes, and
> >>> >>>>> speed up
> >>> >>>>>>>>> implementation. And there is one "complexity" that I wanted
> to
> >>> >>>>> avoid
> >>> >>>>>>>>> specifically - having to have separate , additional
> >>> configuration
> >>> >>>>> per
> >>> >>>>>>> team.
> >>> >>>>>>>>> Not only because it complicates already complex configuration
> >>> >>>>> handling
> >>> >>>>>>> (I
> >>> >>>>>>>>> know we have PR for that) but mostly because if it is not
> >>> needed,
> >>> >>>>> we
> >>> >>>>>> can
> >>> >>>>>>>>> simplify documentation and explain to our users easier what
> >>> they
> >>> >>>>> need
> >>> >>>>>>> to do
> >>> >>>>>>>>> to have their own multi-team setup. And I am quite open to
> >>> keeping
> >>> >>>>>>>>> multiple-executors if we can avoid complicating
> configuration.
> >>> >>>>>>>>>
> >>> >>>>>>>>> But I think some details of that and whether we really need
> >>> >>>>> separate
> >>> >>>>>>>>> configuration might also come as a result of updating the AIP
> >>> - I
> >>> >>>>> am
> >>> >>>>>> not
> >>> >>>>>>>>> quite sure now if we need it, but we can discuss it when we
> >>> >>>>> iterate on
> >>> >>>>>>> the
> >>> >>>>>>>>> AIP.
> >>> >>>>>>>>>
> >>> >>>>>>>>> J.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> ---------------------------------------------------------------------
> >>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>> >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>
> >>> >>>>>
> >>> >>>>
> >>> >>
> >>> >>
> >>> >>
> ---------------------------------------------------------------------
> >>> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>> >> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>
> >>>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

["Oliveira, Niko" <on...@amazon.com.INVALID>]

Hey Jarek,


The direction this one is taking is interesting. If you're really just trying to make the feature barely possible and mostly targeted towards managed providers to implement the rest, then I suppose this hits the mark.

But this is not something we're asking for at Amazon and personally I think we should make the feature reasonably usable for those running self-managed OSS Airflow as well. There are many users running an on-prem Airflow. Getting too hyper-fixated on an implementation that's so simplified that it's obtuse and difficult to use by most users seems like the wrong approach to me. But you and I have already discussed this at length and I haven't convinced you so far, so if I'm the only one with this thinking then I'm happy to disagree and commit as we say at Amazon :)


> So I would be rather strong on **not** touching the current configuration and

simply adding configuration for per-team executors in executor config -
even if it is uglier and more "low-level".

Can you explain what "adding configuration for per-team executors in executor config" would look like? I don't have a concrete sense of what you mean by this.

Thanks for your efforts on trying to get this feature agreed to and voted on. Looking forward to working on the project in the coming weeks!

Cheers,
Niko

________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Tuesday, July 1, 2025 10:26:55 PM
To: dev@airflow.apache.org
Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.



Any last comments ? There is a long weekend coming up in the US, so I will
likely start voting on the updated AIP on Monday 7th.

On Fri, Jun 27, 2025 at 12:41 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> I'd really love to finalise discussion and put it up to a vote some time
> after the recording from the last dev call is posted - so that more
> context, details and the LONG discussion we had on it. There is no *huge*
> hurry  - we have strong dependency on Task Isolation and it seems that it
> will still take a bit of time to complete, so I'd say I would love to start
> voting in about a week time - so that maybe at the next dev call we can
> "seal" the subject. Happy to see any more comments - especially from those
> who have opinions but they had no opportunity to express them.
>
> I am personally very happy with the direction it took - simplification and
> "MVP" kind of approach - also I invite the stakeholders of ours to take a
> close look at the scope and what we really propose - I have a feeling that
> we can balance it out - there is something we can make to make it not
> "worse" for the offerings they have. I think we have a really good
> symbiotic relationship here, and I would love to leverage that. For one -
> my goal here is to have a minimum number of changes that are impacting
> maintainability of the open-source airflow - but mostly "opening up some
> possibilities" - rather than provide turn-key solutions. And mostly because
> this is good for all sides - less maintenance and complexity for OSS
> maintainers, but more opportunities to make it into "turn-key" solutions by
> the stakeholders, while also allowing the "on-prem" users - if they are
> highly motivated - to use those features by adding the "turn-key" layer on
> their own. Also adding multi-team should not be at the expense of "simple"
> installations - they should be virtually unaffected.
>
> One example of applying this is cutting on "separate config files". I
> think it moves us closer to a "turn-key" solution but it is not really
> necessary to achieve the three goals above - that's why in the current
> proposal this part is completely removed - Sorry Niko, but I still think
> it's one of the things that falls into this bucket. We can easily remove
> it, they complicate code, documentation and options the users have, and
> even if it is a "little" more complex to manage configuration by motivated
> users, it's also an opportunity for "turn-key" option that stakeholders can
> build in their products - and we do not have to maintain it in the
> open-source. So I would be rather strong on **not** touching the current
> configuration and simply adding configuration for per-team executors in
> executor config - even if it is uglier and more "low-level".
>
> So if there are some constructive ideas on what can be done to make it
> "simpler" and less "turn-key" in that respect - I would highly value such
> ideas and comments. If we can cut down something more that is not
> "necessary" for the three primary goals I came up with - I am more than
> happy to do it.
>
> Just to remind - those are the "extracted" goals. I slightly updated them
> and added to the preamble of the AIP:
>
> * less operational overhead for managing multi-team (once AIP-72 is
> complete) where separate execution environments are important
> * virtual assets sharing between teams
> * ability of having "admin" and "team sharing" capability where dags from
> multiple teams can be seen in a single Airflow UI  (requires custom RBAC an
> AIP-56 implementation of Auth Manager - with KeyCloak Auth Manager being a
> reference implementation)
>
> J.
>
>
> On Thu, Jun 26, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>>
>>> One technical observation: Now that the dag table no longer has a
>>> team_id in it, what would the behaviour be when a DAG is attempted to move
>>> between bundles? How do we detect this? (I’m not all convinced that we
>>> correctly detect duplicate dag ids across bundles today, so I wouldn’t
>>> assume or rely on the current behaviour.)
>>>
>>
>> Of course - yes, I realise that - that problem was also not handled in
>> the previous iteration to be honest. That is something that dag bundle
>> solution allows to solve eventually - but I do not think it's a blocker for
>> the proposed implementation. We will have to eventually add some way of
>> blocking dags to jump between bundles, we might tackle this separately. I
>> already wanted to propose a separate update to that - but I did not want to
>> complicate the current proposal. One thing at a time. I can, however - if
>> you consider that as a blocker, extend the current AIP with it. Not a big
>> problem. This is however a bit independent from the team_id introduction.
>>
>> Overall, I am still unconvinced this proposal has enough real user
>>> benefit over actually separate deployments, and on balance of the added
>>> complexity and maintenance burden I do not think it is worth it.
>>>
>>
>> That makes me sad, I thought that over the course of the discussion I
>> addressed all the concerns (in this case the concern was "is it worth with
>> the cost and little benefit", but when I did it and heavily limited the
>> impact, now the concern is "is it worth at all as changes are really
>> minimal" - and surely, anyone can change and adapt their concerns, over
>> time,  but that one seems like ever-moving target. I hoped at least for
>> some acknowledgment of some concerns (complexity in this case) is
>> addressed, but it seems that you are deeply convinced that we do not need
>> multi-team at all (which is in stark contrast with at least a dozen of
>> bigger and smaller users of Airflow who submitted talks to Airflow summit
>> (including about 5 or 6 submissions for Airflow 2025) on how they spent
>> their engineering effort, time and money on trying to achieve something
>> similar - they assessed that it's worth, you  assess that it's not worth.
>> Somehow I trust our users that they were not spending the money, time and
>> engineering effort to achieve this because they wanted to spend more money.
>> I think they assessed it's worth it. So I want to make it a bit easier and
>> more "proper" way for them to do that.
>>
>>>
>>> Upgrades: it is not easier to upgrade under this multi team proposal,
>>> but much much harder. This is based on hard earned experience from helping
>>> Astronomer users — having to coordinate upgrades between multiple teams
>>> turns in to a months long slog of the hardest kind of work —  people work:
>>> getting other teams to agree to do things that they don’t directly care
>>> about — “It’s working for me, I don’t care about upgrading, we’ll get to it
>>> next quarter” is a refrain I’ve heard many times.
>>>
>>
>> Yes. absolutely - this is why we deferred it until we knew what shape
>> task isolation and other AIPs we depend on take on. Because it is clear
>> that pretty much all the problem you explain above are going to be solved
>> with task isolation. And it's not just my opinion. If you want to argue
>> with it, you likely need to argue with yourself:
>> https://github.com/apache/airflow/issues/51545#issuecomment-2980038478.
>> Let me quote what you wrote there last week:
>>
>> Ash Berlin Taylor wrote:
>>
>> > A tight coupling between task-sdk and any "server side" component is
>> the opposite to one of the goals of AIP-72 (I'm not sure we ever explicitly
>> said this, but the first point of motivation for the AIP says "Dependency
>> conflicts for administrators supporting data teams using different versions
>> of providers, libraries, or python packages")
>> > In short, my goal with TaskSDK, and the reason for introducing CalVer
>> and Cadwyn with the execution API is to end up in a world where you can
>> upgrade the Airflow Scheduler/API server interdependently of any worker
>> nodes (with the exception that the server must be at least as new as the
>> clients)
>> > This ability to have version-skew is pretty much non-negotiable to me
>> and is (other than other languages) one of primary benefits of AIP-72
>>
>> If you read yourself from that quote it basically means "it will be easy
>> to upgrade airflow independently of workers". So I am a bit confused here.
>> Yes, I agree it was difficult, but you yourself explain that when AIP-72
>> (which since API-67 has been accepted has always beem prerequisite of it)
>> wrote it will be "easy". So I am not sure why you are bringing it now. We
>> assume AIP-72 will be completed and this problem will be gone. Let's not
>> mention it any more please.
>>
>> The true separation from TaskSDK will likely only land in about 3.2 time
>>> frame. We are actively working on it, but it’s a slow process of untangling
>>> lots of assumptions made in the code base over the years. Maybe once we
>>> have that my view would be different, but right now I think this makes the
>>> proposal a non-starter. Especially as you are saying that most teams will
>>> have unique connections. If they’ve got those already, then having an asset
>>> trigger use those conns to watch/poll for activity is a much easier
>>> solution to operate and crucially, to scale and upgrade.
>>>
>>
>> Yes. I perfectly understand that and I am fully aware of potentially 3.2
>> time-frame. And that's fine. Actually I heartily invite you to listen to
>> the part of my talk from Berlin Buzzwords when I was asked for the timeline
>> - https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the exact
>> timeline in my talk . My answer was basically - "3.1" or "3.2", and I
>> sincerely hope "3.1" but we might not be able to complete it because we
>> have other things to do (other - is indeed the Task Isolation work that you
>> are leading). And that's perfectly fine. And it absolutely does not prevent
>> us from voting on the AIP now - similarly as we voted on the previous
>> version of the AIP - knowing that it has some prerequisites a few months
>> ago. Especially that we know that the feature we need from task isolation
>> is "non-negotiable". I.e. it WILL happen. We don't hope for it, we know it
>> will be there. Those are your own words.
>>
>>
>>> >  I think we can’t compare AIP-82 to sharing virtual assets due to
>>> complexity of it.
>>>
>>> Virtual Assets was a mistake, and not how users actually want to use
>>> them. Mea culpa
>>>
>>
>> This is the first time I hear this - certainly you never raised this
>> concern on the devlist. So if you have some concerns about virtual assets I
>> think you should raise it on the devlist, because I think everyone here is
>> missing some conversation (or maybe it's just your private opinion that
>> you never shared with anyone, but maybe it's worth). I would be
>> interested to hear how the feature that was absolutely most successful
>> feature of airflow 2 was a mistake. According to the 2024 survey
>> https://airflow.apache.org/blog/airflow-survey-2024/  - 48% of Airflow
>> users have been using it, even if it was added as one of the last
>> big features of Airflow 2. It's the MOST used feature out of all the
>> features out there. I would be really curious to see how it was a mistake
>> (but  please start a separate thread explaining why you think it was a
>> mistake, what are your data points and what do you think should be fixed.
>> Just dropping "virtual assets were a mistake" in the middle of multi-team
>> conversation seems completely unjustified without knowing what you are
>> talking about. So I think, until we know more, this argument has no base.
>>
>>
>>>
>>> S
>>> To restate my points:
>>>
>>> - Sharing a deployment between teams today/in 3.1 is operationally more
>>> complex (both scaling, and upgrades) — this is a con, not a plus.
>>>
>>
>> Surely. But it will be easier when AIP-72 is complete (which I am
>> definitely looking forward to and as clearly explained in AIP-82, is a
>> prerequisite of it). Nothing changed here.
>>
>>
>>> - The main user benefit appears to be “allow teams’ DAGs to communicate
>>> via Assets”, in which case we can do that today by putting more work in to
>>> AIP-82’s Asset triggers
>>>
>>
>> No. Lower operational complexity for multi-teams (providing that we
>> deliver AIP-72) is another benefit. Virtual assets is another, and since
>> there is no ground in "virtual assets is a mistake" statement (not until
>> you explain what you mean by that in a separate discussion) - this is also
>> still a very valid point.
>>
>>
>>> Soon, we will have then be asked about cross-team governance, policy
>>> enforcement, and potentially unbounded edge cases (e.g., team-specific
>>> secrets, roles, quotas). ain, you get this for free with truely separate
>>> deployments already
>>> allow different teams to use different executors (including multiple
>>> executors per-team following AIP-61)
>>>
>>
>> Not really. We very explicitly say in the AIP that his is not a goal and
>> that we have no plans for. And yes, using separate executors per team is
>> actually back in the AIP-82 in case you did not notice (and the code needed
>> for it's even implemented and merged already in main by Vincent).
>>
>>
>>> Provably not true right now, and until ~3.2 delivers the full Task
>>> SDK/Core dependency separation this would be _more_ work to upgrade, not
>>> less, and that work is not shared but still on a central team.
>>>
>>
>> Absolutely - we will wait for AIP-72 completion. I do not want to say 3.1
>> or 3.2 directly - because there are - as you said - a lot of moving pieces.
>> So my target for multi-team is "After AIP-72 is completed". Full stop. But
>> there is nothing wrong with accepting the AIP now and doing preparatory
>> work in parallel. Similarly as there is no way to have a baby in 1 month by
>> 9 women, there is no way adding more effort to task-sdk isolation will
>> speed it up - we alredy have not only 3 people (you leading it, Kaxil and
>> Amog) but also all the help from me and even 10s of different contributors
>> (for example with the recent db_test cleanup that I took leadership on) -
>> and there are people who wish to work on adding multi-team features. Since
>> the design heavily limits impact on airflow codebase and interactions with
>> task-sdk implementation, there is nothing wrong with starting
>> implementation in parallel either- amazon team is keen to move it forward -
>> they even already implemented SQS trigger for assets, and we are working
>> together on FAB removal, Keycloak authentication manager - and they seem to
>> still have capacity and drive to progress multi-team. So I am not sure if
>> we are trading off something. There is no "if we work on more on task sdk
>> and drop multi-team things will be faster". Generally in open source people
>> work in the area where they feel they can provide best value - such as you
>> working on task-sdk, me on CI,dev env, they will deliver more value on
>> multi-team
>>
>>
>>>
>>> So please, as succinctly as possible, please tell me what the direct
>>> benefit to users this proposal is over us putting this effort in to writing
>>> better Asset triggers instead?
>>>
>>
>>
>> * less operational overhead for managing multi-team (once AIP-72 is
>> complete) where separate execution environments are important
>> * virtual assets sharing
>> * ability of having "admin" and "team sharing" capability where dags from
>> multiple teams can be seen in a single Airflow UI  (requires custom RBAC)
>>
>> None of this can be done via beter asset triggers
>>
>>
>>>
>>> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >
>>> > My counter-points:
>>> >
>>> >
>>> >> 1. Managing a multi team deployment is not materially different from
>>> >> managing a deployment per team
>>> >>
>>> >
>>> > It's a bit easier - especially when it comes to upgrades (especially
>>> in the
>>> > case we are targetting when we are not targetting multi-tenant, but
>>> several
>>> > relatively closely cooperating teams with different dependncy
>>> requiremens
>>> > and isolation need.
>>> >
>>> > 2. The database changes were quite wide-reaching
>>> >>
>>> >
>>> > Yes. that is addressed.
>>> >
>>> >
>>> >> 3. I don’t believe the original AIP (again, I haven’t read the updated
>>> >> proposal or recent messages on the thread. yet) will meet what many
>>> users
>>> >> want out of a multiteam solution
>>> >>
>>> >
>>> > I think we will only see when we try. A lot of people thing they would,
>>> > even if they are warned. I know at least one user (Wealthsimple) who
>>> > definitely want to use it and they got a very detailed explanation of
>>> the
>>> > idea and understand it well. So I am sure that **some** users would.
>>> But we
>>> > do not know how many.
>>> >
>>> >
>>> >> To expand on those points a bit more
>>> >>
>>> >> On 1. The only components that are shared are, I think, the scheduler
>>> and
>>> >> the API server, and it’s arguable if that is actually a good idea
>>> given
>>> >> those are likely to be the most performance sensitive components
>>> anyway.
>>> >>
>>> >> Additionally the fact that the scheduler is a shared component makes
>>> >> upgrading it almost a non starter as you would likely need buy-in,
>>> changes,
>>> >> and testing form ALL teams using it. I’d argue that this is a huge
>>> negative
>>> >> until we finish off the version indepence work of AIP-72.
>>> >>
>>> >
>>> > Quite disagree here - especially that our target is that task-sdk is
>>> > supposed to provide all isolation that is needed. There should be 0
>>> changes
>>> > in the dags needed to upgrade scheduler, api_server, triggerer -
>>> precisely
>>> > because we introduced backwards-compatible task-sdk.
>>> >
>>> > On 3 my complaint is essentially that this doesn’t go nearly far
>>> enough. It
>>> >> doesn’t allow read only views to other teams dags. I don’t think it
>>> allows
>>> >> you to be in multiple teams at once. You can’t share a connection
>>> between
>>> >> teams but only allow certain specified dags to access it, but would
>>> have to
>>> >> either be globally usable, or duplicated-and-kept-in-sync between
>>> teams. In
>>> >> short I think it fall short of being useful..
>>> >>
>>> >
>>> > Oh absolutely all that is possible (except sharing single connections
>>> > between multiple teams - which is a very niche use cases and
>>> duplication
>>> > here is perfectly ok as first approximation - and if we need more we
>>> can
>>> > add it later).
>>> >
>>> > Auth manager RBAC and access is abstracted away, and the Keyclock
>>> Manager
>>> > implemented by Vincent allows to manage completely independent and
>>> separate
>>> > RBAC based on arguments and resources provided by Airflow. There is
>>> nothing
>>> > to prevent the user who configures KeyCloak RBAC to define it in the
>>> way:
>>> >
>>> > if group a > allow to read a and write b
>>> > if group b > alllow to write b but not a
>>> >
>>> > and any other combinations. KeyCloak implementation - pretty advanced
>>> > already - (and design of auth manager) completely abstracts away both
>>> > authentication and authorization to KeyCloak and KeyCloak has RBAC
>>> > management built in. Also any of the users can write their own - even
>>> > hard-coded authentication manager to do the same if they do not want to
>>> > have configurable KeyCloak. Even SimpleAuthManager could be hard-coded
>>> to
>>> > provide thiose features.
>>> >
>>> >
>>> >>
>>> >> So on the surface, I’m no more in favour of using dag bundle as a
>>> >> replacement for team id as I think most of the above points still
>>> stand.
>>> >>
>>> >
>>> > We disagree here.
>>> >
>>> >>
>>> >> My counter proposal: We do _nothing_ to core airflow. We work on
>>> improving
>>> >> the event-based trigger o fdags (write more triggers for read/check
>>> remote
>>> >> Assets etc) so that teams can have 100% isolated deployments but still
>>> >> trigger dags based on asset events from other teams.
>>> >>
>>> >
>>> > That does not solve any of the other design goals - only allows to
>>> trigger
>>> > assets a bit more easily (but also it's not entirely solved by AIP-82
>>> > because it does not solve virtual assets - only ones that have defined
>>> > triggerer and "something" to listen on - which is way more complex than
>>> > just defining asset in a Dag and using it in another). I think we can't
>>> > compare AIP-82 to sharing virtual assets due to complexity of it. I
>>> > explained it in the doc.
>>> >
>>> >
>>> > I will now go and catch up with the long thread and updated proposal
>>> and
>>> >> come back.
>>> >>
>>> >
>>> > Please. I hope the above explaination will help in better
>>> understanding of
>>> > the proposal, because I think you had some assumptions that do not
>>> hold any
>>> > more with the new proposal.
>>> >
>>> > J.
>>> >
>>> >
>>> >>
>>> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >>>
>>> >>> Just to clarify the relation - I updated the AIP now to refer to
>>> AIP-82
>>> >> and
>>> >>> to explain relation between the "cross-team" and "cross-airflow"
>>> asset
>>> >>> triggering - this is what I added:
>>> >>>
>>> >>> Note that there is a relation between AIP-82 ("External Driven
>>> >> Scheduling")
>>> >>> and this part of the functionality. When you have multiple instances
>>> of
>>> >>> Airflow, you can use shared datasets - "Physical datasets" - that
>>> several
>>> >>> Airflow Instances can use - for example there could be an S3 object
>>> that
>>> >> is
>>> >>> produced by one airflow instance, and consumed by another. That
>>> requires
>>> >>> deferred trigger to monitor for such datasets, and appropriate
>>> >> permissions
>>> >>> to the external dataset, and you could achive similar result to
>>> >> cross-team
>>> >>> dataset triggering (but cross airflow). However the feature of
>>> sharing
>>> >>> datasets between the teams also works for virtual assets, that do not
>>> >> have
>>> >>> physically shared "objects" and trigger that is monitoring for
>>> changes in
>>> >>> such asset.
>>> >>>
>>> >>> J.
>>> >>>
>>> >>>
>>> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> >>>
>>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>>> reference
>>> >> to
>>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>>> through.
>>> >>>>
>>> >>>> Yep. It did not - because I did not think it was needed or even very
>>> >>>> important after the simplifications. AIP-82 has a different scope,
>>> >> really.
>>> >>>> It only helps when the Assets are "real" data files which we have
>>> >> physical
>>> >>>> triggers for, it's slightly related - sharing datasets between teams
>>> >>>> (including those that do not require physical files and triggers) is
>>> >> still
>>> >>>> possible in the design we have now, but it's not (and never was) the
>>> >>>> **only** reason for having multi-team. There always was (and still
>>> is)
>>> >> the
>>> >>>> possibility of having a common, distinct environments (i.e.
>>> dependencies
>>> >>>> and providers) per team, the possibility of having connections and
>>> >>>> variables that are only accessible to one team and not the other,
>>> and
>>> >>>> isolating workload execution (all that while allowing to manage
>>> multiple
>>> >>>> team and schedule things with single deployment). That did not
>>> change.
>>> >> What
>>> >>>> changed a lot is that it is now way simpler, something that we can
>>> >>>> implement without heavy changes to the codebase - and give it to our
>>> >> users,
>>> >>>> so that they can assess if this is something they need without too
>>> much
>>> >>>> risk and effort.
>>> >>>>
>>> >>>> This was - I believe the main concern, that the value we get from
>>> it is
>>> >>>> not dramatic, but the required changes are huge. This "redesign"
>>> changes
>>> >>>> the equation - the value is still unchanged, but the cost of
>>> >> implementing
>>> >>>> it and impact on the Airflow codebase is much smaller. I still have
>>> not
>>> >>>> heard back from Ash if my proposal responds to his original concern
>>> >> though,
>>> >>>> so I am mostly guessing (also based on the positive impact of
>>> others)
>>> >> that
>>> >>>> yes it does. But to be honest I am not sure and I would love to hear
>>> >> back,
>>> >>>> I decided to update the AIP to reflect it - regardless, because I
>>> think
>>> >> the
>>> >>>> simplification I proposed keeps the original goals, but is indeed
>>> way
>>> >>>> simpler.
>>> >>>>
>>> >>>>> This is a very difficult thread to catch up on.
>>> >>>>
>>> >>>> Valid point. Let me summarize what is the result:
>>> >>>>
>>> >>>> * I significantly simplified the implementation proposal comparing
>>> to
>>> >> the
>>> >>>> original version
>>> >>>> * main simplification is very limited impact on existing database -
>>> >>>> without "ripple effect" that would require us to change a lot of
>>> tables,
>>> >>>> including their primary keys, and heavily impact the UI
>>> >>>> * this is now more of an incremental change that can be implemented
>>> way
>>> >>>> faster and with far less risk
>>> >>>> * updated idea is based on leveraging bundles (already part of our
>>> data
>>> >>>> model) to map them (many-to-one) to a team - which requires to just
>>> >> extend
>>> >>>> the data model with bundle mapping and add team_id to connections
>>> and
>>> >>>> variables. Those are all needed DB changes.
>>> >>>>
>>> >>>> The AIP is updated - in a one single big change so It should be
>>> easy to
>>> >>>> compare the changes:
>>> >>>>
>>> >>
>>> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
>>> >>>> -> I even named the version appropriately "Simplified multi-team
>>> AIP" -
>>> >> you
>>> >>>> can select and compare v.65 with v.66 to see the exact differences I
>>> >>>> proposed.
>>> >>>>
>>> >>>> I hope it will be helpful to catch up and for those who did not
>>> follow,
>>> >> to
>>> >>>> be able to make up their minds about it.
>>> >>>>
>>> >>>> J.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
>>> >> <vi...@astronomer.io.invalid>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> This is a very difficult thread to catch up on.
>>> >>>>> I will take a detailed look at the AIP update to try to figure out
>>> the
>>> >>>>> changes in the proposal.
>>> >>>>>
>>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>>> reference
>>> >> to
>>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>>> through.
>>> >>>>>
>>> >>>>> Vikram
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>>> >>>>> gopidesupavan@gmail.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's much
>>> more
>>> >> slim
>>> >>>>>> down.
>>> >>>>>>
>>> >>>>>> left a minor comment. :) Overall looking great.
>>> >>>>>>
>>> >>>>>> Pavan
>>> >>>>>>
>>> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>>> >>>>> <j_scheffler@gmx.de.invalid
>>> >>>>>>>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> Thanks for the rework/update of the AIP-72!
>>> >>>>>>>
>>> >>>>>>> Just a few small comments but overall I like it as it is much
>>> leaner
>>> >>>>>>> than originally planned and is in a level of complexity that it
>>> >> really
>>> >>>>>>> seems to be a benefit to close the gap as described.
>>> >>>>>>>
>>> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
>>> >>>>>>>> I updated the AIP - including architecture images and reviewed
>>> it
>>> >>>>>> (again)
>>> >>>>>>>> and corrected any ambiguities and places where it needed to be
>>> >>>>> changed.
>>> >>>>>>>>
>>> >>>>>>>> I think the current state
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>>> >>>>>>>> - nicely describes the proposal.
>>> >>>>>>>>
>>> >>>>>>>> Comparing to the previous one:
>>> >>>>>>>>
>>> >>>>>>>> 1. The DB changes are far less intrusive - no ripple effect on
>>> >>>>> Airflow
>>> >>>>>>>> 2. There is no need to merge configurations and provide
>>> different
>>> >>>>> set
>>> >>>>>> of
>>> >>>>>>>> configs per team - we can add it later but I do not see why we
>>> need
>>> >>>>> it
>>> >>>>>> in
>>> >>>>>>>> this simplified version
>>> >>>>>>>> 3. We can still configure a different set of executors per team
>>> -
>>> >>>>> that
>>> >>>>>> is
>>> >>>>>>>> already implemented (we just need to wire it to the bundle ->
>>> team
>>> >>>>>>> mapping).
>>> >>>>>>>>
>>> >>>>>>>> I think it will be way simpler and faster to implement this way
>>> and
>>> >>>>> it
>>> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can
>>> give
>>> >>>>> our
>>> >>>>>>>> users so that they can provide feedback.
>>> >>>>>>>>
>>> >>>>>>>> J.
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
>>> >>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>> I like this iteration a bit more now for sure, thanks for
>>> being
>>> >>>>>>> receptive
>>> >>>>>>>>>> to feedback! :)
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>> This now becomes quite close to what was proposing before, we
>>> now
>>> >>>>>> again
>>> >>>>>>>>>> have a team ID (which I think is really needed here, glad to
>>> see
>>> >>>>> it
>>> >>>>>>> back)
>>> >>>>>>>>>> and it will be used for auth management, configuration
>>> >>>>> specification,
>>> >>>>>>> etc
>>> >>>>>>>>>> but will be carried by Bundle instead of the dag model. Which
>>> as
>>> >>>>> you
>>> >>>>>>> say
>>> >>>>>>>>>> “For that we will need to make sure that both api-server,
>>> >>>>> scheduler
>>> >>>>>> and
>>> >>>>>>>>>> triggerer have access to the "bundle definition" (to perform
>>> the
>>> >>>>>>> mapping)"
>>> >>>>>>>>>> which honestly doesn’t feel too much different from the
>>> original
>>> >>>>>>> proposal
>>> >>>>>>>>>> we had last week of adding it to Dag table and ensuring it’s
>>> >>>>>> available
>>> >>>>>>>>>> everywhere. but either way I’m happy to meet in the middle and
>>> >>>>> keep
>>> >>>>>> it
>>> >>>>>>> on
>>> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable location.
>>> >>>>>>>>>>
>>> >>>>>>>>> I think the big difference is the "ripple effect" that was
>>> >>>>> discussed
>>> >>>>>> in
>>> >>>>>>>>>
>>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>>> >>>>>> (and I
>>> >>>>>>>>> believe - correct me if I am wrong Ash - important trigger for
>>> the
>>> >>>>>>>>> discussion) so far what we wanted is to extend the primary key
>>> and
>>> >>>>> it
>>> >>>>>>> would
>>> >>>>>>>>> ripple through all the pieces of Airflow -> models, API, UI
>>> etc.
>>> >>>>> ...
>>> >>>>>>>>> However - we already have `bundle_name" and "bundle_version"
>>> in the
>>> >>>>>> Dag
>>> >>>>>>>>> model. So I think when we add a separate table where we map the
>>> >>>>> bundle
>>> >>>>>>> to
>>> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not want
>>> to
>>> >>>>>> change
>>> >>>>>>>>> primary key, we do not want to change UI in any way (except
>>> >>>>> filtering
>>> >>>>>> of
>>> >>>>>>>>> DAGs available based on your team - but that will be handled in
>>> >>>>> Auth
>>> >>>>>>>>> Manager and will not impact UI in any way, I think that's a
>>> huge
>>> >>>>>>>>> simplification of the implementation, and if we agree to it - i
>>> >>>>> think
>>> >>>>>> it
>>> >>>>>>>>> should speed up the implementation significantly. There are
>>> only a
>>> >>>>>>> limited
>>> >>>>>>>>> number of times where you need to look up the team_id - so
>>> having
>>> >>>>> the
>>> >>>>>>>>> bundle -> team mapping in a separate table and having to look
>>> them
>>> >>>>> up
>>> >>>>>>>>> should not be a problem. And it has much less complexity and
>>> >>>>>>>>> "ripple-effect" through the codebase (for example I could
>>> imagine
>>> >>>>> 100s
>>> >>>>>>> or
>>> >>>>>>>>> thousands already written tests that would have to be adapted
>>> if we
>>> >>>>>>> changed
>>> >>>>>>>>> the primary key - where there will be pretty much zero impact
>>> on
>>> >>>>>>> existing
>>> >>>>>>>>> tests if we just add bundle -> team lookup table.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>> One other thing I’d point out is that I think including
>>> executors
>>> >>>>> per
>>> >>>>>>>>>> team is a very easy win and quite possible without much work.
>>> I
>>> >>>>>> already
>>> >>>>>>>>>> have much of the code written. Executors are already aware of
>>> >>>>> Teams
>>> >>>>>>> that
>>> >>>>>>>>>> own them (merged), I have a PR open to have configuration per
>>> team
>>> >>>>>>> (with a
>>> >>>>>>>>>> quite simple and isolated approach, I believe you approved
>>> Jarek).
>>> >>>>>> The
>>> >>>>>>> last
>>> >>>>>>>>>> piece is updating the scheduling logic to route tasks from a
>>> >>>>>> particular
>>> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much work
>>> >>>>> (though
>>> >>>>>> it
>>> >>>>>>>>>> would be easier if the Task models had a column for the team
>>> they
>>> >>>>>>> belong
>>> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to get
>>> the
>>> >>>>>> team) I
>>> >>>>>>>>>> have a branch where I was experimenting with this logic
>>> already.
>>> >>>>>>>>>> Any who, long story short, I don’t think we necessarily need
>>> to
>>> >>>>>> remove
>>> >>>>>>>>>> this piece from the project's scope if it is already partly
>>> done
>>> >>>>> and
>>> >>>>>>> not
>>> >>>>>>>>>> too difficult.
>>> >>>>>>>>>>
>>> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want to just
>>> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we have
>>> it
>>> >>>>> in.
>>> >>>>>>> And
>>> >>>>>>>>> if we can make it in, easily (which it seems we can) - we can
>>> also
>>> >>>>>>> include
>>> >>>>>>>>> it in the first iteration. What I wanted to avoid really (from
>>> the
>>> >>>>>>> original
>>> >>>>>>>>> design) - again trying to simplify it, limit the changes, and
>>> >>>>> speed up
>>> >>>>>>>>> implementation. And there is one "complexity" that I wanted to
>>> >>>>> avoid
>>> >>>>>>>>> specifically - having to have separate , additional
>>> configuration
>>> >>>>> per
>>> >>>>>>> team.
>>> >>>>>>>>> Not only because it complicates already complex configuration
>>> >>>>> handling
>>> >>>>>>> (I
>>> >>>>>>>>> know we have PR for that) but mostly because if it is not
>>> needed,
>>> >>>>> we
>>> >>>>>> can
>>> >>>>>>>>> simplify documentation and explain to our users easier what
>>> they
>>> >>>>> need
>>> >>>>>>> to do
>>> >>>>>>>>> to have their own multi-team setup. And I am quite open to
>>> keeping
>>> >>>>>>>>> multiple-executors if we can avoid complicating configuration.
>>> >>>>>>>>>
>>> >>>>>>>>> But I think some details of that and whether we really need
>>> >>>>> separate
>>> >>>>>>>>> configuration might also come as a result of updating the AIP
>>> - I
>>> >>>>> am
>>> >>>>>> not
>>> >>>>>>>>> quite sure now if we need it, but we can discuss it when we
>>> >>>>> iterate on
>>> >>>>>>> the
>>> >>>>>>>>> AIP.
>>> >>>>>>>>>
>>> >>>>>>>>> J.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> ---------------------------------------------------------------------
>>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> >> For additional commands, e-mail: dev-help@airflow.apache.org
>>>
>>>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

Any last comments ? There is a long weekend coming up in the US, so I will
likely start voting on the updated AIP on Monday 7th.

On Fri, Jun 27, 2025 at 12:41 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> I'd really love to finalise discussion and put it up to a vote some time
> after the recording from the last dev call is posted - so that more
> context, details and the LONG discussion we had on it. There is no *huge*
> hurry  - we have strong dependency on Task Isolation and it seems that it
> will still take a bit of time to complete, so I'd say I would love to start
> voting in about a week time - so that maybe at the next dev call we can
> "seal" the subject. Happy to see any more comments - especially from those
> who have opinions but they had no opportunity to express them.
>
> I am personally very happy with the direction it took - simplification and
> "MVP" kind of approach - also I invite the stakeholders of ours to take a
> close look at the scope and what we really propose - I have a feeling that
> we can balance it out - there is something we can make to make it not
> "worse" for the offerings they have. I think we have a really good
> symbiotic relationship here, and I would love to leverage that. For one -
> my goal here is to have a minimum number of changes that are impacting
> maintainability of the open-source airflow - but mostly "opening up some
> possibilities" - rather than provide turn-key solutions. And mostly because
> this is good for all sides - less maintenance and complexity for OSS
> maintainers, but more opportunities to make it into "turn-key" solutions by
> the stakeholders, while also allowing the "on-prem" users - if they are
> highly motivated - to use those features by adding the "turn-key" layer on
> their own. Also adding multi-team should not be at the expense of "simple"
> installations - they should be virtually unaffected.
>
> One example of applying this is cutting on "separate config files". I
> think it moves us closer to a "turn-key" solution but it is not really
> necessary to achieve the three goals above - that's why in the current
> proposal this part is completely removed - Sorry Niko, but I still think
> it's one of the things that falls into this bucket. We can easily remove
> it, they complicate code, documentation and options the users have, and
> even if it is a "little" more complex to manage configuration by motivated
> users, it's also an opportunity for "turn-key" option that stakeholders can
> build in their products - and we do not have to maintain it in the
> open-source. So I would be rather strong on **not** touching the current
> configuration and simply adding configuration for per-team executors in
> executor config - even if it is uglier and more "low-level".
>
> So if there are some constructive ideas on what can be done to make it
> "simpler" and less "turn-key" in that respect - I would highly value such
> ideas and comments. If we can cut down something more that is not
> "necessary" for the three primary goals I came up with - I am more than
> happy to do it.
>
> Just to remind - those are the "extracted" goals. I slightly updated them
> and added to the preamble of the AIP:
>
> * less operational overhead for managing multi-team (once AIP-72 is
> complete) where separate execution environments are important
> * virtual assets sharing between teams
> * ability of having "admin" and "team sharing" capability where dags from
> multiple teams can be seen in a single Airflow UI  (requires custom RBAC an
> AIP-56 implementation of Auth Manager - with KeyCloak Auth Manager being a
> reference implementation)
>
> J.
>
>
> On Thu, Jun 26, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>>
>>> One technical observation: Now that the dag table no longer has a
>>> team_id in it, what would the behaviour be when a DAG is attempted to move
>>> between bundles? How do we detect this? (I’m not all convinced that we
>>> correctly detect duplicate dag ids across bundles today, so I wouldn’t
>>> assume or rely on the current behaviour.)
>>>
>>
>> Of course - yes, I realise that - that problem was also not handled in
>> the previous iteration to be honest. That is something that dag bundle
>> solution allows to solve eventually - but I do not think it's a blocker for
>> the proposed implementation. We will have to eventually add some way of
>> blocking dags to jump between bundles, we might tackle this separately. I
>> already wanted to propose a separate update to that - but I did not want to
>> complicate the current proposal. One thing at a time. I can, however - if
>> you consider that as a blocker, extend the current AIP with it. Not a big
>> problem. This is however a bit independent from the team_id introduction.
>>
>> Overall, I am still unconvinced this proposal has enough real user
>>> benefit over actually separate deployments, and on balance of the added
>>> complexity and maintenance burden I do not think it is worth it.
>>>
>>
>> That makes me sad, I thought that over the course of the discussion I
>> addressed all the concerns (in this case the concern was "is it worth with
>> the cost and little benefit", but when I did it and heavily limited the
>> impact, now the concern is "is it worth at all as changes are really
>> minimal" - and surely, anyone can change and adapt their concerns, over
>> time,  but that one seems like ever-moving target. I hoped at least for
>> some acknowledgment of some concerns (complexity in this case) is
>> addressed, but it seems that you are deeply convinced that we do not need
>> multi-team at all (which is in stark contrast with at least a dozen of
>> bigger and smaller users of Airflow who submitted talks to Airflow summit
>> (including about 5 or 6 submissions for Airflow 2025) on how they spent
>> their engineering effort, time and money on trying to achieve something
>> similar - they assessed that it's worth, you  assess that it's not worth.
>> Somehow I trust our users that they were not spending the money, time and
>> engineering effort to achieve this because they wanted to spend more money.
>> I think they assessed it's worth it. So I want to make it a bit easier and
>> more "proper" way for them to do that.
>>
>>>
>>> Upgrades: it is not easier to upgrade under this multi team proposal,
>>> but much much harder. This is based on hard earned experience from helping
>>> Astronomer users — having to coordinate upgrades between multiple teams
>>> turns in to a months long slog of the hardest kind of work —  people work:
>>> getting other teams to agree to do things that they don’t directly care
>>> about — “It’s working for me, I don’t care about upgrading, we’ll get to it
>>> next quarter” is a refrain I’ve heard many times.
>>>
>>
>> Yes. absolutely - this is why we deferred it until we knew what shape
>> task isolation and other AIPs we depend on take on. Because it is clear
>> that pretty much all the problem you explain above are going to be solved
>> with task isolation. And it's not just my opinion. If you want to argue
>> with it, you likely need to argue with yourself:
>> https://github.com/apache/airflow/issues/51545#issuecomment-2980038478.
>> Let me quote what you wrote there last week:
>>
>> Ash Berlin Taylor wrote:
>>
>> > A tight coupling between task-sdk and any "server side" component is
>> the opposite to one of the goals of AIP-72 (I'm not sure we ever explicitly
>> said this, but the first point of motivation for the AIP says "Dependency
>> conflicts for administrators supporting data teams using different versions
>> of providers, libraries, or python packages")
>> > In short, my goal with TaskSDK, and the reason for introducing CalVer
>> and Cadwyn with the execution API is to end up in a world where you can
>> upgrade the Airflow Scheduler/API server interdependently of any worker
>> nodes (with the exception that the server must be at least as new as the
>> clients)
>> > This ability to have version-skew is pretty much non-negotiable to me
>> and is (other than other languages) one of primary benefits of AIP-72
>>
>> If you read yourself from that quote it basically means "it will be easy
>> to upgrade airflow independently of workers". So I am a bit confused here.
>> Yes, I agree it was difficult, but you yourself explain that when AIP-72
>> (which since API-67 has been accepted has always beem prerequisite of it)
>> wrote it will be "easy". So I am not sure why you are bringing it now. We
>> assume AIP-72 will be completed and this problem will be gone. Let's not
>> mention it any more please.
>>
>> The true separation from TaskSDK will likely only land in about 3.2 time
>>> frame. We are actively working on it, but it’s a slow process of untangling
>>> lots of assumptions made in the code base over the years. Maybe once we
>>> have that my view would be different, but right now I think this makes the
>>> proposal a non-starter. Especially as you are saying that most teams will
>>> have unique connections. If they’ve got those already, then having an asset
>>> trigger use those conns to watch/poll for activity is a much easier
>>> solution to operate and crucially, to scale and upgrade.
>>>
>>
>> Yes. I perfectly understand that and I am fully aware of potentially 3.2
>> time-frame. And that's fine. Actually I heartily invite you to listen to
>> the part of my talk from Berlin Buzzwords when I was asked for the timeline
>> - https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the exact
>> timeline in my talk . My answer was basically - "3.1" or "3.2", and I
>> sincerely hope "3.1" but we might not be able to complete it because we
>> have other things to do (other - is indeed the Task Isolation work that you
>> are leading). And that's perfectly fine. And it absolutely does not prevent
>> us from voting on the AIP now - similarly as we voted on the previous
>> version of the AIP - knowing that it has some prerequisites a few months
>> ago. Especially that we know that the feature we need from task isolation
>> is "non-negotiable". I.e. it WILL happen. We don't hope for it, we know it
>> will be there. Those are your own words.
>>
>>
>>> >  I think we can’t compare AIP-82 to sharing virtual assets due to
>>> complexity of it.
>>>
>>> Virtual Assets was a mistake, and not how users actually want to use
>>> them. Mea culpa
>>>
>>
>> This is the first time I hear this - certainly you never raised this
>> concern on the devlist. So if you have some concerns about virtual assets I
>> think you should raise it on the devlist, because I think everyone here is
>> missing some conversation (or maybe it's just your private opinion that
>> you never shared with anyone, but maybe it's worth). I would be
>> interested to hear how the feature that was absolutely most successful
>> feature of airflow 2 was a mistake. According to the 2024 survey
>> https://airflow.apache.org/blog/airflow-survey-2024/  - 48% of Airflow
>> users have been using it, even if it was added as one of the last
>> big features of Airflow 2. It's the MOST used feature out of all the
>> features out there. I would be really curious to see how it was a mistake
>> (but  please start a separate thread explaining why you think it was a
>> mistake, what are your data points and what do you think should be fixed.
>> Just dropping "virtual assets were a mistake" in the middle of multi-team
>> conversation seems completely unjustified without knowing what you are
>> talking about. So I think, until we know more, this argument has no base.
>>
>>
>>>
>>> S
>>> To restate my points:
>>>
>>> - Sharing a deployment between teams today/in 3.1 is operationally more
>>> complex (both scaling, and upgrades) — this is a con, not a plus.
>>>
>>
>> Surely. But it will be easier when AIP-72 is complete (which I am
>> definitely looking forward to and as clearly explained in AIP-82, is a
>> prerequisite of it). Nothing changed here.
>>
>>
>>> - The main user benefit appears to be “allow teams’ DAGs to communicate
>>> via Assets”, in which case we can do that today by putting more work in to
>>> AIP-82’s Asset triggers
>>>
>>
>> No. Lower operational complexity for multi-teams (providing that we
>> deliver AIP-72) is another benefit. Virtual assets is another, and since
>> there is no ground in "virtual assets is a mistake" statement (not until
>> you explain what you mean by that in a separate discussion) - this is also
>> still a very valid point.
>>
>>
>>> Soon, we will have then be asked about cross-team governance, policy
>>> enforcement, and potentially unbounded edge cases (e.g., team-specific
>>> secrets, roles, quotas). ain, you get this for free with truely separate
>>> deployments already
>>> allow different teams to use different executors (including multiple
>>> executors per-team following AIP-61)
>>>
>>
>> Not really. We very explicitly say in the AIP that his is not a goal and
>> that we have no plans for. And yes, using separate executors per team is
>> actually back in the AIP-82 in case you did not notice (and the code needed
>> for it's even implemented and merged already in main by Vincent).
>>
>>
>>> Provably not true right now, and until ~3.2 delivers the full Task
>>> SDK/Core dependency separation this would be _more_ work to upgrade, not
>>> less, and that work is not shared but still on a central team.
>>>
>>
>> Absolutely - we will wait for AIP-72 completion. I do not want to say 3.1
>> or 3.2 directly - because there are - as you said - a lot of moving pieces.
>> So my target for multi-team is "After AIP-72 is completed". Full stop. But
>> there is nothing wrong with accepting the AIP now and doing preparatory
>> work in parallel. Similarly as there is no way to have a baby in 1 month by
>> 9 women, there is no way adding more effort to task-sdk isolation will
>> speed it up - we alredy have not only 3 people (you leading it, Kaxil and
>> Amog) but also all the help from me and even 10s of different contributors
>> (for example with the recent db_test cleanup that I took leadership on) -
>> and there are people who wish to work on adding multi-team features. Since
>> the design heavily limits impact on airflow codebase and interactions with
>> task-sdk implementation, there is nothing wrong with starting
>> implementation in parallel either- amazon team is keen to move it forward -
>> they even already implemented SQS trigger for assets, and we are working
>> together on FAB removal, Keycloak authentication manager - and they seem to
>> still have capacity and drive to progress multi-team. So I am not sure if
>> we are trading off something. There is no "if we work on more on task sdk
>> and drop multi-team things will be faster". Generally in open source people
>> work in the area where they feel they can provide best value - such as you
>> working on task-sdk, me on CI,dev env, they will deliver more value on
>> multi-team
>>
>>
>>>
>>> So please, as succinctly as possible, please tell me what the direct
>>> benefit to users this proposal is over us putting this effort in to writing
>>> better Asset triggers instead?
>>>
>>
>>
>> * less operational overhead for managing multi-team (once AIP-72 is
>> complete) where separate execution environments are important
>> * virtual assets sharing
>> * ability of having "admin" and "team sharing" capability where dags from
>> multiple teams can be seen in a single Airflow UI  (requires custom RBAC)
>>
>> None of this can be done via beter asset triggers
>>
>>
>>>
>>> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >
>>> > My counter-points:
>>> >
>>> >
>>> >> 1. Managing a multi team deployment is not materially different from
>>> >> managing a deployment per team
>>> >>
>>> >
>>> > It's a bit easier - especially when it comes to upgrades (especially
>>> in the
>>> > case we are targetting when we are not targetting multi-tenant, but
>>> several
>>> > relatively closely cooperating teams with different dependncy
>>> requiremens
>>> > and isolation need.
>>> >
>>> > 2. The database changes were quite wide-reaching
>>> >>
>>> >
>>> > Yes. that is addressed.
>>> >
>>> >
>>> >> 3. I don’t believe the original AIP (again, I haven’t read the updated
>>> >> proposal or recent messages on the thread. yet) will meet what many
>>> users
>>> >> want out of a multiteam solution
>>> >>
>>> >
>>> > I think we will only see when we try. A lot of people thing they would,
>>> > even if they are warned. I know at least one user (Wealthsimple) who
>>> > definitely want to use it and they got a very detailed explanation of
>>> the
>>> > idea and understand it well. So I am sure that **some** users would.
>>> But we
>>> > do not know how many.
>>> >
>>> >
>>> >> To expand on those points a bit more
>>> >>
>>> >> On 1. The only components that are shared are, I think, the scheduler
>>> and
>>> >> the API server, and it’s arguable if that is actually a good idea
>>> given
>>> >> those are likely to be the most performance sensitive components
>>> anyway.
>>> >>
>>> >> Additionally the fact that the scheduler is a shared component makes
>>> >> upgrading it almost a non starter as you would likely need buy-in,
>>> changes,
>>> >> and testing form ALL teams using it. I’d argue that this is a huge
>>> negative
>>> >> until we finish off the version indepence work of AIP-72.
>>> >>
>>> >
>>> > Quite disagree here - especially that our target is that task-sdk is
>>> > supposed to provide all isolation that is needed. There should be 0
>>> changes
>>> > in the dags needed to upgrade scheduler, api_server, triggerer -
>>> precisely
>>> > because we introduced backwards-compatible task-sdk.
>>> >
>>> > On 3 my complaint is essentially that this doesn’t go nearly far
>>> enough. It
>>> >> doesn’t allow read only views to other teams dags. I don’t think it
>>> allows
>>> >> you to be in multiple teams at once. You can’t share a connection
>>> between
>>> >> teams but only allow certain specified dags to access it, but would
>>> have to
>>> >> either be globally usable, or duplicated-and-kept-in-sync between
>>> teams. In
>>> >> short I think it fall short of being useful..
>>> >>
>>> >
>>> > Oh absolutely all that is possible (except sharing single connections
>>> > between multiple teams - which is a very niche use cases and
>>> duplication
>>> > here is perfectly ok as first approximation - and if we need more we
>>> can
>>> > add it later).
>>> >
>>> > Auth manager RBAC and access is abstracted away, and the Keyclock
>>> Manager
>>> > implemented by Vincent allows to manage completely independent and
>>> separate
>>> > RBAC based on arguments and resources provided by Airflow. There is
>>> nothing
>>> > to prevent the user who configures KeyCloak RBAC to define it in the
>>> way:
>>> >
>>> > if group a > allow to read a and write b
>>> > if group b > alllow to write b but not a
>>> >
>>> > and any other combinations. KeyCloak implementation - pretty advanced
>>> > already - (and design of auth manager) completely abstracts away both
>>> > authentication and authorization to KeyCloak and KeyCloak has RBAC
>>> > management built in. Also any of the users can write their own - even
>>> > hard-coded authentication manager to do the same if they do not want to
>>> > have configurable KeyCloak. Even SimpleAuthManager could be hard-coded
>>> to
>>> > provide thiose features.
>>> >
>>> >
>>> >>
>>> >> So on the surface, I’m no more in favour of using dag bundle as a
>>> >> replacement for team id as I think most of the above points still
>>> stand.
>>> >>
>>> >
>>> > We disagree here.
>>> >
>>> >>
>>> >> My counter proposal: We do _nothing_ to core airflow. We work on
>>> improving
>>> >> the event-based trigger o fdags (write more triggers for read/check
>>> remote
>>> >> Assets etc) so that teams can have 100% isolated deployments but still
>>> >> trigger dags based on asset events from other teams.
>>> >>
>>> >
>>> > That does not solve any of the other design goals - only allows to
>>> trigger
>>> > assets a bit more easily (but also it's not entirely solved by AIP-82
>>> > because it does not solve virtual assets - only ones that have defined
>>> > triggerer and "something" to listen on - which is way more complex than
>>> > just defining asset in a Dag and using it in another). I think we can't
>>> > compare AIP-82 to sharing virtual assets due to complexity of it. I
>>> > explained it in the doc.
>>> >
>>> >
>>> > I will now go and catch up with the long thread and updated proposal
>>> and
>>> >> come back.
>>> >>
>>> >
>>> > Please. I hope the above explaination will help in better
>>> understanding of
>>> > the proposal, because I think you had some assumptions that do not
>>> hold any
>>> > more with the new proposal.
>>> >
>>> > J.
>>> >
>>> >
>>> >>
>>> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >>>
>>> >>> Just to clarify the relation - I updated the AIP now to refer to
>>> AIP-82
>>> >> and
>>> >>> to explain relation between the "cross-team" and "cross-airflow"
>>> asset
>>> >>> triggering - this is what I added:
>>> >>>
>>> >>> Note that there is a relation between AIP-82 ("External Driven
>>> >> Scheduling")
>>> >>> and this part of the functionality. When you have multiple instances
>>> of
>>> >>> Airflow, you can use shared datasets - "Physical datasets" - that
>>> several
>>> >>> Airflow Instances can use - for example there could be an S3 object
>>> that
>>> >> is
>>> >>> produced by one airflow instance, and consumed by another. That
>>> requires
>>> >>> deferred trigger to monitor for such datasets, and appropriate
>>> >> permissions
>>> >>> to the external dataset, and you could achive similar result to
>>> >> cross-team
>>> >>> dataset triggering (but cross airflow). However the feature of
>>> sharing
>>> >>> datasets between the teams also works for virtual assets, that do not
>>> >> have
>>> >>> physically shared "objects" and trigger that is monitoring for
>>> changes in
>>> >>> such asset.
>>> >>>
>>> >>> J.
>>> >>>
>>> >>>
>>> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> >>>
>>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>>> reference
>>> >> to
>>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>>> through.
>>> >>>>
>>> >>>> Yep. It did not - because I did not think it was needed or even very
>>> >>>> important after the simplifications. AIP-82 has a different scope,
>>> >> really.
>>> >>>> It only helps when the Assets are "real" data files which we have
>>> >> physical
>>> >>>> triggers for, it's slightly related - sharing datasets between teams
>>> >>>> (including those that do not require physical files and triggers) is
>>> >> still
>>> >>>> possible in the design we have now, but it's not (and never was) the
>>> >>>> **only** reason for having multi-team. There always was (and still
>>> is)
>>> >> the
>>> >>>> possibility of having a common, distinct environments (i.e.
>>> dependencies
>>> >>>> and providers) per team, the possibility of having connections and
>>> >>>> variables that are only accessible to one team and not the other,
>>> and
>>> >>>> isolating workload execution (all that while allowing to manage
>>> multiple
>>> >>>> team and schedule things with single deployment). That did not
>>> change.
>>> >> What
>>> >>>> changed a lot is that it is now way simpler, something that we can
>>> >>>> implement without heavy changes to the codebase - and give it to our
>>> >> users,
>>> >>>> so that they can assess if this is something they need without too
>>> much
>>> >>>> risk and effort.
>>> >>>>
>>> >>>> This was - I believe the main concern, that the value we get from
>>> it is
>>> >>>> not dramatic, but the required changes are huge. This "redesign"
>>> changes
>>> >>>> the equation - the value is still unchanged, but the cost of
>>> >> implementing
>>> >>>> it and impact on the Airflow codebase is much smaller. I still have
>>> not
>>> >>>> heard back from Ash if my proposal responds to his original concern
>>> >> though,
>>> >>>> so I am mostly guessing (also based on the positive impact of
>>> others)
>>> >> that
>>> >>>> yes it does. But to be honest I am not sure and I would love to hear
>>> >> back,
>>> >>>> I decided to update the AIP to reflect it - regardless, because I
>>> think
>>> >> the
>>> >>>> simplification I proposed keeps the original goals, but is indeed
>>> way
>>> >>>> simpler.
>>> >>>>
>>> >>>>> This is a very difficult thread to catch up on.
>>> >>>>
>>> >>>> Valid point. Let me summarize what is the result:
>>> >>>>
>>> >>>> * I significantly simplified the implementation proposal comparing
>>> to
>>> >> the
>>> >>>> original version
>>> >>>> * main simplification is very limited impact on existing database -
>>> >>>> without "ripple effect" that would require us to change a lot of
>>> tables,
>>> >>>> including their primary keys, and heavily impact the UI
>>> >>>> * this is now more of an incremental change that can be implemented
>>> way
>>> >>>> faster and with far less risk
>>> >>>> * updated idea is based on leveraging bundles (already part of our
>>> data
>>> >>>> model) to map them (many-to-one) to a team - which requires to just
>>> >> extend
>>> >>>> the data model with bundle mapping and add team_id to connections
>>> and
>>> >>>> variables. Those are all needed DB changes.
>>> >>>>
>>> >>>> The AIP is updated - in a one single big change so It should be
>>> easy to
>>> >>>> compare the changes:
>>> >>>>
>>> >>
>>> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
>>> >>>> -> I even named the version appropriately "Simplified multi-team
>>> AIP" -
>>> >> you
>>> >>>> can select and compare v.65 with v.66 to see the exact differences I
>>> >>>> proposed.
>>> >>>>
>>> >>>> I hope it will be helpful to catch up and for those who did not
>>> follow,
>>> >> to
>>> >>>> be able to make up their minds about it.
>>> >>>>
>>> >>>> J.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
>>> >> <vi...@astronomer.io.invalid>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> This is a very difficult thread to catch up on.
>>> >>>>> I will take a detailed look at the AIP update to try to figure out
>>> the
>>> >>>>> changes in the proposal.
>>> >>>>>
>>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>>> reference
>>> >> to
>>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>>> through.
>>> >>>>>
>>> >>>>> Vikram
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>>> >>>>> gopidesupavan@gmail.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's much
>>> more
>>> >> slim
>>> >>>>>> down.
>>> >>>>>>
>>> >>>>>> left a minor comment. :) Overall looking great.
>>> >>>>>>
>>> >>>>>> Pavan
>>> >>>>>>
>>> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>>> >>>>> <j_scheffler@gmx.de.invalid
>>> >>>>>>>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> Thanks for the rework/update of the AIP-72!
>>> >>>>>>>
>>> >>>>>>> Just a few small comments but overall I like it as it is much
>>> leaner
>>> >>>>>>> than originally planned and is in a level of complexity that it
>>> >> really
>>> >>>>>>> seems to be a benefit to close the gap as described.
>>> >>>>>>>
>>> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
>>> >>>>>>>> I updated the AIP - including architecture images and reviewed
>>> it
>>> >>>>>> (again)
>>> >>>>>>>> and corrected any ambiguities and places where it needed to be
>>> >>>>> changed.
>>> >>>>>>>>
>>> >>>>>>>> I think the current state
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>>> >>>>>>>> - nicely describes the proposal.
>>> >>>>>>>>
>>> >>>>>>>> Comparing to the previous one:
>>> >>>>>>>>
>>> >>>>>>>> 1. The DB changes are far less intrusive - no ripple effect on
>>> >>>>> Airflow
>>> >>>>>>>> 2. There is no need to merge configurations and provide
>>> different
>>> >>>>> set
>>> >>>>>> of
>>> >>>>>>>> configs per team - we can add it later but I do not see why we
>>> need
>>> >>>>> it
>>> >>>>>> in
>>> >>>>>>>> this simplified version
>>> >>>>>>>> 3. We can still configure a different set of executors per team
>>> -
>>> >>>>> that
>>> >>>>>> is
>>> >>>>>>>> already implemented (we just need to wire it to the bundle ->
>>> team
>>> >>>>>>> mapping).
>>> >>>>>>>>
>>> >>>>>>>> I think it will be way simpler and faster to implement this way
>>> and
>>> >>>>> it
>>> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can
>>> give
>>> >>>>> our
>>> >>>>>>>> users so that they can provide feedback.
>>> >>>>>>>>
>>> >>>>>>>> J.
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
>>> >>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>> I like this iteration a bit more now for sure, thanks for
>>> being
>>> >>>>>>> receptive
>>> >>>>>>>>>> to feedback! :)
>>> >>>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>> This now becomes quite close to what was proposing before, we
>>> now
>>> >>>>>> again
>>> >>>>>>>>>> have a team ID (which I think is really needed here, glad to
>>> see
>>> >>>>> it
>>> >>>>>>> back)
>>> >>>>>>>>>> and it will be used for auth management, configuration
>>> >>>>> specification,
>>> >>>>>>> etc
>>> >>>>>>>>>> but will be carried by Bundle instead of the dag model. Which
>>> as
>>> >>>>> you
>>> >>>>>>> say
>>> >>>>>>>>>> “For that we will need to make sure that both api-server,
>>> >>>>> scheduler
>>> >>>>>> and
>>> >>>>>>>>>> triggerer have access to the "bundle definition" (to perform
>>> the
>>> >>>>>>> mapping)"
>>> >>>>>>>>>> which honestly doesn’t feel too much different from the
>>> original
>>> >>>>>>> proposal
>>> >>>>>>>>>> we had last week of adding it to Dag table and ensuring it’s
>>> >>>>>> available
>>> >>>>>>>>>> everywhere. but either way I’m happy to meet in the middle and
>>> >>>>> keep
>>> >>>>>> it
>>> >>>>>>> on
>>> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable location.
>>> >>>>>>>>>>
>>> >>>>>>>>> I think the big difference is the "ripple effect" that was
>>> >>>>> discussed
>>> >>>>>> in
>>> >>>>>>>>>
>>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>>> >>>>>> (and I
>>> >>>>>>>>> believe - correct me if I am wrong Ash - important trigger for
>>> the
>>> >>>>>>>>> discussion) so far what we wanted is to extend the primary key
>>> and
>>> >>>>> it
>>> >>>>>>> would
>>> >>>>>>>>> ripple through all the pieces of Airflow -> models, API, UI
>>> etc.
>>> >>>>> ...
>>> >>>>>>>>> However - we already have `bundle_name" and "bundle_version"
>>> in the
>>> >>>>>> Dag
>>> >>>>>>>>> model. So I think when we add a separate table where we map the
>>> >>>>> bundle
>>> >>>>>>> to
>>> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not want
>>> to
>>> >>>>>> change
>>> >>>>>>>>> primary key, we do not want to change UI in any way (except
>>> >>>>> filtering
>>> >>>>>> of
>>> >>>>>>>>> DAGs available based on your team - but that will be handled in
>>> >>>>> Auth
>>> >>>>>>>>> Manager and will not impact UI in any way, I think that's a
>>> huge
>>> >>>>>>>>> simplification of the implementation, and if we agree to it - i
>>> >>>>> think
>>> >>>>>> it
>>> >>>>>>>>> should speed up the implementation significantly. There are
>>> only a
>>> >>>>>>> limited
>>> >>>>>>>>> number of times where you need to look up the team_id - so
>>> having
>>> >>>>> the
>>> >>>>>>>>> bundle -> team mapping in a separate table and having to look
>>> them
>>> >>>>> up
>>> >>>>>>>>> should not be a problem. And it has much less complexity and
>>> >>>>>>>>> "ripple-effect" through the codebase (for example I could
>>> imagine
>>> >>>>> 100s
>>> >>>>>>> or
>>> >>>>>>>>> thousands already written tests that would have to be adapted
>>> if we
>>> >>>>>>> changed
>>> >>>>>>>>> the primary key - where there will be pretty much zero impact
>>> on
>>> >>>>>>> existing
>>> >>>>>>>>> tests if we just add bundle -> team lookup table.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>> One other thing I’d point out is that I think including
>>> executors
>>> >>>>> per
>>> >>>>>>>>>> team is a very easy win and quite possible without much work.
>>> I
>>> >>>>>> already
>>> >>>>>>>>>> have much of the code written. Executors are already aware of
>>> >>>>> Teams
>>> >>>>>>> that
>>> >>>>>>>>>> own them (merged), I have a PR open to have configuration per
>>> team
>>> >>>>>>> (with a
>>> >>>>>>>>>> quite simple and isolated approach, I believe you approved
>>> Jarek).
>>> >>>>>> The
>>> >>>>>>> last
>>> >>>>>>>>>> piece is updating the scheduling logic to route tasks from a
>>> >>>>>> particular
>>> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much work
>>> >>>>> (though
>>> >>>>>> it
>>> >>>>>>>>>> would be easier if the Task models had a column for the team
>>> they
>>> >>>>>>> belong
>>> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to get
>>> the
>>> >>>>>> team) I
>>> >>>>>>>>>> have a branch where I was experimenting with this logic
>>> already.
>>> >>>>>>>>>> Any who, long story short, I don’t think we necessarily need
>>> to
>>> >>>>>> remove
>>> >>>>>>>>>> this piece from the project's scope if it is already partly
>>> done
>>> >>>>> and
>>> >>>>>>> not
>>> >>>>>>>>>> too difficult.
>>> >>>>>>>>>>
>>> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want to just
>>> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we have
>>> it
>>> >>>>> in.
>>> >>>>>>> And
>>> >>>>>>>>> if we can make it in, easily (which it seems we can) - we can
>>> also
>>> >>>>>>> include
>>> >>>>>>>>> it in the first iteration. What I wanted to avoid really (from
>>> the
>>> >>>>>>> original
>>> >>>>>>>>> design) - again trying to simplify it, limit the changes, and
>>> >>>>> speed up
>>> >>>>>>>>> implementation. And there is one "complexity" that I wanted to
>>> >>>>> avoid
>>> >>>>>>>>> specifically - having to have separate , additional
>>> configuration
>>> >>>>> per
>>> >>>>>>> team.
>>> >>>>>>>>> Not only because it complicates already complex configuration
>>> >>>>> handling
>>> >>>>>>> (I
>>> >>>>>>>>> know we have PR for that) but mostly because if it is not
>>> needed,
>>> >>>>> we
>>> >>>>>> can
>>> >>>>>>>>> simplify documentation and explain to our users easier what
>>> they
>>> >>>>> need
>>> >>>>>>> to do
>>> >>>>>>>>> to have their own multi-team setup. And I am quite open to
>>> keeping
>>> >>>>>>>>> multiple-executors if we can avoid complicating configuration.
>>> >>>>>>>>>
>>> >>>>>>>>> But I think some details of that and whether we really need
>>> >>>>> separate
>>> >>>>>>>>> configuration might also come as a result of updating the AIP
>>> - I
>>> >>>>> am
>>> >>>>>> not
>>> >>>>>>>>> quite sure now if we need it, but we can discuss it when we
>>> >>>>> iterate on
>>> >>>>>>> the
>>> >>>>>>>>> AIP.
>>> >>>>>>>>>
>>> >>>>>>>>> J.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> ---------------------------------------------------------------------
>>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>> >> For additional commands, e-mail: dev-help@airflow.apache.org
>>>
>>>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

I'd really love to finalise discussion and put it up to a vote some time
after the recording from the last dev call is posted - so that more
context, details and the LONG discussion we had on it. There is no *huge*
hurry  - we have strong dependency on Task Isolation and it seems that it
will still take a bit of time to complete, so I'd say I would love to start
voting in about a week time - so that maybe at the next dev call we can
"seal" the subject. Happy to see any more comments - especially from those
who have opinions but they had no opportunity to express them.

I am personally very happy with the direction it took - simplification and
"MVP" kind of approach - also I invite the stakeholders of ours to take a
close look at the scope and what we really propose - I have a feeling that
we can balance it out - there is something we can make to make it not
"worse" for the offerings they have. I think we have a really good
symbiotic relationship here, and I would love to leverage that. For one -
my goal here is to have a minimum number of changes that are impacting
maintainability of the open-source airflow - but mostly "opening up some
possibilities" - rather than provide turn-key solutions. And mostly because
this is good for all sides - less maintenance and complexity for OSS
maintainers, but more opportunities to make it into "turn-key" solutions by
the stakeholders, while also allowing the "on-prem" users - if they are
highly motivated - to use those features by adding the "turn-key" layer on
their own. Also adding multi-team should not be at the expense of "simple"
installations - they should be virtually unaffected.

One example of applying this is cutting on "separate config files". I think
it moves us closer to a "turn-key" solution but it is not really necessary
to achieve the three goals above - that's why in the current proposal this
part is completely removed - Sorry Niko, but I still think it's one of the
things that falls into this bucket. We can easily remove it, they
complicate code, documentation and options the users have, and even if it
is a "little" more complex to manage configuration by motivated users, it's
also an opportunity for "turn-key" option that stakeholders can build in
their products - and we do not have to maintain it in the open-source. So I
would be rather strong on **not** touching the current configuration and
simply adding configuration for per-team executors in executor config -
even if it is uglier and more "low-level".

So if there are some constructive ideas on what can be done to make it
"simpler" and less "turn-key" in that respect - I would highly value such
ideas and comments. If we can cut down something more that is not
"necessary" for the three primary goals I came up with - I am more than
happy to do it.

Just to remind - those are the "extracted" goals. I slightly updated them
and added to the preamble of the AIP:

* less operational overhead for managing multi-team (once AIP-72 is
complete) where separate execution environments are important
* virtual assets sharing between teams
* ability of having "admin" and "team sharing" capability where dags from
multiple teams can be seen in a single Airflow UI  (requires custom RBAC an
AIP-56 implementation of Auth Manager - with KeyCloak Auth Manager being a
reference implementation)

J.


On Thu, Jun 26, 2025 at 10:53 AM Jarek Potiuk <ja...@potiuk.com> wrote:

>
>> One technical observation: Now that the dag table no longer has a team_id
>> in it, what would the behaviour be when a DAG is attempted to move between
>> bundles? How do we detect this? (I’m not all convinced that we correctly
>> detect duplicate dag ids across bundles today, so I wouldn’t assume or rely
>> on the current behaviour.)
>>
>
> Of course - yes, I realise that - that problem was also not handled in the
> previous iteration to be honest. That is something that dag bundle solution
> allows to solve eventually - but I do not think it's a blocker for the
> proposed implementation. We will have to eventually add some way of
> blocking dags to jump between bundles, we might tackle this separately. I
> already wanted to propose a separate update to that - but I did not want to
> complicate the current proposal. One thing at a time. I can, however - if
> you consider that as a blocker, extend the current AIP with it. Not a big
> problem. This is however a bit independent from the team_id introduction.
>
> Overall, I am still unconvinced this proposal has enough real user benefit
>> over actually separate deployments, and on balance of the added complexity
>> and maintenance burden I do not think it is worth it.
>>
>
> That makes me sad, I thought that over the course of the discussion I
> addressed all the concerns (in this case the concern was "is it worth with
> the cost and little benefit", but when I did it and heavily limited the
> impact, now the concern is "is it worth at all as changes are really
> minimal" - and surely, anyone can change and adapt their concerns, over
> time,  but that one seems like ever-moving target. I hoped at least for
> some acknowledgment of some concerns (complexity in this case) is
> addressed, but it seems that you are deeply convinced that we do not need
> multi-team at all (which is in stark contrast with at least a dozen of
> bigger and smaller users of Airflow who submitted talks to Airflow summit
> (including about 5 or 6 submissions for Airflow 2025) on how they spent
> their engineering effort, time and money on trying to achieve something
> similar - they assessed that it's worth, you  assess that it's not worth.
> Somehow I trust our users that they were not spending the money, time and
> engineering effort to achieve this because they wanted to spend more money.
> I think they assessed it's worth it. So I want to make it a bit easier and
> more "proper" way for them to do that.
>
>>
>> Upgrades: it is not easier to upgrade under this multi team proposal, but
>> much much harder. This is based on hard earned experience from helping
>> Astronomer users — having to coordinate upgrades between multiple teams
>> turns in to a months long slog of the hardest kind of work —  people work:
>> getting other teams to agree to do things that they don’t directly care
>> about — “It’s working for me, I don’t care about upgrading, we’ll get to it
>> next quarter” is a refrain I’ve heard many times.
>>
>
> Yes. absolutely - this is why we deferred it until we knew what shape task
> isolation and other AIPs we depend on take on. Because it is clear that
> pretty much all the problem you explain above are going to be solved with
> task isolation. And it's not just my opinion. If you want to argue with it,
> you likely need to argue with yourself:
> https://github.com/apache/airflow/issues/51545#issuecomment-2980038478.
> Let me quote what you wrote there last week:
>
> Ash Berlin Taylor wrote:
>
> > A tight coupling between task-sdk and any "server side" component is the
> opposite to one of the goals of AIP-72 (I'm not sure we ever explicitly
> said this, but the first point of motivation for the AIP says "Dependency
> conflicts for administrators supporting data teams using different versions
> of providers, libraries, or python packages")
> > In short, my goal with TaskSDK, and the reason for introducing CalVer
> and Cadwyn with the execution API is to end up in a world where you can
> upgrade the Airflow Scheduler/API server interdependently of any worker
> nodes (with the exception that the server must be at least as new as the
> clients)
> > This ability to have version-skew is pretty much non-negotiable to me
> and is (other than other languages) one of primary benefits of AIP-72
>
> If you read yourself from that quote it basically means "it will be easy
> to upgrade airflow independently of workers". So I am a bit confused here.
> Yes, I agree it was difficult, but you yourself explain that when AIP-72
> (which since API-67 has been accepted has always beem prerequisite of it)
> wrote it will be "easy". So I am not sure why you are bringing it now. We
> assume AIP-72 will be completed and this problem will be gone. Let's not
> mention it any more please.
>
> The true separation from TaskSDK will likely only land in about 3.2 time
>> frame. We are actively working on it, but it’s a slow process of untangling
>> lots of assumptions made in the code base over the years. Maybe once we
>> have that my view would be different, but right now I think this makes the
>> proposal a non-starter. Especially as you are saying that most teams will
>> have unique connections. If they’ve got those already, then having an asset
>> trigger use those conns to watch/poll for activity is a much easier
>> solution to operate and crucially, to scale and upgrade.
>>
>
> Yes. I perfectly understand that and I am fully aware of potentially 3.2
> time-frame. And that's fine. Actually I heartily invite you to listen to
> the part of my talk from Berlin Buzzwords when I was asked for the timeline
> - https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the exact
> timeline in my talk . My answer was basically - "3.1" or "3.2", and I
> sincerely hope "3.1" but we might not be able to complete it because we
> have other things to do (other - is indeed the Task Isolation work that you
> are leading). And that's perfectly fine. And it absolutely does not prevent
> us from voting on the AIP now - similarly as we voted on the previous
> version of the AIP - knowing that it has some prerequisites a few months
> ago. Especially that we know that the feature we need from task isolation
> is "non-negotiable". I.e. it WILL happen. We don't hope for it, we know it
> will be there. Those are your own words.
>
>
>> >  I think we can’t compare AIP-82 to sharing virtual assets due to
>> complexity of it.
>>
>> Virtual Assets was a mistake, and not how users actually want to use
>> them. Mea culpa
>>
>
> This is the first time I hear this - certainly you never raised this
> concern on the devlist. So if you have some concerns about virtual assets I
> think you should raise it on the devlist, because I think everyone here is
> missing some conversation (or maybe it's just your private opinion that
> you never shared with anyone, but maybe it's worth). I would be
> interested to hear how the feature that was absolutely most successful
> feature of airflow 2 was a mistake. According to the 2024 survey
> https://airflow.apache.org/blog/airflow-survey-2024/  - 48% of Airflow
> users have been using it, even if it was added as one of the last
> big features of Airflow 2. It's the MOST used feature out of all the
> features out there. I would be really curious to see how it was a mistake
> (but  please start a separate thread explaining why you think it was a
> mistake, what are your data points and what do you think should be fixed.
> Just dropping "virtual assets were a mistake" in the middle of multi-team
> conversation seems completely unjustified without knowing what you are
> talking about. So I think, until we know more, this argument has no base.
>
>
>>
>> S
>> To restate my points:
>>
>> - Sharing a deployment between teams today/in 3.1 is operationally more
>> complex (both scaling, and upgrades) — this is a con, not a plus.
>>
>
> Surely. But it will be easier when AIP-72 is complete (which I am
> definitely looking forward to and as clearly explained in AIP-82, is a
> prerequisite of it). Nothing changed here.
>
>
>> - The main user benefit appears to be “allow teams’ DAGs to communicate
>> via Assets”, in which case we can do that today by putting more work in to
>> AIP-82’s Asset triggers
>>
>
> No. Lower operational complexity for multi-teams (providing that we
> deliver AIP-72) is another benefit. Virtual assets is another, and since
> there is no ground in "virtual assets is a mistake" statement (not until
> you explain what you mean by that in a separate discussion) - this is also
> still a very valid point.
>
>
>> Soon, we will have then be asked about cross-team governance, policy
>> enforcement, and potentially unbounded edge cases (e.g., team-specific
>> secrets, roles, quotas). ain, you get this for free with truely separate
>> deployments already
>> allow different teams to use different executors (including multiple
>> executors per-team following AIP-61)
>>
>
> Not really. We very explicitly say in the AIP that his is not a goal and
> that we have no plans for. And yes, using separate executors per team is
> actually back in the AIP-82 in case you did not notice (and the code needed
> for it's even implemented and merged already in main by Vincent).
>
>
>> Provably not true right now, and until ~3.2 delivers the full Task
>> SDK/Core dependency separation this would be _more_ work to upgrade, not
>> less, and that work is not shared but still on a central team.
>>
>
> Absolutely - we will wait for AIP-72 completion. I do not want to say 3.1
> or 3.2 directly - because there are - as you said - a lot of moving pieces.
> So my target for multi-team is "After AIP-72 is completed". Full stop. But
> there is nothing wrong with accepting the AIP now and doing preparatory
> work in parallel. Similarly as there is no way to have a baby in 1 month by
> 9 women, there is no way adding more effort to task-sdk isolation will
> speed it up - we alredy have not only 3 people (you leading it, Kaxil and
> Amog) but also all the help from me and even 10s of different contributors
> (for example with the recent db_test cleanup that I took leadership on) -
> and there are people who wish to work on adding multi-team features. Since
> the design heavily limits impact on airflow codebase and interactions with
> task-sdk implementation, there is nothing wrong with starting
> implementation in parallel either- amazon team is keen to move it forward -
> they even already implemented SQS trigger for assets, and we are working
> together on FAB removal, Keycloak authentication manager - and they seem to
> still have capacity and drive to progress multi-team. So I am not sure if
> we are trading off something. There is no "if we work on more on task sdk
> and drop multi-team things will be faster". Generally in open source people
> work in the area where they feel they can provide best value - such as you
> working on task-sdk, me on CI,dev env, they will deliver more value on
> multi-team
>
>
>>
>> So please, as succinctly as possible, please tell me what the direct
>> benefit to users this proposal is over us putting this effort in to writing
>> better Asset triggers instead?
>>
>
>
> * less operational overhead for managing multi-team (once AIP-72 is
> complete) where separate execution environments are important
> * virtual assets sharing
> * ability of having "admin" and "team sharing" capability where dags from
> multiple teams can be seen in a single Airflow UI  (requires custom RBAC)
>
> None of this can be done via beter asset triggers
>
>
>>
>> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > My counter-points:
>> >
>> >
>> >> 1. Managing a multi team deployment is not materially different from
>> >> managing a deployment per team
>> >>
>> >
>> > It's a bit easier - especially when it comes to upgrades (especially in
>> the
>> > case we are targetting when we are not targetting multi-tenant, but
>> several
>> > relatively closely cooperating teams with different dependncy
>> requiremens
>> > and isolation need.
>> >
>> > 2. The database changes were quite wide-reaching
>> >>
>> >
>> > Yes. that is addressed.
>> >
>> >
>> >> 3. I don’t believe the original AIP (again, I haven’t read the updated
>> >> proposal or recent messages on the thread. yet) will meet what many
>> users
>> >> want out of a multiteam solution
>> >>
>> >
>> > I think we will only see when we try. A lot of people thing they would,
>> > even if they are warned. I know at least one user (Wealthsimple) who
>> > definitely want to use it and they got a very detailed explanation of
>> the
>> > idea and understand it well. So I am sure that **some** users would.
>> But we
>> > do not know how many.
>> >
>> >
>> >> To expand on those points a bit more
>> >>
>> >> On 1. The only components that are shared are, I think, the scheduler
>> and
>> >> the API server, and it’s arguable if that is actually a good idea given
>> >> those are likely to be the most performance sensitive components
>> anyway.
>> >>
>> >> Additionally the fact that the scheduler is a shared component makes
>> >> upgrading it almost a non starter as you would likely need buy-in,
>> changes,
>> >> and testing form ALL teams using it. I’d argue that this is a huge
>> negative
>> >> until we finish off the version indepence work of AIP-72.
>> >>
>> >
>> > Quite disagree here - especially that our target is that task-sdk is
>> > supposed to provide all isolation that is needed. There should be 0
>> changes
>> > in the dags needed to upgrade scheduler, api_server, triggerer -
>> precisely
>> > because we introduced backwards-compatible task-sdk.
>> >
>> > On 3 my complaint is essentially that this doesn’t go nearly far
>> enough. It
>> >> doesn’t allow read only views to other teams dags. I don’t think it
>> allows
>> >> you to be in multiple teams at once. You can’t share a connection
>> between
>> >> teams but only allow certain specified dags to access it, but would
>> have to
>> >> either be globally usable, or duplicated-and-kept-in-sync between
>> teams. In
>> >> short I think it fall short of being useful..
>> >>
>> >
>> > Oh absolutely all that is possible (except sharing single connections
>> > between multiple teams - which is a very niche use cases and duplication
>> > here is perfectly ok as first approximation - and if we need more we can
>> > add it later).
>> >
>> > Auth manager RBAC and access is abstracted away, and the Keyclock
>> Manager
>> > implemented by Vincent allows to manage completely independent and
>> separate
>> > RBAC based on arguments and resources provided by Airflow. There is
>> nothing
>> > to prevent the user who configures KeyCloak RBAC to define it in the
>> way:
>> >
>> > if group a > allow to read a and write b
>> > if group b > alllow to write b but not a
>> >
>> > and any other combinations. KeyCloak implementation - pretty advanced
>> > already - (and design of auth manager) completely abstracts away both
>> > authentication and authorization to KeyCloak and KeyCloak has RBAC
>> > management built in. Also any of the users can write their own - even
>> > hard-coded authentication manager to do the same if they do not want to
>> > have configurable KeyCloak. Even SimpleAuthManager could be hard-coded
>> to
>> > provide thiose features.
>> >
>> >
>> >>
>> >> So on the surface, I’m no more in favour of using dag bundle as a
>> >> replacement for team id as I think most of the above points still
>> stand.
>> >>
>> >
>> > We disagree here.
>> >
>> >>
>> >> My counter proposal: We do _nothing_ to core airflow. We work on
>> improving
>> >> the event-based trigger o fdags (write more triggers for read/check
>> remote
>> >> Assets etc) so that teams can have 100% isolated deployments but still
>> >> trigger dags based on asset events from other teams.
>> >>
>> >
>> > That does not solve any of the other design goals - only allows to
>> trigger
>> > assets a bit more easily (but also it's not entirely solved by AIP-82
>> > because it does not solve virtual assets - only ones that have defined
>> > triggerer and "something" to listen on - which is way more complex than
>> > just defining asset in a Dag and using it in another). I think we can't
>> > compare AIP-82 to sharing virtual assets due to complexity of it. I
>> > explained it in the doc.
>> >
>> >
>> > I will now go and catch up with the long thread and updated proposal and
>> >> come back.
>> >>
>> >
>> > Please. I hope the above explaination will help in better understanding
>> of
>> > the proposal, because I think you had some assumptions that do not hold
>> any
>> > more with the new proposal.
>> >
>> > J.
>> >
>> >
>> >>
>> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>>
>> >>> Just to clarify the relation - I updated the AIP now to refer to
>> AIP-82
>> >> and
>> >>> to explain relation between the "cross-team" and "cross-airflow" asset
>> >>> triggering - this is what I added:
>> >>>
>> >>> Note that there is a relation between AIP-82 ("External Driven
>> >> Scheduling")
>> >>> and this part of the functionality. When you have multiple instances
>> of
>> >>> Airflow, you can use shared datasets - "Physical datasets" - that
>> several
>> >>> Airflow Instances can use - for example there could be an S3 object
>> that
>> >> is
>> >>> produced by one airflow instance, and consumed by another. That
>> requires
>> >>> deferred trigger to monitor for such datasets, and appropriate
>> >> permissions
>> >>> to the external dataset, and you could achive similar result to
>> >> cross-team
>> >>> dataset triggering (but cross airflow). However the feature of sharing
>> >>> datasets between the teams also works for virtual assets, that do not
>> >> have
>> >>> physically shared "objects" and trigger that is monitoring for
>> changes in
>> >>> such asset.
>> >>>
>> >>> J.
>> >>>
>> >>>
>> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> >>>
>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>> reference
>> >> to
>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>> through.
>> >>>>
>> >>>> Yep. It did not - because I did not think it was needed or even very
>> >>>> important after the simplifications. AIP-82 has a different scope,
>> >> really.
>> >>>> It only helps when the Assets are "real" data files which we have
>> >> physical
>> >>>> triggers for, it's slightly related - sharing datasets between teams
>> >>>> (including those that do not require physical files and triggers) is
>> >> still
>> >>>> possible in the design we have now, but it's not (and never was) the
>> >>>> **only** reason for having multi-team. There always was (and still
>> is)
>> >> the
>> >>>> possibility of having a common, distinct environments (i.e.
>> dependencies
>> >>>> and providers) per team, the possibility of having connections and
>> >>>> variables that are only accessible to one team and not the other, and
>> >>>> isolating workload execution (all that while allowing to manage
>> multiple
>> >>>> team and schedule things with single deployment). That did not
>> change.
>> >> What
>> >>>> changed a lot is that it is now way simpler, something that we can
>> >>>> implement without heavy changes to the codebase - and give it to our
>> >> users,
>> >>>> so that they can assess if this is something they need without too
>> much
>> >>>> risk and effort.
>> >>>>
>> >>>> This was - I believe the main concern, that the value we get from it
>> is
>> >>>> not dramatic, but the required changes are huge. This "redesign"
>> changes
>> >>>> the equation - the value is still unchanged, but the cost of
>> >> implementing
>> >>>> it and impact on the Airflow codebase is much smaller. I still have
>> not
>> >>>> heard back from Ash if my proposal responds to his original concern
>> >> though,
>> >>>> so I am mostly guessing (also based on the positive impact of others)
>> >> that
>> >>>> yes it does. But to be honest I am not sure and I would love to hear
>> >> back,
>> >>>> I decided to update the AIP to reflect it - regardless, because I
>> think
>> >> the
>> >>>> simplification I proposed keeps the original goals, but is indeed way
>> >>>> simpler.
>> >>>>
>> >>>>> This is a very difficult thread to catch up on.
>> >>>>
>> >>>> Valid point. Let me summarize what is the result:
>> >>>>
>> >>>> * I significantly simplified the implementation proposal comparing to
>> >> the
>> >>>> original version
>> >>>> * main simplification is very limited impact on existing database -
>> >>>> without "ripple effect" that would require us to change a lot of
>> tables,
>> >>>> including their primary keys, and heavily impact the UI
>> >>>> * this is now more of an incremental change that can be implemented
>> way
>> >>>> faster and with far less risk
>> >>>> * updated idea is based on leveraging bundles (already part of our
>> data
>> >>>> model) to map them (many-to-one) to a team - which requires to just
>> >> extend
>> >>>> the data model with bundle mapping and add team_id to connections and
>> >>>> variables. Those are all needed DB changes.
>> >>>>
>> >>>> The AIP is updated - in a one single big change so It should be easy
>> to
>> >>>> compare the changes:
>> >>>>
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
>> >>>> -> I even named the version appropriately "Simplified multi-team
>> AIP" -
>> >> you
>> >>>> can select and compare v.65 with v.66 to see the exact differences I
>> >>>> proposed.
>> >>>>
>> >>>> I hope it will be helpful to catch up and for those who did not
>> follow,
>> >> to
>> >>>> be able to make up their minds about it.
>> >>>>
>> >>>> J.
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
>> >> <vi...@astronomer.io.invalid>
>> >>>> wrote:
>> >>>>
>> >>>>> This is a very difficult thread to catch up on.
>> >>>>> I will take a detailed look at the AIP update to try to figure out
>> the
>> >>>>> changes in the proposal.
>> >>>>>
>> >>>>> From a quick glance, the updated AIP didn't seem to have any
>> reference
>> >> to
>> >>>>> AIP-82, which surprised me, but will take a more detailed read
>> through.
>> >>>>>
>> >>>>> Vikram
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>> >>>>> gopidesupavan@gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's much more
>> >> slim
>> >>>>>> down.
>> >>>>>>
>> >>>>>> left a minor comment. :) Overall looking great.
>> >>>>>>
>> >>>>>> Pavan
>> >>>>>>
>> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>> >>>>> <j_scheffler@gmx.de.invalid
>> >>>>>>>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Thanks for the rework/update of the AIP-72!
>> >>>>>>>
>> >>>>>>> Just a few small comments but overall I like it as it is much
>> leaner
>> >>>>>>> than originally planned and is in a level of complexity that it
>> >> really
>> >>>>>>> seems to be a benefit to close the gap as described.
>> >>>>>>>
>> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
>> >>>>>>>> I updated the AIP - including architecture images and reviewed it
>> >>>>>> (again)
>> >>>>>>>> and corrected any ambiguities and places where it needed to be
>> >>>>> changed.
>> >>>>>>>>
>> >>>>>>>> I think the current state
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>> >>>>>>>> - nicely describes the proposal.
>> >>>>>>>>
>> >>>>>>>> Comparing to the previous one:
>> >>>>>>>>
>> >>>>>>>> 1. The DB changes are far less intrusive - no ripple effect on
>> >>>>> Airflow
>> >>>>>>>> 2. There is no need to merge configurations and provide different
>> >>>>> set
>> >>>>>> of
>> >>>>>>>> configs per team - we can add it later but I do not see why we
>> need
>> >>>>> it
>> >>>>>> in
>> >>>>>>>> this simplified version
>> >>>>>>>> 3. We can still configure a different set of executors per team -
>> >>>>> that
>> >>>>>> is
>> >>>>>>>> already implemented (we just need to wire it to the bundle ->
>> team
>> >>>>>>> mapping).
>> >>>>>>>>
>> >>>>>>>> I think it will be way simpler and faster to implement this way
>> and
>> >>>>> it
>> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can
>> give
>> >>>>> our
>> >>>>>>>> users so that they can provide feedback.
>> >>>>>>>>
>> >>>>>>>> J.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
>> >>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> I like this iteration a bit more now for sure, thanks for being
>> >>>>>>> receptive
>> >>>>>>>>>> to feedback! :)
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> This now becomes quite close to what was proposing before, we
>> now
>> >>>>>> again
>> >>>>>>>>>> have a team ID (which I think is really needed here, glad to
>> see
>> >>>>> it
>> >>>>>>> back)
>> >>>>>>>>>> and it will be used for auth management, configuration
>> >>>>> specification,
>> >>>>>>> etc
>> >>>>>>>>>> but will be carried by Bundle instead of the dag model. Which
>> as
>> >>>>> you
>> >>>>>>> say
>> >>>>>>>>>> “For that we will need to make sure that both api-server,
>> >>>>> scheduler
>> >>>>>> and
>> >>>>>>>>>> triggerer have access to the "bundle definition" (to perform
>> the
>> >>>>>>> mapping)"
>> >>>>>>>>>> which honestly doesn’t feel too much different from the
>> original
>> >>>>>>> proposal
>> >>>>>>>>>> we had last week of adding it to Dag table and ensuring it’s
>> >>>>>> available
>> >>>>>>>>>> everywhere. but either way I’m happy to meet in the middle and
>> >>>>> keep
>> >>>>>> it
>> >>>>>>> on
>> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable location.
>> >>>>>>>>>>
>> >>>>>>>>> I think the big difference is the "ripple effect" that was
>> >>>>> discussed
>> >>>>>> in
>> >>>>>>>>>
>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>> >>>>>> (and I
>> >>>>>>>>> believe - correct me if I am wrong Ash - important trigger for
>> the
>> >>>>>>>>> discussion) so far what we wanted is to extend the primary key
>> and
>> >>>>> it
>> >>>>>>> would
>> >>>>>>>>> ripple through all the pieces of Airflow -> models, API, UI etc.
>> >>>>> ...
>> >>>>>>>>> However - we already have `bundle_name" and "bundle_version" in
>> the
>> >>>>>> Dag
>> >>>>>>>>> model. So I think when we add a separate table where we map the
>> >>>>> bundle
>> >>>>>>> to
>> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not want
>> to
>> >>>>>> change
>> >>>>>>>>> primary key, we do not want to change UI in any way (except
>> >>>>> filtering
>> >>>>>> of
>> >>>>>>>>> DAGs available based on your team - but that will be handled in
>> >>>>> Auth
>> >>>>>>>>> Manager and will not impact UI in any way, I think that's a huge
>> >>>>>>>>> simplification of the implementation, and if we agree to it - i
>> >>>>> think
>> >>>>>> it
>> >>>>>>>>> should speed up the implementation significantly. There are
>> only a
>> >>>>>>> limited
>> >>>>>>>>> number of times where you need to look up the team_id - so
>> having
>> >>>>> the
>> >>>>>>>>> bundle -> team mapping in a separate table and having to look
>> them
>> >>>>> up
>> >>>>>>>>> should not be a problem. And it has much less complexity and
>> >>>>>>>>> "ripple-effect" through the codebase (for example I could
>> imagine
>> >>>>> 100s
>> >>>>>>> or
>> >>>>>>>>> thousands already written tests that would have to be adapted
>> if we
>> >>>>>>> changed
>> >>>>>>>>> the primary key - where there will be pretty much zero impact on
>> >>>>>>> existing
>> >>>>>>>>> tests if we just add bundle -> team lookup table.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> One other thing I’d point out is that I think including
>> executors
>> >>>>> per
>> >>>>>>>>>> team is a very easy win and quite possible without much work. I
>> >>>>>> already
>> >>>>>>>>>> have much of the code written. Executors are already aware of
>> >>>>> Teams
>> >>>>>>> that
>> >>>>>>>>>> own them (merged), I have a PR open to have configuration per
>> team
>> >>>>>>> (with a
>> >>>>>>>>>> quite simple and isolated approach, I believe you approved
>> Jarek).
>> >>>>>> The
>> >>>>>>> last
>> >>>>>>>>>> piece is updating the scheduling logic to route tasks from a
>> >>>>>> particular
>> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much work
>> >>>>> (though
>> >>>>>> it
>> >>>>>>>>>> would be easier if the Task models had a column for the team
>> they
>> >>>>>>> belong
>> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to get the
>> >>>>>> team) I
>> >>>>>>>>>> have a branch where I was experimenting with this logic
>> already.
>> >>>>>>>>>> Any who, long story short, I don’t think we necessarily need to
>> >>>>>> remove
>> >>>>>>>>>> this piece from the project's scope if it is already partly
>> done
>> >>>>> and
>> >>>>>>> not
>> >>>>>>>>>> too difficult.
>> >>>>>>>>>>
>> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want to just
>> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we have
>> it
>> >>>>> in.
>> >>>>>>> And
>> >>>>>>>>> if we can make it in, easily (which it seems we can) - we can
>> also
>> >>>>>>> include
>> >>>>>>>>> it in the first iteration. What I wanted to avoid really (from
>> the
>> >>>>>>> original
>> >>>>>>>>> design) - again trying to simplify it, limit the changes, and
>> >>>>> speed up
>> >>>>>>>>> implementation. And there is one "complexity" that I wanted to
>> >>>>> avoid
>> >>>>>>>>> specifically - having to have separate , additional
>> configuration
>> >>>>> per
>> >>>>>>> team.
>> >>>>>>>>> Not only because it complicates already complex configuration
>> >>>>> handling
>> >>>>>>> (I
>> >>>>>>>>> know we have PR for that) but mostly because if it is not
>> needed,
>> >>>>> we
>> >>>>>> can
>> >>>>>>>>> simplify documentation and explain to our users easier what they
>> >>>>> need
>> >>>>>>> to do
>> >>>>>>>>> to have their own multi-team setup. And I am quite open to
>> keeping
>> >>>>>>>>> multiple-executors if we can avoid complicating configuration.
>> >>>>>>>>>
>> >>>>>>>>> But I think some details of that and whether we really need
>> >>>>> separate
>> >>>>>>>>> configuration might also come as a result of updating the AIP -
>> I
>> >>>>> am
>> >>>>>> not
>> >>>>>>>>> quite sure now if we need it, but we can discuss it when we
>> >>>>> iterate on
>> >>>>>>> the
>> >>>>>>>>> AIP.
>> >>>>>>>>>
>> >>>>>>>>> J.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> ---------------------------------------------------------------------
>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

>
>
> One technical observation: Now that the dag table no longer has a team_id
> in it, what would the behaviour be when a DAG is attempted to move between
> bundles? How do we detect this? (I’m not all convinced that we correctly
> detect duplicate dag ids across bundles today, so I wouldn’t assume or rely
> on the current behaviour.)
>

Of course - yes, I realise that - that problem was also not handled in the
previous iteration to be honest. That is something that dag bundle solution
allows to solve eventually - but I do not think it's a blocker for the
proposed implementation. We will have to eventually add some way of
blocking dags to jump between bundles, we might tackle this separately. I
already wanted to propose a separate update to that - but I did not want to
complicate the current proposal. One thing at a time. I can, however - if
you consider that as a blocker, extend the current AIP with it. Not a big
problem. This is however a bit independent from the team_id introduction.

Overall, I am still unconvinced this proposal has enough real user benefit
> over actually separate deployments, and on balance of the added complexity
> and maintenance burden I do not think it is worth it.
>

That makes me sad, I thought that over the course of the discussion I
addressed all the concerns (in this case the concern was "is it worth with
the cost and little benefit", but when I did it and heavily limited the
impact, now the concern is "is it worth at all as changes are really
minimal" - and surely, anyone can change and adapt their concerns, over
time,  but that one seems like ever-moving target. I hoped at least for
some acknowledgment of some concerns (complexity in this case) is
addressed, but it seems that you are deeply convinced that we do not need
multi-team at all (which is in stark contrast with at least a dozen of
bigger and smaller users of Airflow who submitted talks to Airflow summit
(including about 5 or 6 submissions for Airflow 2025) on how they spent
their engineering effort, time and money on trying to achieve something
similar - they assessed that it's worth, you  assess that it's not worth.
Somehow I trust our users that they were not spending the money, time and
engineering effort to achieve this because they wanted to spend more money.
I think they assessed it's worth it. So I want to make it a bit easier and
more "proper" way for them to do that.

>
> Upgrades: it is not easier to upgrade under this multi team proposal, but
> much much harder. This is based on hard earned experience from helping
> Astronomer users — having to coordinate upgrades between multiple teams
> turns in to a months long slog of the hardest kind of work —  people work:
> getting other teams to agree to do things that they don’t directly care
> about — “It’s working for me, I don’t care about upgrading, we’ll get to it
> next quarter” is a refrain I’ve heard many times.
>

Yes. absolutely - this is why we deferred it until we knew what shape task
isolation and other AIPs we depend on take on. Because it is clear that
pretty much all the problem you explain above are going to be solved with
task isolation. And it's not just my opinion. If you want to argue with it,
you likely need to argue with yourself:
https://github.com/apache/airflow/issues/51545#issuecomment-2980038478. Let
me quote what you wrote there last week:

Ash Berlin Taylor wrote:

> A tight coupling between task-sdk and any "server side" component is the
opposite to one of the goals of AIP-72 (I'm not sure we ever explicitly
said this, but the first point of motivation for the AIP says "Dependency
conflicts for administrators supporting data teams using different versions
of providers, libraries, or python packages")
> In short, my goal with TaskSDK, and the reason for introducing CalVer and
Cadwyn with the execution API is to end up in a world where you can upgrade
the Airflow Scheduler/API server interdependently of any worker nodes (with
the exception that the server must be at least as new as the clients)
> This ability to have version-skew is pretty much non-negotiable to me and
is (other than other languages) one of primary benefits of AIP-72

If you read yourself from that quote it basically means "it will be easy to
upgrade airflow independently of workers". So I am a bit confused here.
Yes, I agree it was difficult, but you yourself explain that when AIP-72
(which since API-67 has been accepted has always beem prerequisite of it)
wrote it will be "easy". So I am not sure why you are bringing it now. We
assume AIP-72 will be completed and this problem will be gone. Let's not
mention it any more please.

The true separation from TaskSDK will likely only land in about 3.2 time
> frame. We are actively working on it, but it’s a slow process of untangling
> lots of assumptions made in the code base over the years. Maybe once we
> have that my view would be different, but right now I think this makes the
> proposal a non-starter. Especially as you are saying that most teams will
> have unique connections. If they’ve got those already, then having an asset
> trigger use those conns to watch/poll for activity is a much easier
> solution to operate and crucially, to scale and upgrade.
>

Yes. I perfectly understand that and I am fully aware of potentially 3.2
time-frame. And that's fine. Actually I heartily invite you to listen to
the part of my talk from Berlin Buzzwords when I was asked for the timeline
- https://youtu.be/EyhZOnbwc-4?t=2226 - this link leads to the exact
timeline in my talk . My answer was basically - "3.1" or "3.2", and I
sincerely hope "3.1" but we might not be able to complete it because we
have other things to do (other - is indeed the Task Isolation work that you
are leading). And that's perfectly fine. And it absolutely does not prevent
us from voting on the AIP now - similarly as we voted on the previous
version of the AIP - knowing that it has some prerequisites a few months
ago. Especially that we know that the feature we need from task isolation
is "non-negotiable". I.e. it WILL happen. We don't hope for it, we know it
will be there. Those are your own words.


> >  I think we can’t compare AIP-82 to sharing virtual assets due to
> complexity of it.
>
> Virtual Assets was a mistake, and not how users actually want to use them.
> Mea culpa
>

This is the first time I hear this - certainly you never raised this
concern on the devlist. So if you have some concerns about virtual assets I
think you should raise it on the devlist, because I think everyone here is
missing some conversation (or maybe it's just your private opinion that
you never shared with anyone, but maybe it's worth). I would be
interested to hear how the feature that was absolutely most successful
feature of airflow 2 was a mistake. According to the 2024 survey
https://airflow.apache.org/blog/airflow-survey-2024/  - 48% of Airflow
users have been using it, even if it was added as one of the last
big features of Airflow 2. It's the MOST used feature out of all the
features out there. I would be really curious to see how it was a mistake
(but  please start a separate thread explaining why you think it was a
mistake, what are your data points and what do you think should be fixed.
Just dropping "virtual assets were a mistake" in the middle of multi-team
conversation seems completely unjustified without knowing what you are
talking about. So I think, until we know more, this argument has no base.


>
> S
> To restate my points:
>
> - Sharing a deployment between teams today/in 3.1 is operationally more
> complex (both scaling, and upgrades) — this is a con, not a plus.
>

Surely. But it will be easier when AIP-72 is complete (which I am
definitely looking forward to and as clearly explained in AIP-82, is a
prerequisite of it). Nothing changed here.


> - The main user benefit appears to be “allow teams’ DAGs to communicate
> via Assets”, in which case we can do that today by putting more work in to
> AIP-82’s Asset triggers
>

No. Lower operational complexity for multi-teams (providing that we deliver
AIP-72) is another benefit. Virtual assets is another, and since there is
no ground in "virtual assets is a mistake" statement (not until you explain
what you mean by that in a separate discussion) - this is also still a very
valid point.


> Soon, we will have then be asked about cross-team governance, policy
> enforcement, and potentially unbounded edge cases (e.g., team-specific
> secrets, roles, quotas). ain, you get this for free with truely separate
> deployments already
> allow different teams to use different executors (including multiple
> executors per-team following AIP-61)
>

Not really. We very explicitly say in the AIP that his is not a goal and
that we have no plans for. And yes, using separate executors per team is
actually back in the AIP-82 in case you did not notice (and the code needed
for it's even implemented and merged already in main by Vincent).


> Provably not true right now, and until ~3.2 delivers the full Task
> SDK/Core dependency separation this would be _more_ work to upgrade, not
> less, and that work is not shared but still on a central team.
>

Absolutely - we will wait for AIP-72 completion. I do not want to say 3.1
or 3.2 directly - because there are - as you said - a lot of moving pieces.
So my target for multi-team is "After AIP-72 is completed". Full stop. But
there is nothing wrong with accepting the AIP now and doing preparatory
work in parallel. Similarly as there is no way to have a baby in 1 month by
9 women, there is no way adding more effort to task-sdk isolation will
speed it up - we alredy have not only 3 people (you leading it, Kaxil and
Amog) but also all the help from me and even 10s of different contributors
(for example with the recent db_test cleanup that I took leadership on) -
and there are people who wish to work on adding multi-team features. Since
the design heavily limits impact on airflow codebase and interactions with
task-sdk implementation, there is nothing wrong with starting
implementation in parallel either- amazon team is keen to move it forward -
they even already implemented SQS trigger for assets, and we are working
together on FAB removal, Keycloak authentication manager - and they seem to
still have capacity and drive to progress multi-team. So I am not sure if
we are trading off something. There is no "if we work on more on task sdk
and drop multi-team things will be faster". Generally in open source people
work in the area where they feel they can provide best value - such as you
working on task-sdk, me on CI,dev env, they will deliver more value on
multi-team


>
> So please, as succinctly as possible, please tell me what the direct
> benefit to users this proposal is over us putting this effort in to writing
> better Asset triggers instead?
>


* less operational overhead for managing multi-team (once AIP-72 is
complete) where separate execution environments are important
* virtual assets sharing
* ability of having "admin" and "team sharing" capability where dags from
multiple teams can be seen in a single Airflow UI  (requires custom RBAC)

None of this can be done via beter asset triggers


>
> > On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > My counter-points:
> >
> >
> >> 1. Managing a multi team deployment is not materially different from
> >> managing a deployment per team
> >>
> >
> > It's a bit easier - especially when it comes to upgrades (especially in
> the
> > case we are targetting when we are not targetting multi-tenant, but
> several
> > relatively closely cooperating teams with different dependncy requiremens
> > and isolation need.
> >
> > 2. The database changes were quite wide-reaching
> >>
> >
> > Yes. that is addressed.
> >
> >
> >> 3. I don’t believe the original AIP (again, I haven’t read the updated
> >> proposal or recent messages on the thread. yet) will meet what many
> users
> >> want out of a multiteam solution
> >>
> >
> > I think we will only see when we try. A lot of people thing they would,
> > even if they are warned. I know at least one user (Wealthsimple) who
> > definitely want to use it and they got a very detailed explanation of the
> > idea and understand it well. So I am sure that **some** users would. But
> we
> > do not know how many.
> >
> >
> >> To expand on those points a bit more
> >>
> >> On 1. The only components that are shared are, I think, the scheduler
> and
> >> the API server, and it’s arguable if that is actually a good idea given
> >> those are likely to be the most performance sensitive components anyway.
> >>
> >> Additionally the fact that the scheduler is a shared component makes
> >> upgrading it almost a non starter as you would likely need buy-in,
> changes,
> >> and testing form ALL teams using it. I’d argue that this is a huge
> negative
> >> until we finish off the version indepence work of AIP-72.
> >>
> >
> > Quite disagree here - especially that our target is that task-sdk is
> > supposed to provide all isolation that is needed. There should be 0
> changes
> > in the dags needed to upgrade scheduler, api_server, triggerer -
> precisely
> > because we introduced backwards-compatible task-sdk.
> >
> > On 3 my complaint is essentially that this doesn’t go nearly far enough.
> It
> >> doesn’t allow read only views to other teams dags. I don’t think it
> allows
> >> you to be in multiple teams at once. You can’t share a connection
> between
> >> teams but only allow certain specified dags to access it, but would
> have to
> >> either be globally usable, or duplicated-and-kept-in-sync between
> teams. In
> >> short I think it fall short of being useful..
> >>
> >
> > Oh absolutely all that is possible (except sharing single connections
> > between multiple teams - which is a very niche use cases and duplication
> > here is perfectly ok as first approximation - and if we need more we can
> > add it later).
> >
> > Auth manager RBAC and access is abstracted away, and the Keyclock Manager
> > implemented by Vincent allows to manage completely independent and
> separate
> > RBAC based on arguments and resources provided by Airflow. There is
> nothing
> > to prevent the user who configures KeyCloak RBAC to define it in the way:
> >
> > if group a > allow to read a and write b
> > if group b > alllow to write b but not a
> >
> > and any other combinations. KeyCloak implementation - pretty advanced
> > already - (and design of auth manager) completely abstracts away both
> > authentication and authorization to KeyCloak and KeyCloak has RBAC
> > management built in. Also any of the users can write their own - even
> > hard-coded authentication manager to do the same if they do not want to
> > have configurable KeyCloak. Even SimpleAuthManager could be hard-coded to
> > provide thiose features.
> >
> >
> >>
> >> So on the surface, I’m no more in favour of using dag bundle as a
> >> replacement for team id as I think most of the above points still stand.
> >>
> >
> > We disagree here.
> >
> >>
> >> My counter proposal: We do _nothing_ to core airflow. We work on
> improving
> >> the event-based trigger o fdags (write more triggers for read/check
> remote
> >> Assets etc) so that teams can have 100% isolated deployments but still
> >> trigger dags based on asset events from other teams.
> >>
> >
> > That does not solve any of the other design goals - only allows to
> trigger
> > assets a bit more easily (but also it's not entirely solved by AIP-82
> > because it does not solve virtual assets - only ones that have defined
> > triggerer and "something" to listen on - which is way more complex than
> > just defining asset in a Dag and using it in another). I think we can't
> > compare AIP-82 to sharing virtual assets due to complexity of it. I
> > explained it in the doc.
> >
> >
> > I will now go and catch up with the long thread and updated proposal and
> >> come back.
> >>
> >
> > Please. I hope the above explaination will help in better understanding
> of
> > the proposal, because I think you had some assumptions that do not hold
> any
> > more with the new proposal.
> >
> > J.
> >
> >
> >>
> >>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>> Just to clarify the relation - I updated the AIP now to refer to AIP-82
> >> and
> >>> to explain relation between the "cross-team" and "cross-airflow" asset
> >>> triggering - this is what I added:
> >>>
> >>> Note that there is a relation between AIP-82 ("External Driven
> >> Scheduling")
> >>> and this part of the functionality. When you have multiple instances of
> >>> Airflow, you can use shared datasets - "Physical datasets" - that
> several
> >>> Airflow Instances can use - for example there could be an S3 object
> that
> >> is
> >>> produced by one airflow instance, and consumed by another. That
> requires
> >>> deferred trigger to monitor for such datasets, and appropriate
> >> permissions
> >>> to the external dataset, and you could achive similar result to
> >> cross-team
> >>> dataset triggering (but cross airflow). However the feature of sharing
> >>> datasets between the teams also works for virtual assets, that do not
> >> have
> >>> physically shared "objects" and trigger that is monitoring for changes
> in
> >>> such asset.
> >>>
> >>> J.
> >>>
> >>>
> >>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>>>> From a quick glance, the updated AIP didn't seem to have any
> reference
> >> to
> >>>>> AIP-82, which surprised me, but will take a more detailed read
> through.
> >>>>
> >>>> Yep. It did not - because I did not think it was needed or even very
> >>>> important after the simplifications. AIP-82 has a different scope,
> >> really.
> >>>> It only helps when the Assets are "real" data files which we have
> >> physical
> >>>> triggers for, it's slightly related - sharing datasets between teams
> >>>> (including those that do not require physical files and triggers) is
> >> still
> >>>> possible in the design we have now, but it's not (and never was) the
> >>>> **only** reason for having multi-team. There always was (and still is)
> >> the
> >>>> possibility of having a common, distinct environments (i.e.
> dependencies
> >>>> and providers) per team, the possibility of having connections and
> >>>> variables that are only accessible to one team and not the other, and
> >>>> isolating workload execution (all that while allowing to manage
> multiple
> >>>> team and schedule things with single deployment). That did not change.
> >> What
> >>>> changed a lot is that it is now way simpler, something that we can
> >>>> implement without heavy changes to the codebase - and give it to our
> >> users,
> >>>> so that they can assess if this is something they need without too
> much
> >>>> risk and effort.
> >>>>
> >>>> This was - I believe the main concern, that the value we get from it
> is
> >>>> not dramatic, but the required changes are huge. This "redesign"
> changes
> >>>> the equation - the value is still unchanged, but the cost of
> >> implementing
> >>>> it and impact on the Airflow codebase is much smaller. I still have
> not
> >>>> heard back from Ash if my proposal responds to his original concern
> >> though,
> >>>> so I am mostly guessing (also based on the positive impact of others)
> >> that
> >>>> yes it does. But to be honest I am not sure and I would love to hear
> >> back,
> >>>> I decided to update the AIP to reflect it - regardless, because I
> think
> >> the
> >>>> simplification I proposed keeps the original goals, but is indeed way
> >>>> simpler.
> >>>>
> >>>>> This is a very difficult thread to catch up on.
> >>>>
> >>>> Valid point. Let me summarize what is the result:
> >>>>
> >>>> * I significantly simplified the implementation proposal comparing to
> >> the
> >>>> original version
> >>>> * main simplification is very limited impact on existing database -
> >>>> without "ripple effect" that would require us to change a lot of
> tables,
> >>>> including their primary keys, and heavily impact the UI
> >>>> * this is now more of an incremental change that can be implemented
> way
> >>>> faster and with far less risk
> >>>> * updated idea is based on leveraging bundles (already part of our
> data
> >>>> model) to map them (many-to-one) to a team - which requires to just
> >> extend
> >>>> the data model with bundle mapping and add team_id to connections and
> >>>> variables. Those are all needed DB changes.
> >>>>
> >>>> The AIP is updated - in a one single big change so It should be easy
> to
> >>>> compare the changes:
> >>>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
> >>>> -> I even named the version appropriately "Simplified multi-team AIP"
> -
> >> you
> >>>> can select and compare v.65 with v.66 to see the exact differences I
> >>>> proposed.
> >>>>
> >>>> I hope it will be helpful to catch up and for those who did not
> follow,
> >> to
> >>>> be able to make up their minds about it.
> >>>>
> >>>> J.
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
> >> <vi...@astronomer.io.invalid>
> >>>> wrote:
> >>>>
> >>>>> This is a very difficult thread to catch up on.
> >>>>> I will take a detailed look at the AIP update to try to figure out
> the
> >>>>> changes in the proposal.
> >>>>>
> >>>>> From a quick glance, the updated AIP didn't seem to have any
> reference
> >> to
> >>>>> AIP-82, which surprised me, but will take a more detailed read
> through.
> >>>>>
> >>>>> Vikram
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
> >>>>> gopidesupavan@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks Jarek, that's a great update on this AIP, now it's much more
> >> slim
> >>>>>> down.
> >>>>>>
> >>>>>> left a minor comment. :) Overall looking great.
> >>>>>>
> >>>>>> Pavan
> >>>>>>
> >>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
> >>>>> <j_scheffler@gmx.de.invalid
> >>>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks for the rework/update of the AIP-72!
> >>>>>>>
> >>>>>>> Just a few small comments but overall I like it as it is much
> leaner
> >>>>>>> than originally planned and is in a level of complexity that it
> >> really
> >>>>>>> seems to be a benefit to close the gap as described.
> >>>>>>>
> >>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
> >>>>>>>> I updated the AIP - including architecture images and reviewed it
> >>>>>> (again)
> >>>>>>>> and corrected any ambiguities and places where it needed to be
> >>>>> changed.
> >>>>>>>>
> >>>>>>>> I think the current state
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >>>>>>>> - nicely describes the proposal.
> >>>>>>>>
> >>>>>>>> Comparing to the previous one:
> >>>>>>>>
> >>>>>>>> 1. The DB changes are far less intrusive - no ripple effect on
> >>>>> Airflow
> >>>>>>>> 2. There is no need to merge configurations and provide different
> >>>>> set
> >>>>>> of
> >>>>>>>> configs per team - we can add it later but I do not see why we
> need
> >>>>> it
> >>>>>> in
> >>>>>>>> this simplified version
> >>>>>>>> 3. We can still configure a different set of executors per team -
> >>>>> that
> >>>>>> is
> >>>>>>>> already implemented (we just need to wire it to the bundle -> team
> >>>>>>> mapping).
> >>>>>>>>
> >>>>>>>> I think it will be way simpler and faster to implement this way
> and
> >>>>> it
> >>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can give
> >>>>> our
> >>>>>>>> users so that they can provide feedback.
> >>>>>>>>
> >>>>>>>> J.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> I like this iteration a bit more now for sure, thanks for being
> >>>>>>> receptive
> >>>>>>>>>> to feedback! :)
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> This now becomes quite close to what was proposing before, we
> now
> >>>>>> again
> >>>>>>>>>> have a team ID (which I think is really needed here, glad to see
> >>>>> it
> >>>>>>> back)
> >>>>>>>>>> and it will be used for auth management, configuration
> >>>>> specification,
> >>>>>>> etc
> >>>>>>>>>> but will be carried by Bundle instead of the dag model. Which as
> >>>>> you
> >>>>>>> say
> >>>>>>>>>> “For that we will need to make sure that both api-server,
> >>>>> scheduler
> >>>>>> and
> >>>>>>>>>> triggerer have access to the "bundle definition" (to perform the
> >>>>>>> mapping)"
> >>>>>>>>>> which honestly doesn’t feel too much different from the original
> >>>>>>> proposal
> >>>>>>>>>> we had last week of adding it to Dag table and ensuring it’s
> >>>>>> available
> >>>>>>>>>> everywhere. but either way I’m happy to meet in the middle and
> >>>>> keep
> >>>>>> it
> >>>>>>> on
> >>>>>>>>>> Bundle if everyone else feels that’s a more suitable location.
> >>>>>>>>>>
> >>>>>>>>> I think the big difference is the "ripple effect" that was
> >>>>> discussed
> >>>>>> in
> >>>>>>>>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
> >>>>>> (and I
> >>>>>>>>> believe - correct me if I am wrong Ash - important trigger for
> the
> >>>>>>>>> discussion) so far what we wanted is to extend the primary key
> and
> >>>>> it
> >>>>>>> would
> >>>>>>>>> ripple through all the pieces of Airflow -> models, API, UI etc.
> >>>>> ...
> >>>>>>>>> However - we already have `bundle_name" and "bundle_version" in
> the
> >>>>>> Dag
> >>>>>>>>> model. So I think when we add a separate table where we map the
> >>>>> bundle
> >>>>>>> to
> >>>>>>>>> the team, the "ripple effect" will be almost 0. We do not want to
> >>>>>> change
> >>>>>>>>> primary key, we do not want to change UI in any way (except
> >>>>> filtering
> >>>>>> of
> >>>>>>>>> DAGs available based on your team - but that will be handled in
> >>>>> Auth
> >>>>>>>>> Manager and will not impact UI in any way, I think that's a huge
> >>>>>>>>> simplification of the implementation, and if we agree to it - i
> >>>>> think
> >>>>>> it
> >>>>>>>>> should speed up the implementation significantly. There are only
> a
> >>>>>>> limited
> >>>>>>>>> number of times where you need to look up the team_id - so having
> >>>>> the
> >>>>>>>>> bundle -> team mapping in a separate table and having to look
> them
> >>>>> up
> >>>>>>>>> should not be a problem. And it has much less complexity and
> >>>>>>>>> "ripple-effect" through the codebase (for example I could imagine
> >>>>> 100s
> >>>>>>> or
> >>>>>>>>> thousands already written tests that would have to be adapted if
> we
> >>>>>>> changed
> >>>>>>>>> the primary key - where there will be pretty much zero impact on
> >>>>>>> existing
> >>>>>>>>> tests if we just add bundle -> team lookup table.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> One other thing I’d point out is that I think including
> executors
> >>>>> per
> >>>>>>>>>> team is a very easy win and quite possible without much work. I
> >>>>>> already
> >>>>>>>>>> have much of the code written. Executors are already aware of
> >>>>> Teams
> >>>>>>> that
> >>>>>>>>>> own them (merged), I have a PR open to have configuration per
> team
> >>>>>>> (with a
> >>>>>>>>>> quite simple and isolated approach, I believe you approved
> Jarek).
> >>>>>> The
> >>>>>>> last
> >>>>>>>>>> piece is updating the scheduling logic to route tasks from a
> >>>>>> particular
> >>>>>>>>>> Bundle to the correct executor, which shouldn’t be much work
> >>>>> (though
> >>>>>> it
> >>>>>>>>>> would be easier if the Task models had a column for the team
> they
> >>>>>>> belong
> >>>>>>>>>> to, rather than having to look up the Dag and Bundle to get the
> >>>>>> team) I
> >>>>>>>>>> have a branch where I was experimenting with this logic already.
> >>>>>>>>>> Any who, long story short, I don’t think we necessarily need to
> >>>>>> remove
> >>>>>>>>>> this piece from the project's scope if it is already partly done
> >>>>> and
> >>>>>>> not
> >>>>>>>>>> too difficult.
> >>>>>>>>>>
> >>>>>>>>> Yeah. I hear you here again. Certainly I would not want to just
> >>>>>>>>> **remove** it from the code. And, yep I totally forgot we have it
> >>>>> in.
> >>>>>>> And
> >>>>>>>>> if we can make it in, easily (which it seems we can) - we can
> also
> >>>>>>> include
> >>>>>>>>> it in the first iteration. What I wanted to avoid really (from
> the
> >>>>>>> original
> >>>>>>>>> design) - again trying to simplify it, limit the changes, and
> >>>>> speed up
> >>>>>>>>> implementation. And there is one "complexity" that I wanted to
> >>>>> avoid
> >>>>>>>>> specifically - having to have separate , additional configuration
> >>>>> per
> >>>>>>> team.
> >>>>>>>>> Not only because it complicates already complex configuration
> >>>>> handling
> >>>>>>> (I
> >>>>>>>>> know we have PR for that) but mostly because if it is not needed,
> >>>>> we
> >>>>>> can
> >>>>>>>>> simplify documentation and explain to our users easier what they
> >>>>> need
> >>>>>>> to do
> >>>>>>>>> to have their own multi-team setup. And I am quite open to
> keeping
> >>>>>>>>> multiple-executors if we can avoid complicating configuration.
> >>>>>>>>>
> >>>>>>>>> But I think some details of that and whether we really need
> >>>>> separate
> >>>>>>>>> configuration might also come as a result of updating the AIP - I
> >>>>> am
> >>>>>> not
> >>>>>>>>> quite sure now if we need it, but we can discuss it when we
> >>>>> iterate on
> >>>>>>> the
> >>>>>>>>> AIP.
> >>>>>>>>>
> >>>>>>>>> J.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Ash Berlin-Taylor <as...@apache.org>]

Okay, so I’ve read the updated AIP-67 document.

One technical observation: Now that the dag table no longer has a team_id in it, what would the behaviour be when a DAG is attempted to move between bundles? How do we detect this? (I’m not all convinced that we correctly detect duplicate dag ids across bundles today, so I wouldn’t assume or rely on the current behaviour.)

Overall, I am still unconvinced this proposal has enough real user benefit over actually separate deployments, and on balance of the added complexity and maintenance burden I do not think it is worth it.

Upgrades: it is not easier to upgrade under this multi team proposal, but much much harder. This is based on hard earned experience from helping Astronomer users — having to coordinate upgrades between multiple teams turns in to a months long slog of the hardest kind of work —  people work: getting other teams to agree to do things that they don’t directly care about — “It’s working for me, I don’t care about upgrading, we’ll get to it next quarter” is a refrain I’ve heard many times.

The true separation from TaskSDK will likely only land in about 3.2 time frame. We are actively working on it, but it’s a slow process of untangling lots of assumptions made in the code base over the years. Maybe once we have that my view would be different, but right now I think this makes the proposal a non-starter. Especially as you are saying that most teams will have unique connections. If they’ve got those already, then having an asset trigger use those conns to watch/poll for activity is a much easier solution to operate and crucially, to scale and upgrade.

>  I think we can’t compare AIP-82 to sharing virtual assets due to complexity of it.

Virtual Assets was a mistake, and not how users actually want to use them. Mea culpa

So you have almost entirely missed my point in the original email in your re-draft.

To restate my points:

- Sharing a deployment between teams today/in 3.1 is operationally more complex (both scaling, and upgrades) — this is a con, not a plus.
- The main user benefit appears to be “allow teams’ DAGs to communicate via Assets”, in which case we can do that today by putting more work in to AIP-82’s Asset triggers


Covering all the points from the summary of the AIP: 

access to team-specific configuration (variables, connections)
For free/all you get if deployments are truly separate
execute the code submitted by team-specific DAG authors in isolated environments (both parsing and execution)
Again, this is the only way it could work with separate deployments. And if one team want’s multiple envs you can get this today with worker queues.
allow different teams to use different set of dependencies/execution environment libraries
You get this for free with truly separate deployments already without additional complexity and guarantees of true isolation & security without risking fairness/resource contention (e.g for the Scheduler scheduling resources fairly).

Soon, we will have then be asked about cross-team governance, policy enforcement, and potentially unbounded edge cases (e.g., team-specific secrets, roles, quotas). ain, you get this for free with truely separate deployments already
allow different teams to use different executors (including multiple executors per-team following AIP-61)
Ditto
allows to link DAGs between different teams via a “dataset” feature. Datasets can be produced/consumed by different teams - without AIP-82 "External event-riven scheduling" and without configuring credentials for authorisation between different Airlfow instances
Ignoring the addition of “without AIP-82” which feels like a post-hoc justification, by adding better/more asset triggers to Airflow everyone benefits.
allow the UI users to to see a subset of single team or multiple teams DAGs/ Connections / Variables/ DAGRuns etc.
Being able to view multiple team’s dags is the one thing my idea wouldn’t deliver.
reduce (and allows to distribute) maintenance/upgrade load on Devops/deployment managers where they have to provide services to different teams
Provably not true right now, and until ~3.2 delivers the full Task SDK/Core dependency separation this would be _more_ work to upgrade, not less, and that work is not shared but still on a central team.

So please, as succinctly as possible, please tell me what the direct benefit to users this proposal is over us putting this effort in to writing better Asset triggers instead?


> On 23 Jun 2025, at 10:57, Jarek Potiuk <ja...@potiuk.com> wrote:
> 
> My counter-points:
> 
> 
>> 1. Managing a multi team deployment is not materially different from
>> managing a deployment per team
>> 
> 
> It's a bit easier - especially when it comes to upgrades (especially in the
> case we are targetting when we are not targetting multi-tenant, but several
> relatively closely cooperating teams with different dependncy requiremens
> and isolation need.
> 
> 2. The database changes were quite wide-reaching
>> 
> 
> Yes. that is addressed.
> 
> 
>> 3. I don’t believe the original AIP (again, I haven’t read the updated
>> proposal or recent messages on the thread. yet) will meet what many users
>> want out of a multiteam solution
>> 
> 
> I think we will only see when we try. A lot of people thing they would,
> even if they are warned. I know at least one user (Wealthsimple) who
> definitely want to use it and they got a very detailed explanation of the
> idea and understand it well. So I am sure that **some** users would. But we
> do not know how many.
> 
> 
>> To expand on those points a bit more
>> 
>> On 1. The only components that are shared are, I think, the scheduler and
>> the API server, and it’s arguable if that is actually a good idea given
>> those are likely to be the most performance sensitive components anyway.
>> 
>> Additionally the fact that the scheduler is a shared component makes
>> upgrading it almost a non starter as you would likely need buy-in, changes,
>> and testing form ALL teams using it. I’d argue that this is a huge negative
>> until we finish off the version indepence work of AIP-72.
>> 
> 
> Quite disagree here - especially that our target is that task-sdk is
> supposed to provide all isolation that is needed. There should be 0 changes
> in the dags needed to upgrade scheduler, api_server, triggerer - precisely
> because we introduced backwards-compatible task-sdk.
> 
> On 3 my complaint is essentially that this doesn’t go nearly far enough. It
>> doesn’t allow read only views to other teams dags. I don’t think it allows
>> you to be in multiple teams at once. You can’t share a connection between
>> teams but only allow certain specified dags to access it, but would have to
>> either be globally usable, or duplicated-and-kept-in-sync between teams. In
>> short I think it fall short of being useful..
>> 
> 
> Oh absolutely all that is possible (except sharing single connections
> between multiple teams - which is a very niche use cases and duplication
> here is perfectly ok as first approximation - and if we need more we can
> add it later).
> 
> Auth manager RBAC and access is abstracted away, and the Keyclock Manager
> implemented by Vincent allows to manage completely independent and separate
> RBAC based on arguments and resources provided by Airflow. There is nothing
> to prevent the user who configures KeyCloak RBAC to define it in the way:
> 
> if group a > allow to read a and write b
> if group b > alllow to write b but not a
> 
> and any other combinations. KeyCloak implementation - pretty advanced
> already - (and design of auth manager) completely abstracts away both
> authentication and authorization to KeyCloak and KeyCloak has RBAC
> management built in. Also any of the users can write their own - even
> hard-coded authentication manager to do the same if they do not want to
> have configurable KeyCloak. Even SimpleAuthManager could be hard-coded to
> provide thiose features.
> 
> 
>> 
>> So on the surface, I’m no more in favour of using dag bundle as a
>> replacement for team id as I think most of the above points still stand.
>> 
> 
> We disagree here.
> 
>> 
>> My counter proposal: We do _nothing_ to core airflow. We work on improving
>> the event-based trigger o fdags (write more triggers for read/check remote
>> Assets etc) so that teams can have 100% isolated deployments but still
>> trigger dags based on asset events from other teams.
>> 
> 
> That does not solve any of the other design goals - only allows to trigger
> assets a bit more easily (but also it's not entirely solved by AIP-82
> because it does not solve virtual assets - only ones that have defined
> triggerer and "something" to listen on - which is way more complex than
> just defining asset in a Dag and using it in another). I think we can't
> compare AIP-82 to sharing virtual assets due to complexity of it. I
> explained it in the doc.
> 
> 
> I will now go and catch up with the long thread and updated proposal and
>> come back.
>> 
> 
> Please. I hope the above explaination will help in better understanding of
> the proposal, because I think you had some assumptions that do not hold any
> more with the new proposal.
> 
> J.
> 
> 
>> 
>>> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
>>> 
>>> Just to clarify the relation - I updated the AIP now to refer to AIP-82
>> and
>>> to explain relation between the "cross-team" and "cross-airflow" asset
>>> triggering - this is what I added:
>>> 
>>> Note that there is a relation between AIP-82 ("External Driven
>> Scheduling")
>>> and this part of the functionality. When you have multiple instances of
>>> Airflow, you can use shared datasets - "Physical datasets" - that several
>>> Airflow Instances can use - for example there could be an S3 object that
>> is
>>> produced by one airflow instance, and consumed by another. That requires
>>> deferred trigger to monitor for such datasets, and appropriate
>> permissions
>>> to the external dataset, and you could achive similar result to
>> cross-team
>>> dataset triggering (but cross airflow). However the feature of sharing
>>> datasets between the teams also works for virtual assets, that do not
>> have
>>> physically shared "objects" and trigger that is monitoring for changes in
>>> such asset.
>>> 
>>> J.
>>> 
>>> 
>>> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>> 
>>>>> From a quick glance, the updated AIP didn't seem to have any reference
>> to
>>>>> AIP-82, which surprised me, but will take a more detailed read through.
>>>> 
>>>> Yep. It did not - because I did not think it was needed or even very
>>>> important after the simplifications. AIP-82 has a different scope,
>> really.
>>>> It only helps when the Assets are "real" data files which we have
>> physical
>>>> triggers for, it's slightly related - sharing datasets between teams
>>>> (including those that do not require physical files and triggers) is
>> still
>>>> possible in the design we have now, but it's not (and never was) the
>>>> **only** reason for having multi-team. There always was (and still is)
>> the
>>>> possibility of having a common, distinct environments (i.e. dependencies
>>>> and providers) per team, the possibility of having connections and
>>>> variables that are only accessible to one team and not the other, and
>>>> isolating workload execution (all that while allowing to manage multiple
>>>> team and schedule things with single deployment). That did not change.
>> What
>>>> changed a lot is that it is now way simpler, something that we can
>>>> implement without heavy changes to the codebase - and give it to our
>> users,
>>>> so that they can assess if this is something they need without too much
>>>> risk and effort.
>>>> 
>>>> This was - I believe the main concern, that the value we get from it is
>>>> not dramatic, but the required changes are huge. This "redesign" changes
>>>> the equation - the value is still unchanged, but the cost of
>> implementing
>>>> it and impact on the Airflow codebase is much smaller. I still have not
>>>> heard back from Ash if my proposal responds to his original concern
>> though,
>>>> so I am mostly guessing (also based on the positive impact of others)
>> that
>>>> yes it does. But to be honest I am not sure and I would love to hear
>> back,
>>>> I decided to update the AIP to reflect it - regardless, because I think
>> the
>>>> simplification I proposed keeps the original goals, but is indeed way
>>>> simpler.
>>>> 
>>>>> This is a very difficult thread to catch up on.
>>>> 
>>>> Valid point. Let me summarize what is the result:
>>>> 
>>>> * I significantly simplified the implementation proposal comparing to
>> the
>>>> original version
>>>> * main simplification is very limited impact on existing database -
>>>> without "ripple effect" that would require us to change a lot of tables,
>>>> including their primary keys, and heavily impact the UI
>>>> * this is now more of an incremental change that can be implemented way
>>>> faster and with far less risk
>>>> * updated idea is based on leveraging bundles (already part of our data
>>>> model) to map them (many-to-one) to a team - which requires to just
>> extend
>>>> the data model with bundle mapping and add team_id to connections and
>>>> variables. Those are all needed DB changes.
>>>> 
>>>> The AIP is updated - in a one single big change so It should be easy to
>>>> compare the changes:
>>>> 
>> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
>>>> -> I even named the version appropriately "Simplified multi-team AIP" -
>> you
>>>> can select and compare v.65 with v.66 to see the exact differences I
>>>> proposed.
>>>> 
>>>> I hope it will be helpful to catch up and for those who did not follow,
>> to
>>>> be able to make up their minds about it.
>>>> 
>>>> J.
>>>> 
>>>> 
>>>> 
>>>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
>> <vi...@astronomer.io.invalid>
>>>> wrote:
>>>> 
>>>>> This is a very difficult thread to catch up on.
>>>>> I will take a detailed look at the AIP update to try to figure out the
>>>>> changes in the proposal.
>>>>> 
>>>>> From a quick glance, the updated AIP didn't seem to have any reference
>> to
>>>>> AIP-82, which surprised me, but will take a more detailed read through.
>>>>> 
>>>>> Vikram
>>>>> 
>>>>> 
>>>>> 
>>>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>>>>> gopidesupavan@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> Thanks Jarek, that's a great update on this AIP, now it's much more
>> slim
>>>>>> down.
>>>>>> 
>>>>>> left a minor comment. :) Overall looking great.
>>>>>> 
>>>>>> Pavan
>>>>>> 
>>>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>>>>> <j_scheffler@gmx.de.invalid
>>>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Thanks for the rework/update of the AIP-72!
>>>>>>> 
>>>>>>> Just a few small comments but overall I like it as it is much leaner
>>>>>>> than originally planned and is in a level of complexity that it
>> really
>>>>>>> seems to be a benefit to close the gap as described.
>>>>>>> 
>>>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
>>>>>>>> I updated the AIP - including architecture images and reviewed it
>>>>>> (again)
>>>>>>>> and corrected any ambiguities and places where it needed to be
>>>>> changed.
>>>>>>>> 
>>>>>>>> I think the current state
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>>>>>>>> - nicely describes the proposal.
>>>>>>>> 
>>>>>>>> Comparing to the previous one:
>>>>>>>> 
>>>>>>>> 1. The DB changes are far less intrusive - no ripple effect on
>>>>> Airflow
>>>>>>>> 2. There is no need to merge configurations and provide different
>>>>> set
>>>>>> of
>>>>>>>> configs per team - we can add it later but I do not see why we need
>>>>> it
>>>>>> in
>>>>>>>> this simplified version
>>>>>>>> 3. We can still configure a different set of executors per team -
>>>>> that
>>>>>> is
>>>>>>>> already implemented (we just need to wire it to the bundle -> team
>>>>>>> mapping).
>>>>>>>> 
>>>>>>>> I think it will be way simpler and faster to implement this way and
>>>>> it
>>>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can give
>>>>> our
>>>>>>>> users so that they can provide feedback.
>>>>>>>> 
>>>>>>>> J.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> I like this iteration a bit more now for sure, thanks for being
>>>>>>> receptive
>>>>>>>>>> to feedback! :)
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> This now becomes quite close to what was proposing before, we now
>>>>>> again
>>>>>>>>>> have a team ID (which I think is really needed here, glad to see
>>>>> it
>>>>>>> back)
>>>>>>>>>> and it will be used for auth management, configuration
>>>>> specification,
>>>>>>> etc
>>>>>>>>>> but will be carried by Bundle instead of the dag model. Which as
>>>>> you
>>>>>>> say
>>>>>>>>>> “For that we will need to make sure that both api-server,
>>>>> scheduler
>>>>>> and
>>>>>>>>>> triggerer have access to the "bundle definition" (to perform the
>>>>>>> mapping)"
>>>>>>>>>> which honestly doesn’t feel too much different from the original
>>>>>>> proposal
>>>>>>>>>> we had last week of adding it to Dag table and ensuring it’s
>>>>>> available
>>>>>>>>>> everywhere. but either way I’m happy to meet in the middle and
>>>>> keep
>>>>>> it
>>>>>>> on
>>>>>>>>>> Bundle if everyone else feels that’s a more suitable location.
>>>>>>>>>> 
>>>>>>>>> I think the big difference is the "ripple effect" that was
>>>>> discussed
>>>>>> in
>>>>>>>>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>>>>>> (and I
>>>>>>>>> believe - correct me if I am wrong Ash - important trigger for the
>>>>>>>>> discussion) so far what we wanted is to extend the primary key and
>>>>> it
>>>>>>> would
>>>>>>>>> ripple through all the pieces of Airflow -> models, API, UI etc.
>>>>> ...
>>>>>>>>> However - we already have `bundle_name" and "bundle_version" in the
>>>>>> Dag
>>>>>>>>> model. So I think when we add a separate table where we map the
>>>>> bundle
>>>>>>> to
>>>>>>>>> the team, the "ripple effect" will be almost 0. We do not want to
>>>>>> change
>>>>>>>>> primary key, we do not want to change UI in any way (except
>>>>> filtering
>>>>>> of
>>>>>>>>> DAGs available based on your team - but that will be handled in
>>>>> Auth
>>>>>>>>> Manager and will not impact UI in any way, I think that's a huge
>>>>>>>>> simplification of the implementation, and if we agree to it - i
>>>>> think
>>>>>> it
>>>>>>>>> should speed up the implementation significantly. There are only a
>>>>>>> limited
>>>>>>>>> number of times where you need to look up the team_id - so having
>>>>> the
>>>>>>>>> bundle -> team mapping in a separate table and having to look them
>>>>> up
>>>>>>>>> should not be a problem. And it has much less complexity and
>>>>>>>>> "ripple-effect" through the codebase (for example I could imagine
>>>>> 100s
>>>>>>> or
>>>>>>>>> thousands already written tests that would have to be adapted if we
>>>>>>> changed
>>>>>>>>> the primary key - where there will be pretty much zero impact on
>>>>>>> existing
>>>>>>>>> tests if we just add bundle -> team lookup table.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> One other thing I’d point out is that I think including executors
>>>>> per
>>>>>>>>>> team is a very easy win and quite possible without much work. I
>>>>>> already
>>>>>>>>>> have much of the code written. Executors are already aware of
>>>>> Teams
>>>>>>> that
>>>>>>>>>> own them (merged), I have a PR open to have configuration per team
>>>>>>> (with a
>>>>>>>>>> quite simple and isolated approach, I believe you approved Jarek).
>>>>>> The
>>>>>>> last
>>>>>>>>>> piece is updating the scheduling logic to route tasks from a
>>>>>> particular
>>>>>>>>>> Bundle to the correct executor, which shouldn’t be much work
>>>>> (though
>>>>>> it
>>>>>>>>>> would be easier if the Task models had a column for the team they
>>>>>>> belong
>>>>>>>>>> to, rather than having to look up the Dag and Bundle to get the
>>>>>> team) I
>>>>>>>>>> have a branch where I was experimenting with this logic already.
>>>>>>>>>> Any who, long story short, I don’t think we necessarily need to
>>>>>> remove
>>>>>>>>>> this piece from the project's scope if it is already partly done
>>>>> and
>>>>>>> not
>>>>>>>>>> too difficult.
>>>>>>>>>> 
>>>>>>>>> Yeah. I hear you here again. Certainly I would not want to just
>>>>>>>>> **remove** it from the code. And, yep I totally forgot we have it
>>>>> in.
>>>>>>> And
>>>>>>>>> if we can make it in, easily (which it seems we can) - we can also
>>>>>>> include
>>>>>>>>> it in the first iteration. What I wanted to avoid really (from the
>>>>>>> original
>>>>>>>>> design) - again trying to simplify it, limit the changes, and
>>>>> speed up
>>>>>>>>> implementation. And there is one "complexity" that I wanted to
>>>>> avoid
>>>>>>>>> specifically - having to have separate , additional configuration
>>>>> per
>>>>>>> team.
>>>>>>>>> Not only because it complicates already complex configuration
>>>>> handling
>>>>>>> (I
>>>>>>>>> know we have PR for that) but mostly because if it is not needed,
>>>>> we
>>>>>> can
>>>>>>>>> simplify documentation and explain to our users easier what they
>>>>> need
>>>>>>> to do
>>>>>>>>> to have their own multi-team setup. And I am quite open to keeping
>>>>>>>>> multiple-executors if we can avoid complicating configuration.
>>>>>>>>> 
>>>>>>>>> But I think some details of that and whether we really need
>>>>> separate
>>>>>>>>> configuration might also come as a result of updating the AIP - I
>>>>> am
>>>>>> not
>>>>>>>>> quite sure now if we need it, but we can discuss it when we
>>>>> iterate on
>>>>>>> the
>>>>>>>>> AIP.
>>>>>>>>> 
>>>>>>>>> J.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

My counter-points:


> 1. Managing a multi team deployment is not materially different from
> managing a deployment per team
>

It's a bit easier - especially when it comes to upgrades (especially in the
case we are targetting when we are not targetting multi-tenant, but several
relatively closely cooperating teams with different dependncy requiremens
and isolation need.

2. The database changes were quite wide-reaching
>

Yes. that is addressed.


> 3. I don’t believe the original AIP (again, I haven’t read the updated
> proposal or recent messages on the thread. yet) will meet what many users
> want out of a multiteam solution
>

I think we will only see when we try. A lot of people thing they would,
even if they are warned. I know at least one user (Wealthsimple) who
definitely want to use it and they got a very detailed explanation of the
idea and understand it well. So I am sure that **some** users would. But we
do not know how many.


> To expand on those points a bit more
>
> On 1. The only components that are shared are, I think, the scheduler and
> the API server, and it’s arguable if that is actually a good idea given
> those are likely to be the most performance sensitive components anyway.
>
> Additionally the fact that the scheduler is a shared component makes
> upgrading it almost a non starter as you would likely need buy-in, changes,
> and testing form ALL teams using it. I’d argue that this is a huge negative
> until we finish off the version indepence work of AIP-72.
>

Quite disagree here - especially that our target is that task-sdk is
supposed to provide all isolation that is needed. There should be 0 changes
in the dags needed to upgrade scheduler, api_server, triggerer - precisely
because we introduced backwards-compatible task-sdk.

On 3 my complaint is essentially that this doesn’t go nearly far enough. It
> doesn’t allow read only views to other teams dags. I don’t think it allows
> you to be in multiple teams at once. You can’t share a connection between
> teams but only allow certain specified dags to access it, but would have to
> either be globally usable, or duplicated-and-kept-in-sync between teams. In
> short I think it fall short of being useful..
>

Oh absolutely all that is possible (except sharing single connections
between multiple teams - which is a very niche use cases and duplication
here is perfectly ok as first approximation - and if we need more we can
add it later).

Auth manager RBAC and access is abstracted away, and the Keyclock Manager
implemented by Vincent allows to manage completely independent and separate
RBAC based on arguments and resources provided by Airflow. There is nothing
to prevent the user who configures KeyCloak RBAC to define it in the way:

if group a > allow to read a and write b
if group b > alllow to write b but not a

and any other combinations. KeyCloak implementation - pretty advanced
already - (and design of auth manager) completely abstracts away both
authentication and authorization to KeyCloak and KeyCloak has RBAC
management built in. Also any of the users can write their own - even
hard-coded authentication manager to do the same if they do not want to
have configurable KeyCloak. Even SimpleAuthManager could be hard-coded to
provide thiose features.


>
> So on the surface, I’m no more in favour of using dag bundle as a
> replacement for team id as I think most of the above points still stand.
>

We disagree here.

>
> My counter proposal: We do _nothing_ to core airflow. We work on improving
> the event-based trigger o fdags (write more triggers for read/check remote
> Assets etc) so that teams can have 100% isolated deployments but still
> trigger dags based on asset events from other teams.
>

That does not solve any of the other design goals - only allows to trigger
assets a bit more easily (but also it's not entirely solved by AIP-82
because it does not solve virtual assets - only ones that have defined
triggerer and "something" to listen on - which is way more complex than
just defining asset in a Dag and using it in another). I think we can't
compare AIP-82 to sharing virtual assets due to complexity of it. I
explained it in the doc.


I will now go and catch up with the long thread and updated proposal and
> come back.
>

Please. I hope the above explaination will help in better understanding of
the proposal, because I think you had some assumptions that do not hold any
more with the new proposal.

J.


>
> > On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Just to clarify the relation - I updated the AIP now to refer to AIP-82
> and
> > to explain relation between the "cross-team" and "cross-airflow" asset
> > triggering - this is what I added:
> >
> > Note that there is a relation between AIP-82 ("External Driven
> Scheduling")
> > and this part of the functionality. When you have multiple instances of
> > Airflow, you can use shared datasets - "Physical datasets" - that several
> > Airflow Instances can use - for example there could be an S3 object that
> is
> > produced by one airflow instance, and consumed by another. That requires
> > deferred trigger to monitor for such datasets, and appropriate
> permissions
> > to the external dataset, and you could achive similar result to
> cross-team
> > dataset triggering (but cross airflow). However the feature of sharing
> > datasets between the teams also works for virtual assets, that do not
> have
> > physically shared "objects" and trigger that is monitoring for changes in
> > such asset.
> >
> > J.
> >
> >
> > On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >>> From a quick glance, the updated AIP didn't seem to have any reference
> to
> >>> AIP-82, which surprised me, but will take a more detailed read through.
> >>
> >> Yep. It did not - because I did not think it was needed or even very
> >> important after the simplifications. AIP-82 has a different scope,
> really.
> >> It only helps when the Assets are "real" data files which we have
> physical
> >> triggers for, it's slightly related - sharing datasets between teams
> >> (including those that do not require physical files and triggers) is
> still
> >> possible in the design we have now, but it's not (and never was) the
> >> **only** reason for having multi-team. There always was (and still is)
> the
> >> possibility of having a common, distinct environments (i.e. dependencies
> >> and providers) per team, the possibility of having connections and
> >> variables that are only accessible to one team and not the other, and
> >> isolating workload execution (all that while allowing to manage multiple
> >> team and schedule things with single deployment). That did not change.
> What
> >> changed a lot is that it is now way simpler, something that we can
> >> implement without heavy changes to the codebase - and give it to our
> users,
> >> so that they can assess if this is something they need without too much
> >> risk and effort.
> >>
> >> This was - I believe the main concern, that the value we get from it is
> >> not dramatic, but the required changes are huge. This "redesign" changes
> >> the equation - the value is still unchanged, but the cost of
> implementing
> >> it and impact on the Airflow codebase is much smaller. I still have not
> >> heard back from Ash if my proposal responds to his original concern
> though,
> >> so I am mostly guessing (also based on the positive impact of others)
> that
> >> yes it does. But to be honest I am not sure and I would love to hear
> back,
> >> I decided to update the AIP to reflect it - regardless, because I think
> the
> >> simplification I proposed keeps the original goals, but is indeed way
> >> simpler.
> >>
> >>> This is a very difficult thread to catch up on.
> >>
> >> Valid point. Let me summarize what is the result:
> >>
> >> * I significantly simplified the implementation proposal comparing to
> the
> >> original version
> >> * main simplification is very limited impact on existing database -
> >> without "ripple effect" that would require us to change a lot of tables,
> >> including their primary keys, and heavily impact the UI
> >> * this is now more of an incremental change that can be implemented way
> >> faster and with far less risk
> >> * updated idea is based on leveraging bundles (already part of our data
> >> model) to map them (many-to-one) to a team - which requires to just
> extend
> >> the data model with bundle mapping and add team_id to connections and
> >> variables. Those are all needed DB changes.
> >>
> >> The AIP is updated - in a one single big change so It should be easy to
> >> compare the changes:
> >>
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
> >> -> I even named the version appropriately "Simplified multi-team AIP" -
> you
> >> can select and compare v.65 with v.66 to see the exact differences I
> >> proposed.
> >>
> >> I hope it will be helpful to catch up and for those who did not follow,
> to
> >> be able to make up their minds about it.
> >>
> >> J.
> >>
> >>
> >>
> >> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka
> <vi...@astronomer.io.invalid>
> >> wrote:
> >>
> >>> This is a very difficult thread to catch up on.
> >>> I will take a detailed look at the AIP update to try to figure out the
> >>> changes in the proposal.
> >>>
> >>> From a quick glance, the updated AIP didn't seem to have any reference
> to
> >>> AIP-82, which surprised me, but will take a more detailed read through.
> >>>
> >>> Vikram
> >>>
> >>>
> >>>
> >>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
> >>> gopidesupavan@gmail.com>
> >>> wrote:
> >>>
> >>>> Thanks Jarek, that's a great update on this AIP, now it's much more
> slim
> >>>> down.
> >>>>
> >>>> left a minor comment. :) Overall looking great.
> >>>>
> >>>> Pavan
> >>>>
> >>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
> >>> <j_scheffler@gmx.de.invalid
> >>>>>
> >>>> wrote:
> >>>>
> >>>>> Thanks for the rework/update of the AIP-72!
> >>>>>
> >>>>> Just a few small comments but overall I like it as it is much leaner
> >>>>> than originally planned and is in a level of complexity that it
> really
> >>>>> seems to be a benefit to close the gap as described.
> >>>>>
> >>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
> >>>>>> I updated the AIP - including architecture images and reviewed it
> >>>> (again)
> >>>>>> and corrected any ambiguities and places where it needed to be
> >>> changed.
> >>>>>>
> >>>>>> I think the current state
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >>>>>> - nicely describes the proposal.
> >>>>>>
> >>>>>> Comparing to the previous one:
> >>>>>>
> >>>>>> 1. The DB changes are far less intrusive - no ripple effect on
> >>> Airflow
> >>>>>> 2. There is no need to merge configurations and provide different
> >>> set
> >>>> of
> >>>>>> configs per team - we can add it later but I do not see why we need
> >>> it
> >>>> in
> >>>>>> this simplified version
> >>>>>> 3. We can still configure a different set of executors per team -
> >>> that
> >>>> is
> >>>>>> already implemented (we just need to wire it to the bundle -> team
> >>>>> mapping).
> >>>>>>
> >>>>>> I think it will be way simpler and faster to implement this way and
> >>> it
> >>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can give
> >>> our
> >>>>>> users so that they can provide feedback.
> >>>>>>
> >>>>>> J.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
> >>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> I like this iteration a bit more now for sure, thanks for being
> >>>>> receptive
> >>>>>>>> to feedback! :)
> >>>>>>>>
> >>>>>>>
> >>>>>>>> This now becomes quite close to what was proposing before, we now
> >>>> again
> >>>>>>>> have a team ID (which I think is really needed here, glad to see
> >>> it
> >>>>> back)
> >>>>>>>> and it will be used for auth management, configuration
> >>> specification,
> >>>>> etc
> >>>>>>>> but will be carried by Bundle instead of the dag model. Which as
> >>> you
> >>>>> say
> >>>>>>>> “For that we will need to make sure that both api-server,
> >>> scheduler
> >>>> and
> >>>>>>>> triggerer have access to the "bundle definition" (to perform the
> >>>>> mapping)"
> >>>>>>>> which honestly doesn’t feel too much different from the original
> >>>>> proposal
> >>>>>>>> we had last week of adding it to Dag table and ensuring it’s
> >>>> available
> >>>>>>>> everywhere. but either way I’m happy to meet in the middle and
> >>> keep
> >>>> it
> >>>>> on
> >>>>>>>> Bundle if everyone else feels that’s a more suitable location.
> >>>>>>>>
> >>>>>>> I think the big difference is the "ripple effect" that was
> >>> discussed
> >>>> in
> >>>>>>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
> >>>> (and I
> >>>>>>> believe - correct me if I am wrong Ash - important trigger for the
> >>>>>>> discussion) so far what we wanted is to extend the primary key and
> >>> it
> >>>>> would
> >>>>>>> ripple through all the pieces of Airflow -> models, API, UI etc.
> >>> ...
> >>>>>>> However - we already have `bundle_name" and "bundle_version" in the
> >>>> Dag
> >>>>>>> model. So I think when we add a separate table where we map the
> >>> bundle
> >>>>> to
> >>>>>>> the team, the "ripple effect" will be almost 0. We do not want to
> >>>> change
> >>>>>>> primary key, we do not want to change UI in any way (except
> >>> filtering
> >>>> of
> >>>>>>> DAGs available based on your team - but that will be handled in
> >>> Auth
> >>>>>>> Manager and will not impact UI in any way, I think that's a huge
> >>>>>>> simplification of the implementation, and if we agree to it - i
> >>> think
> >>>> it
> >>>>>>> should speed up the implementation significantly. There are only a
> >>>>> limited
> >>>>>>> number of times where you need to look up the team_id - so having
> >>> the
> >>>>>>> bundle -> team mapping in a separate table and having to look them
> >>> up
> >>>>>>> should not be a problem. And it has much less complexity and
> >>>>>>> "ripple-effect" through the codebase (for example I could imagine
> >>> 100s
> >>>>> or
> >>>>>>> thousands already written tests that would have to be adapted if we
> >>>>> changed
> >>>>>>> the primary key - where there will be pretty much zero impact on
> >>>>> existing
> >>>>>>> tests if we just add bundle -> team lookup table.
> >>>>>>>
> >>>>>>>
> >>>>>>>> One other thing I’d point out is that I think including executors
> >>> per
> >>>>>>>> team is a very easy win and quite possible without much work. I
> >>>> already
> >>>>>>>> have much of the code written. Executors are already aware of
> >>> Teams
> >>>>> that
> >>>>>>>> own them (merged), I have a PR open to have configuration per team
> >>>>> (with a
> >>>>>>>> quite simple and isolated approach, I believe you approved Jarek).
> >>>> The
> >>>>> last
> >>>>>>>> piece is updating the scheduling logic to route tasks from a
> >>>> particular
> >>>>>>>> Bundle to the correct executor, which shouldn’t be much work
> >>> (though
> >>>> it
> >>>>>>>> would be easier if the Task models had a column for the team they
> >>>>> belong
> >>>>>>>> to, rather than having to look up the Dag and Bundle to get the
> >>>> team) I
> >>>>>>>> have a branch where I was experimenting with this logic already.
> >>>>>>>> Any who, long story short, I don’t think we necessarily need to
> >>>> remove
> >>>>>>>> this piece from the project's scope if it is already partly done
> >>> and
> >>>>> not
> >>>>>>>> too difficult.
> >>>>>>>>
> >>>>>>> Yeah. I hear you here again. Certainly I would not want to just
> >>>>>>> **remove** it from the code. And, yep I totally forgot we have it
> >>> in.
> >>>>> And
> >>>>>>> if we can make it in, easily (which it seems we can) - we can also
> >>>>> include
> >>>>>>> it in the first iteration. What I wanted to avoid really (from the
> >>>>> original
> >>>>>>> design) - again trying to simplify it, limit the changes, and
> >>> speed up
> >>>>>>> implementation. And there is one "complexity" that I wanted to
> >>> avoid
> >>>>>>> specifically - having to have separate , additional configuration
> >>> per
> >>>>> team.
> >>>>>>> Not only because it complicates already complex configuration
> >>> handling
> >>>>> (I
> >>>>>>> know we have PR for that) but mostly because if it is not needed,
> >>> we
> >>>> can
> >>>>>>> simplify documentation and explain to our users easier what they
> >>> need
> >>>>> to do
> >>>>>>> to have their own multi-team setup. And I am quite open to keeping
> >>>>>>> multiple-executors if we can avoid complicating configuration.
> >>>>>>>
> >>>>>>> But I think some details of that and whether we really need
> >>> separate
> >>>>>>> configuration might also come as a result of updating the AIP - I
> >>> am
> >>>> not
> >>>>>>> quite sure now if we need it, but we can discuss it when we
> >>> iterate on
> >>>>> the
> >>>>>>> AIP.
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Ash Berlin-Taylor <as...@apache.org>]

Sorry, I was not on emails last week, I’ll catch up, read the updated proposal and come back to this today or tomorrow.

A few notes from skimming the thread.

Yes, the original trigger for this was the discussion about the DB changes (was it in the list or in the dev call? Whichever it was) and my just trusting my “spider-sense” so to speak and looking at it with fresh eyes.

I’m still not convinced that we even need the approach of using bundles in place of teams — my original idea is basically:

1. Managing a multi team deployment is not materially different from managing a deployment per team
2. The database changes were quite wide-reaching
3. I don’t believe the original AIP (again, I haven’t read the updated proposal or recent messages on the thread. yet) will meet what many users want out of a multiteam solution

To expand on those points a bit more

On 1. The only components that are shared are, I think, the scheduler and the API server, and it’s arguable if that is actually a good idea given those are likely to be the most performance sensitive components anyway.

Additionally the fact that the scheduler is a shared component makes upgrading it almost a non starter as you would likely need buy-in, changes, and testing form ALL teams using it. I’d argue that this is a huge negative until we finish off the version indepence work of AIP-72.

On 3 my complaint is essentially that this doesn’t go nearly far enough. It doesn’t allow read only views to other teams dags. I don’t think it allows you to be in multiple teams at once. You can’t share a connection between teams but only allow certain specified dags to access it, but would have to either be globally usable, or duplicated-and-kept-in-sync between teams. In short I think it fall short of being useful..

So on the surface, I’m no more in favour of using dag bundle as a replacement for team id as I think most of the above points still stand.

My counter proposal: We do _nothing_ to core airflow. We work on improving the event-based trigger o fdags (write more triggers for read/check remote Assets etc) so that teams can have 100% isolated deployments but still trigger dags based on asset events from other teams.   

I will now go and catch up with the long thread and updated proposal and come back.

> On 23 Jun 2025, at 05:54, Jarek Potiuk <ja...@potiuk.com> wrote:
> 
> Just to clarify the relation - I updated the AIP now to refer to AIP-82 and
> to explain relation between the "cross-team" and "cross-airflow" asset
> triggering - this is what I added:
> 
> Note that there is a relation between AIP-82 ("External Driven Scheduling")
> and this part of the functionality. When you have multiple instances of
> Airflow, you can use shared datasets - "Physical datasets" - that several
> Airflow Instances can use - for example there could be an S3 object that is
> produced by one airflow instance, and consumed by another. That requires
> deferred trigger to monitor for such datasets, and appropriate permissions
> to the external dataset, and you could achive similar result to cross-team
> dataset triggering (but cross airflow). However the feature of sharing
> datasets between the teams also works for virtual assets, that do not have
> physically shared "objects" and trigger that is monitoring for changes in
> such asset.
> 
> J.
> 
> 
> On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> 
>>> From a quick glance, the updated AIP didn't seem to have any reference to
>>> AIP-82, which surprised me, but will take a more detailed read through.
>> 
>> Yep. It did not - because I did not think it was needed or even very
>> important after the simplifications. AIP-82 has a different scope, really.
>> It only helps when the Assets are "real" data files which we have physical
>> triggers for, it's slightly related - sharing datasets between teams
>> (including those that do not require physical files and triggers) is still
>> possible in the design we have now, but it's not (and never was) the
>> **only** reason for having multi-team. There always was (and still is) the
>> possibility of having a common, distinct environments (i.e. dependencies
>> and providers) per team, the possibility of having connections and
>> variables that are only accessible to one team and not the other, and
>> isolating workload execution (all that while allowing to manage multiple
>> team and schedule things with single deployment). That did not change. What
>> changed a lot is that it is now way simpler, something that we can
>> implement without heavy changes to the codebase - and give it to our users,
>> so that they can assess if this is something they need without too much
>> risk and effort.
>> 
>> This was - I believe the main concern, that the value we get from it is
>> not dramatic, but the required changes are huge. This "redesign" changes
>> the equation - the value is still unchanged, but the cost of implementing
>> it and impact on the Airflow codebase is much smaller. I still have not
>> heard back from Ash if my proposal responds to his original concern though,
>> so I am mostly guessing (also based on the positive impact of others) that
>> yes it does. But to be honest I am not sure and I would love to hear back,
>> I decided to update the AIP to reflect it - regardless, because I think the
>> simplification I proposed keeps the original goals, but is indeed way
>> simpler.
>> 
>>> This is a very difficult thread to catch up on.
>> 
>> Valid point. Let me summarize what is the result:
>> 
>> * I significantly simplified the implementation proposal comparing to the
>> original version
>> * main simplification is very limited impact on existing database -
>> without "ripple effect" that would require us to change a lot of tables,
>> including their primary keys, and heavily impact the UI
>> * this is now more of an incremental change that can be implemented way
>> faster and with far less risk
>> * updated idea is based on leveraging bundles (already part of our data
>> model) to map them (many-to-one) to a team - which requires to just extend
>> the data model with bundle mapping and add team_id to connections and
>> variables. Those are all needed DB changes.
>> 
>> The AIP is updated - in a one single big change so It should be easy to
>> compare the changes:
>> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
>> -> I even named the version appropriately "Simplified multi-team AIP" - you
>> can select and compare v.65 with v.66 to see the exact differences I
>> proposed.
>> 
>> I hope it will be helpful to catch up and for those who did not follow, to
>> be able to make up their minds about it.
>> 
>> J.
>> 
>> 
>> 
>> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka <vi...@astronomer.io.invalid>
>> wrote:
>> 
>>> This is a very difficult thread to catch up on.
>>> I will take a detailed look at the AIP update to try to figure out the
>>> changes in the proposal.
>>> 
>>> From a quick glance, the updated AIP didn't seem to have any reference to
>>> AIP-82, which surprised me, but will take a more detailed read through.
>>> 
>>> Vikram
>>> 
>>> 
>>> 
>>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>>> gopidesupavan@gmail.com>
>>> wrote:
>>> 
>>>> Thanks Jarek, that's a great update on this AIP, now it's much more slim
>>>> down.
>>>> 
>>>> left a minor comment. :) Overall looking great.
>>>> 
>>>> Pavan
>>>> 
>>>> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>>> <j_scheffler@gmx.de.invalid
>>>>> 
>>>> wrote:
>>>> 
>>>>> Thanks for the rework/update of the AIP-72!
>>>>> 
>>>>> Just a few small comments but overall I like it as it is much leaner
>>>>> than originally planned and is in a level of complexity that it really
>>>>> seems to be a benefit to close the gap as described.
>>>>> 
>>>>> On 21.06.25 14:52, Jarek Potiuk wrote:
>>>>>> I updated the AIP - including architecture images and reviewed it
>>>> (again)
>>>>>> and corrected any ambiguities and places where it needed to be
>>> changed.
>>>>>> 
>>>>>> I think the current state
>>>>>> 
>>>>> 
>>>> 
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>>>>>> - nicely describes the proposal.
>>>>>> 
>>>>>> Comparing to the previous one:
>>>>>> 
>>>>>> 1. The DB changes are far less intrusive - no ripple effect on
>>> Airflow
>>>>>> 2. There is no need to merge configurations and provide different
>>> set
>>>> of
>>>>>> configs per team - we can add it later but I do not see why we need
>>> it
>>>> in
>>>>>> this simplified version
>>>>>> 3. We can still configure a different set of executors per team -
>>> that
>>>> is
>>>>>> already implemented (we just need to wire it to the bundle -> team
>>>>> mapping).
>>>>>> 
>>>>>> I think it will be way simpler and faster to implement this way and
>>> it
>>>>>> should serve as MVMT -> Minimum Viable Multi Team that we can give
>>> our
>>>>>> users so that they can provide feedback.
>>>>>> 
>>>>>> J.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> I like this iteration a bit more now for sure, thanks for being
>>>>> receptive
>>>>>>>> to feedback! :)
>>>>>>>> 
>>>>>>> 
>>>>>>>> This now becomes quite close to what was proposing before, we now
>>>> again
>>>>>>>> have a team ID (which I think is really needed here, glad to see
>>> it
>>>>> back)
>>>>>>>> and it will be used for auth management, configuration
>>> specification,
>>>>> etc
>>>>>>>> but will be carried by Bundle instead of the dag model. Which as
>>> you
>>>>> say
>>>>>>>> “For that we will need to make sure that both api-server,
>>> scheduler
>>>> and
>>>>>>>> triggerer have access to the "bundle definition" (to perform the
>>>>> mapping)"
>>>>>>>> which honestly doesn’t feel too much different from the original
>>>>> proposal
>>>>>>>> we had last week of adding it to Dag table and ensuring it’s
>>>> available
>>>>>>>> everywhere. but either way I’m happy to meet in the middle and
>>> keep
>>>> it
>>>>> on
>>>>>>>> Bundle if everyone else feels that’s a more suitable location.
>>>>>>>> 
>>>>>>> I think the big difference is the "ripple effect" that was
>>> discussed
>>>> in
>>>>>>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>>>> (and I
>>>>>>> believe - correct me if I am wrong Ash - important trigger for the
>>>>>>> discussion) so far what we wanted is to extend the primary key and
>>> it
>>>>> would
>>>>>>> ripple through all the pieces of Airflow -> models, API, UI etc.
>>> ...
>>>>>>> However - we already have `bundle_name" and "bundle_version" in the
>>>> Dag
>>>>>>> model. So I think when we add a separate table where we map the
>>> bundle
>>>>> to
>>>>>>> the team, the "ripple effect" will be almost 0. We do not want to
>>>> change
>>>>>>> primary key, we do not want to change UI in any way (except
>>> filtering
>>>> of
>>>>>>> DAGs available based on your team - but that will be handled in
>>> Auth
>>>>>>> Manager and will not impact UI in any way, I think that's a huge
>>>>>>> simplification of the implementation, and if we agree to it - i
>>> think
>>>> it
>>>>>>> should speed up the implementation significantly. There are only a
>>>>> limited
>>>>>>> number of times where you need to look up the team_id - so having
>>> the
>>>>>>> bundle -> team mapping in a separate table and having to look them
>>> up
>>>>>>> should not be a problem. And it has much less complexity and
>>>>>>> "ripple-effect" through the codebase (for example I could imagine
>>> 100s
>>>>> or
>>>>>>> thousands already written tests that would have to be adapted if we
>>>>> changed
>>>>>>> the primary key - where there will be pretty much zero impact on
>>>>> existing
>>>>>>> tests if we just add bundle -> team lookup table.
>>>>>>> 
>>>>>>> 
>>>>>>>> One other thing I’d point out is that I think including executors
>>> per
>>>>>>>> team is a very easy win and quite possible without much work. I
>>>> already
>>>>>>>> have much of the code written. Executors are already aware of
>>> Teams
>>>>> that
>>>>>>>> own them (merged), I have a PR open to have configuration per team
>>>>> (with a
>>>>>>>> quite simple and isolated approach, I believe you approved Jarek).
>>>> The
>>>>> last
>>>>>>>> piece is updating the scheduling logic to route tasks from a
>>>> particular
>>>>>>>> Bundle to the correct executor, which shouldn’t be much work
>>> (though
>>>> it
>>>>>>>> would be easier if the Task models had a column for the team they
>>>>> belong
>>>>>>>> to, rather than having to look up the Dag and Bundle to get the
>>>> team) I
>>>>>>>> have a branch where I was experimenting with this logic already.
>>>>>>>> Any who, long story short, I don’t think we necessarily need to
>>>> remove
>>>>>>>> this piece from the project's scope if it is already partly done
>>> and
>>>>> not
>>>>>>>> too difficult.
>>>>>>>> 
>>>>>>> Yeah. I hear you here again. Certainly I would not want to just
>>>>>>> **remove** it from the code. And, yep I totally forgot we have it
>>> in.
>>>>> And
>>>>>>> if we can make it in, easily (which it seems we can) - we can also
>>>>> include
>>>>>>> it in the first iteration. What I wanted to avoid really (from the
>>>>> original
>>>>>>> design) - again trying to simplify it, limit the changes, and
>>> speed up
>>>>>>> implementation. And there is one "complexity" that I wanted to
>>> avoid
>>>>>>> specifically - having to have separate , additional configuration
>>> per
>>>>> team.
>>>>>>> Not only because it complicates already complex configuration
>>> handling
>>>>> (I
>>>>>>> know we have PR for that) but mostly because if it is not needed,
>>> we
>>>> can
>>>>>>> simplify documentation and explain to our users easier what they
>>> need
>>>>> to do
>>>>>>> to have their own multi-team setup. And I am quite open to keeping
>>>>>>> multiple-executors if we can avoid complicating configuration.
>>>>>>> 
>>>>>>> But I think some details of that and whether we really need
>>> separate
>>>>>>> configuration might also come as a result of updating the AIP - I
>>> am
>>>> not
>>>>>>> quite sure now if we need it, but we can discuss it when we
>>> iterate on
>>>>> the
>>>>>>> AIP.
>>>>>>> 
>>>>>>> J.
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>> 
>>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

Just to clarify the relation - I updated the AIP now to refer to AIP-82 and
to explain relation between the "cross-team" and "cross-airflow" asset
triggering - this is what I added:

Note that there is a relation between AIP-82 ("External Driven Scheduling")
and this part of the functionality. When you have multiple instances of
Airflow, you can use shared datasets - "Physical datasets" - that several
Airflow Instances can use - for example there could be an S3 object that is
produced by one airflow instance, and consumed by another. That requires
deferred trigger to monitor for such datasets, and appropriate permissions
to the external dataset, and you could achive similar result to cross-team
dataset triggering (but cross airflow). However the feature of sharing
datasets between the teams also works for virtual assets, that do not have
physically shared "objects" and trigger that is monitoring for changes in
such asset.

J.


On Mon, Jun 23, 2025 at 6:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> > From a quick glance, the updated AIP didn't seem to have any reference to
> > AIP-82, which surprised me, but will take a more detailed read through.
>
> Yep. It did not - because I did not think it was needed or even very
> important after the simplifications. AIP-82 has a different scope, really.
> It only helps when the Assets are "real" data files which we have physical
> triggers for, it's slightly related - sharing datasets between teams
> (including those that do not require physical files and triggers) is still
> possible in the design we have now, but it's not (and never was) the
> **only** reason for having multi-team. There always was (and still is) the
> possibility of having a common, distinct environments (i.e. dependencies
> and providers) per team, the possibility of having connections and
> variables that are only accessible to one team and not the other, and
> isolating workload execution (all that while allowing to manage multiple
> team and schedule things with single deployment). That did not change. What
> changed a lot is that it is now way simpler, something that we can
> implement without heavy changes to the codebase - and give it to our users,
> so that they can assess if this is something they need without too much
> risk and effort.
>
> This was - I believe the main concern, that the value we get from it is
> not dramatic, but the required changes are huge. This "redesign" changes
> the equation - the value is still unchanged, but the cost of implementing
> it and impact on the Airflow codebase is much smaller. I still have not
> heard back from Ash if my proposal responds to his original concern though,
> so I am mostly guessing (also based on the positive impact of others) that
> yes it does. But to be honest I am not sure and I would love to hear back,
> I decided to update the AIP to reflect it - regardless, because I think the
> simplification I proposed keeps the original goals, but is indeed way
> simpler.
>
> > This is a very difficult thread to catch up on.
>
> Valid point. Let me summarize what is the result:
>
> * I significantly simplified the implementation proposal comparing to the
> original version
> * main simplification is very limited impact on existing database -
> without "ripple effect" that would require us to change a lot of tables,
> including their primary keys, and heavily impact the UI
> * this is now more of an incremental change that can be implemented way
> faster and with far less risk
> * updated idea is based on leveraging bundles (already part of our data
> model) to map them (many-to-one) to a team - which requires to just extend
> the data model with bundle mapping and add team_id to connections and
> variables. Those are all needed DB changes.
>
> The AIP is updated - in a one single big change so It should be easy to
> compare the changes:
> https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
> -> I even named the version appropriately "Simplified multi-team AIP" - you
> can select and compare v.65 with v.66 to see the exact differences I
> proposed.
>
> I hope it will be helpful to catch up and for those who did not follow, to
> be able to make up their minds about it.
>
> J.
>
>
>
> On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka <vi...@astronomer.io.invalid>
> wrote:
>
>> This is a very difficult thread to catch up on.
>> I will take a detailed look at the AIP update to try to figure out the
>> changes in the proposal.
>>
>> From a quick glance, the updated AIP didn't seem to have any reference to
>> AIP-82, which surprised me, but will take a more detailed read through.
>>
>> Vikram
>>
>>
>>
>> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
>> gopidesupavan@gmail.com>
>> wrote:
>>
>> > Thanks Jarek, that's a great update on this AIP, now it's much more slim
>> > down.
>> >
>> > left a minor comment. :) Overall looking great.
>> >
>> > Pavan
>> >
>> > On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
>> <j_scheffler@gmx.de.invalid
>> > >
>> > wrote:
>> >
>> > > Thanks for the rework/update of the AIP-72!
>> > >
>> > > Just a few small comments but overall I like it as it is much leaner
>> > > than originally planned and is in a level of complexity that it really
>> > > seems to be a benefit to close the gap as described.
>> > >
>> > > On 21.06.25 14:52, Jarek Potiuk wrote:
>> > > > I updated the AIP - including architecture images and reviewed it
>> > (again)
>> > > > and corrected any ambiguities and places where it needed to be
>> changed.
>> > > >
>> > > > I think the current state
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>> > > > - nicely describes the proposal.
>> > > >
>> > > > Comparing to the previous one:
>> > > >
>> > > > 1. The DB changes are far less intrusive - no ripple effect on
>> Airflow
>> > > > 2. There is no need to merge configurations and provide different
>> set
>> > of
>> > > > configs per team - we can add it later but I do not see why we need
>> it
>> > in
>> > > > this simplified version
>> > > > 3. We can still configure a different set of executors per team -
>> that
>> > is
>> > > > already implemented (we just need to wire it to the bundle -> team
>> > > mapping).
>> > > >
>> > > > I think it will be way simpler and faster to implement this way and
>> it
>> > > > should serve as MVMT -> Minimum Viable Multi Team that we can give
>> our
>> > > > users so that they can provide feedback.
>> > > >
>> > > > J.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> > > >
>> > > >>
>> > > >>
>> > > >>> I like this iteration a bit more now for sure, thanks for being
>> > > receptive
>> > > >>> to feedback! :)
>> > > >>>
>> > > >>
>> > > >>> This now becomes quite close to what was proposing before, we now
>> > again
>> > > >>> have a team ID (which I think is really needed here, glad to see
>> it
>> > > back)
>> > > >>> and it will be used for auth management, configuration
>> specification,
>> > > etc
>> > > >>> but will be carried by Bundle instead of the dag model. Which as
>> you
>> > > say
>> > > >>> “For that we will need to make sure that both api-server,
>> scheduler
>> > and
>> > > >>> triggerer have access to the "bundle definition" (to perform the
>> > > mapping)"
>> > > >>> which honestly doesn’t feel too much different from the original
>> > > proposal
>> > > >>> we had last week of adding it to Dag table and ensuring it’s
>> > available
>> > > >>> everywhere. but either way I’m happy to meet in the middle and
>> keep
>> > it
>> > > on
>> > > >>> Bundle if everyone else feels that’s a more suitable location.
>> > > >>>
>> > > >> I think the big difference is the "ripple effect" that was
>> discussed
>> > in
>> > > >> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>> > (and I
>> > > >> believe - correct me if I am wrong Ash - important trigger for the
>> > > >> discussion) so far what we wanted is to extend the primary key and
>> it
>> > > would
>> > > >> ripple through all the pieces of Airflow -> models, API, UI etc.
>> ...
>> > > >> However - we already have `bundle_name" and "bundle_version" in the
>> > Dag
>> > > >> model. So I think when we add a separate table where we map the
>> bundle
>> > > to
>> > > >> the team, the "ripple effect" will be almost 0. We do not want to
>> > change
>> > > >> primary key, we do not want to change UI in any way (except
>> filtering
>> > of
>> > > >> DAGs available based on your team - but that will be handled in
>> Auth
>> > > >> Manager and will not impact UI in any way, I think that's a huge
>> > > >> simplification of the implementation, and if we agree to it - i
>> think
>> > it
>> > > >> should speed up the implementation significantly. There are only a
>> > > limited
>> > > >> number of times where you need to look up the team_id - so having
>> the
>> > > >> bundle -> team mapping in a separate table and having to look them
>> up
>> > > >> should not be a problem. And it has much less complexity and
>> > > >> "ripple-effect" through the codebase (for example I could imagine
>> 100s
>> > > or
>> > > >> thousands already written tests that would have to be adapted if we
>> > > changed
>> > > >> the primary key - where there will be pretty much zero impact on
>> > > existing
>> > > >> tests if we just add bundle -> team lookup table.
>> > > >>
>> > > >>
>> > > >>> One other thing I’d point out is that I think including executors
>> per
>> > > >>> team is a very easy win and quite possible without much work. I
>> > already
>> > > >>> have much of the code written. Executors are already aware of
>> Teams
>> > > that
>> > > >>> own them (merged), I have a PR open to have configuration per team
>> > > (with a
>> > > >>> quite simple and isolated approach, I believe you approved Jarek).
>> > The
>> > > last
>> > > >>> piece is updating the scheduling logic to route tasks from a
>> > particular
>> > > >>> Bundle to the correct executor, which shouldn’t be much work
>> (though
>> > it
>> > > >>> would be easier if the Task models had a column for the team they
>> > > belong
>> > > >>> to, rather than having to look up the Dag and Bundle to get the
>> > team) I
>> > > >>> have a branch where I was experimenting with this logic already.
>> > > >>> Any who, long story short, I don’t think we necessarily need to
>> > remove
>> > > >>> this piece from the project's scope if it is already partly done
>> and
>> > > not
>> > > >>> too difficult.
>> > > >>>
>> > > >> Yeah. I hear you here again. Certainly I would not want to just
>> > > >> **remove** it from the code. And, yep I totally forgot we have it
>> in.
>> > > And
>> > > >> if we can make it in, easily (which it seems we can) - we can also
>> > > include
>> > > >> it in the first iteration. What I wanted to avoid really (from the
>> > > original
>> > > >> design) - again trying to simplify it, limit the changes, and
>> speed up
>> > > >> implementation. And there is one "complexity" that I wanted to
>> avoid
>> > > >> specifically - having to have separate , additional configuration
>> per
>> > > team.
>> > > >> Not only because it complicates already complex configuration
>> handling
>> > > (I
>> > > >> know we have PR for that) but mostly because if it is not needed,
>> we
>> > can
>> > > >> simplify documentation and explain to our users easier what they
>> need
>> > > to do
>> > > >> to have their own multi-team setup. And I am quite open to keeping
>> > > >> multiple-executors if we can avoid complicating configuration.
>> > > >>
>> > > >> But I think some details of that and whether we really need
>> separate
>> > > >> configuration might also come as a result of updating the AIP - I
>> am
>> > not
>> > > >> quite sure now if we need it, but we can discuss it when we
>> iterate on
>> > > the
>> > > >> AIP.
>> > > >>
>> > > >> J.
>> > > >>
>> > > >>
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>> > >
>> > >
>> >
>>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

> From a quick glance, the updated AIP didn't seem to have any reference to
> AIP-82, which surprised me, but will take a more detailed read through.

Yep. It did not - because I did not think it was needed or even very
important after the simplifications. AIP-82 has a different scope, really.
It only helps when the Assets are "real" data files which we have physical
triggers for, it's slightly related - sharing datasets between teams
(including those that do not require physical files and triggers) is still
possible in the design we have now, but it's not (and never was) the
**only** reason for having multi-team. There always was (and still is) the
possibility of having a common, distinct environments (i.e. dependencies
and providers) per team, the possibility of having connections and
variables that are only accessible to one team and not the other, and
isolating workload execution (all that while allowing to manage multiple
team and schedule things with single deployment). That did not change. What
changed a lot is that it is now way simpler, something that we can
implement without heavy changes to the codebase - and give it to our users,
so that they can assess if this is something they need without too much
risk and effort.

This was - I believe the main concern, that the value we get from it is not
dramatic, but the required changes are huge. This "redesign" changes the
equation - the value is still unchanged, but the cost of implementing it
and impact on the Airflow codebase is much smaller. I still have not heard
back from Ash if my proposal responds to his original concern though, so I
am mostly guessing (also based on the positive impact of others) that yes
it does. But to be honest I am not sure and I would love to hear back, I
decided to update the AIP to reflect it - regardless, because I think the
simplification I proposed keeps the original goals, but is indeed way
simpler.

> This is a very difficult thread to catch up on.

Valid point. Let me summarize what is the result:

* I significantly simplified the implementation proposal comparing to the
original version
* main simplification is very limited impact on existing database - without
"ripple effect" that would require us to change a lot of tables, including
their primary keys, and heavily impact the UI
* this is now more of an incremental change that can be implemented way
faster and with far less risk
* updated idea is based on leveraging bundles (already part of our data
model) to map them (many-to-one) to a team - which requires to just extend
the data model with bundle mapping and add team_id to connections and
variables. Those are all needed DB changes.

The AIP is updated - in a one single big change so It should be easy to
compare the changes:
https://cwiki.apache.org/confluence/pages/viewpreviousversions.action?pageId=294816378
-> I even named the version appropriately "Simplified multi-team AIP" - you
can select and compare v.65 with v.66 to see the exact differences I
proposed.

I hope it will be helpful to catch up and for those who did not follow, to
be able to make up their minds about it.

J.



On Mon, Jun 23, 2025 at 4:35 AM Vikram Koka <vi...@astronomer.io.invalid>
wrote:

> This is a very difficult thread to catch up on.
> I will take a detailed look at the AIP update to try to figure out the
> changes in the proposal.
>
> From a quick glance, the updated AIP didn't seem to have any reference to
> AIP-82, which surprised me, but will take a more detailed read through.
>
> Vikram
>
>
>
> On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <
> gopidesupavan@gmail.com>
> wrote:
>
> > Thanks Jarek, that's a great update on this AIP, now it's much more slim
> > down.
> >
> > left a minor comment. :) Overall looking great.
> >
> > Pavan
> >
> > On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler
> <j_scheffler@gmx.de.invalid
> > >
> > wrote:
> >
> > > Thanks for the rework/update of the AIP-72!
> > >
> > > Just a few small comments but overall I like it as it is much leaner
> > > than originally planned and is in a level of complexity that it really
> > > seems to be a benefit to close the gap as described.
> > >
> > > On 21.06.25 14:52, Jarek Potiuk wrote:
> > > > I updated the AIP - including architecture images and reviewed it
> > (again)
> > > > and corrected any ambiguities and places where it needed to be
> changed.
> > > >
> > > > I think the current state
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > > > - nicely describes the proposal.
> > > >
> > > > Comparing to the previous one:
> > > >
> > > > 1. The DB changes are far less intrusive - no ripple effect on
> Airflow
> > > > 2. There is no need to merge configurations and provide different set
> > of
> > > > configs per team - we can add it later but I do not see why we need
> it
> > in
> > > > this simplified version
> > > > 3. We can still configure a different set of executors per team -
> that
> > is
> > > > already implemented (we just need to wire it to the bundle -> team
> > > mapping).
> > > >
> > > > I think it will be way simpler and faster to implement this way and
> it
> > > > should serve as MVMT -> Minimum Viable Multi Team that we can give
> our
> > > > users so that they can provide feedback.
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > >>
> > > >>
> > > >>> I like this iteration a bit more now for sure, thanks for being
> > > receptive
> > > >>> to feedback! :)
> > > >>>
> > > >>
> > > >>> This now becomes quite close to what was proposing before, we now
> > again
> > > >>> have a team ID (which I think is really needed here, glad to see it
> > > back)
> > > >>> and it will be used for auth management, configuration
> specification,
> > > etc
> > > >>> but will be carried by Bundle instead of the dag model. Which as
> you
> > > say
> > > >>> “For that we will need to make sure that both api-server, scheduler
> > and
> > > >>> triggerer have access to the "bundle definition" (to perform the
> > > mapping)"
> > > >>> which honestly doesn’t feel too much different from the original
> > > proposal
> > > >>> we had last week of adding it to Dag table and ensuring it’s
> > available
> > > >>> everywhere. but either way I’m happy to meet in the middle and keep
> > it
> > > on
> > > >>> Bundle if everyone else feels that’s a more suitable location.
> > > >>>
> > > >> I think the big difference is the "ripple effect" that was discussed
> > in
> > > >> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
> > (and I
> > > >> believe - correct me if I am wrong Ash - important trigger for the
> > > >> discussion) so far what we wanted is to extend the primary key and
> it
> > > would
> > > >> ripple through all the pieces of Airflow -> models, API, UI etc. ...
> > > >> However - we already have `bundle_name" and "bundle_version" in the
> > Dag
> > > >> model. So I think when we add a separate table where we map the
> bundle
> > > to
> > > >> the team, the "ripple effect" will be almost 0. We do not want to
> > change
> > > >> primary key, we do not want to change UI in any way (except
> filtering
> > of
> > > >> DAGs available based on your team - but that will be handled in Auth
> > > >> Manager and will not impact UI in any way, I think that's a huge
> > > >> simplification of the implementation, and if we agree to it - i
> think
> > it
> > > >> should speed up the implementation significantly. There are only a
> > > limited
> > > >> number of times where you need to look up the team_id - so having
> the
> > > >> bundle -> team mapping in a separate table and having to look them
> up
> > > >> should not be a problem. And it has much less complexity and
> > > >> "ripple-effect" through the codebase (for example I could imagine
> 100s
> > > or
> > > >> thousands already written tests that would have to be adapted if we
> > > changed
> > > >> the primary key - where there will be pretty much zero impact on
> > > existing
> > > >> tests if we just add bundle -> team lookup table.
> > > >>
> > > >>
> > > >>> One other thing I’d point out is that I think including executors
> per
> > > >>> team is a very easy win and quite possible without much work. I
> > already
> > > >>> have much of the code written. Executors are already aware of Teams
> > > that
> > > >>> own them (merged), I have a PR open to have configuration per team
> > > (with a
> > > >>> quite simple and isolated approach, I believe you approved Jarek).
> > The
> > > last
> > > >>> piece is updating the scheduling logic to route tasks from a
> > particular
> > > >>> Bundle to the correct executor, which shouldn’t be much work
> (though
> > it
> > > >>> would be easier if the Task models had a column for the team they
> > > belong
> > > >>> to, rather than having to look up the Dag and Bundle to get the
> > team) I
> > > >>> have a branch where I was experimenting with this logic already.
> > > >>> Any who, long story short, I don’t think we necessarily need to
> > remove
> > > >>> this piece from the project's scope if it is already partly done
> and
> > > not
> > > >>> too difficult.
> > > >>>
> > > >> Yeah. I hear you here again. Certainly I would not want to just
> > > >> **remove** it from the code. And, yep I totally forgot we have it
> in.
> > > And
> > > >> if we can make it in, easily (which it seems we can) - we can also
> > > include
> > > >> it in the first iteration. What I wanted to avoid really (from the
> > > original
> > > >> design) - again trying to simplify it, limit the changes, and speed
> up
> > > >> implementation. And there is one "complexity" that I wanted to avoid
> > > >> specifically - having to have separate , additional configuration
> per
> > > team.
> > > >> Not only because it complicates already complex configuration
> handling
> > > (I
> > > >> know we have PR for that) but mostly because if it is not needed, we
> > can
> > > >> simplify documentation and explain to our users easier what they
> need
> > > to do
> > > >> to have their own multi-team setup. And I am quite open to keeping
> > > >> multiple-executors if we can avoid complicating configuration.
> > > >>
> > > >> But I think some details of that and whether we really need separate
> > > >> configuration might also come as a result of updating the AIP - I am
> > not
> > > >> quite sure now if we need it, but we can discuss it when we iterate
> on
> > > the
> > > >> AIP.
> > > >>
> > > >> J.
> > > >>
> > > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Vikram Koka <vi...@astronomer.io.INVALID>]

This is a very difficult thread to catch up on.
I will take a detailed look at the AIP update to try to figure out the
changes in the proposal.

From a quick glance, the updated AIP didn't seem to have any reference to
AIP-82, which surprised me, but will take a more detailed read through.

Vikram



On Sun, Jun 22, 2025 at 1:44 AM Pavankumar Gopidesu <go...@gmail.com>
wrote:

> Thanks Jarek, that's a great update on this AIP, now it's much more slim
> down.
>
> left a minor comment. :) Overall looking great.
>
> Pavan
>
> On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler <j_scheffler@gmx.de.invalid
> >
> wrote:
>
> > Thanks for the rework/update of the AIP-72!
> >
> > Just a few small comments but overall I like it as it is much leaner
> > than originally planned and is in a level of complexity that it really
> > seems to be a benefit to close the gap as described.
> >
> > On 21.06.25 14:52, Jarek Potiuk wrote:
> > > I updated the AIP - including architecture images and reviewed it
> (again)
> > > and corrected any ambiguities and places where it needed to be changed.
> > >
> > > I think the current state
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > > - nicely describes the proposal.
> > >
> > > Comparing to the previous one:
> > >
> > > 1. The DB changes are far less intrusive - no ripple effect on Airflow
> > > 2. There is no need to merge configurations and provide different set
> of
> > > configs per team - we can add it later but I do not see why we need it
> in
> > > this simplified version
> > > 3. We can still configure a different set of executors per team - that
> is
> > > already implemented (we just need to wire it to the bundle -> team
> > mapping).
> > >
> > > I think it will be way simpler and faster to implement this way and it
> > > should serve as MVMT -> Minimum Viable Multi Team that we can give our
> > > users so that they can provide feedback.
> > >
> > > J.
> > >
> > >
> > >
> > >
> > > On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > >>
> > >>
> > >>> I like this iteration a bit more now for sure, thanks for being
> > receptive
> > >>> to feedback! :)
> > >>>
> > >>
> > >>> This now becomes quite close to what was proposing before, we now
> again
> > >>> have a team ID (which I think is really needed here, glad to see it
> > back)
> > >>> and it will be used for auth management, configuration specification,
> > etc
> > >>> but will be carried by Bundle instead of the dag model. Which as you
> > say
> > >>> “For that we will need to make sure that both api-server, scheduler
> and
> > >>> triggerer have access to the "bundle definition" (to perform the
> > mapping)"
> > >>> which honestly doesn’t feel too much different from the original
> > proposal
> > >>> we had last week of adding it to Dag table and ensuring it’s
> available
> > >>> everywhere. but either way I’m happy to meet in the middle and keep
> it
> > on
> > >>> Bundle if everyone else feels that’s a more suitable location.
> > >>>
> > >> I think the big difference is the "ripple effect" that was discussed
> in
> > >> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
> (and I
> > >> believe - correct me if I am wrong Ash - important trigger for the
> > >> discussion) so far what we wanted is to extend the primary key and it
> > would
> > >> ripple through all the pieces of Airflow -> models, API, UI etc. ...
> > >> However - we already have `bundle_name" and "bundle_version" in the
> Dag
> > >> model. So I think when we add a separate table where we map the bundle
> > to
> > >> the team, the "ripple effect" will be almost 0. We do not want to
> change
> > >> primary key, we do not want to change UI in any way (except filtering
> of
> > >> DAGs available based on your team - but that will be handled in Auth
> > >> Manager and will not impact UI in any way, I think that's a huge
> > >> simplification of the implementation, and if we agree to it - i think
> it
> > >> should speed up the implementation significantly. There are only a
> > limited
> > >> number of times where you need to look up the team_id - so having the
> > >> bundle -> team mapping in a separate table and having to look them up
> > >> should not be a problem. And it has much less complexity and
> > >> "ripple-effect" through the codebase (for example I could imagine 100s
> > or
> > >> thousands already written tests that would have to be adapted if we
> > changed
> > >> the primary key - where there will be pretty much zero impact on
> > existing
> > >> tests if we just add bundle -> team lookup table.
> > >>
> > >>
> > >>> One other thing I’d point out is that I think including executors per
> > >>> team is a very easy win and quite possible without much work. I
> already
> > >>> have much of the code written. Executors are already aware of Teams
> > that
> > >>> own them (merged), I have a PR open to have configuration per team
> > (with a
> > >>> quite simple and isolated approach, I believe you approved Jarek).
> The
> > last
> > >>> piece is updating the scheduling logic to route tasks from a
> particular
> > >>> Bundle to the correct executor, which shouldn’t be much work (though
> it
> > >>> would be easier if the Task models had a column for the team they
> > belong
> > >>> to, rather than having to look up the Dag and Bundle to get the
> team) I
> > >>> have a branch where I was experimenting with this logic already.
> > >>> Any who, long story short, I don’t think we necessarily need to
> remove
> > >>> this piece from the project's scope if it is already partly done and
> > not
> > >>> too difficult.
> > >>>
> > >> Yeah. I hear you here again. Certainly I would not want to just
> > >> **remove** it from the code. And, yep I totally forgot we have it in.
> > And
> > >> if we can make it in, easily (which it seems we can) - we can also
> > include
> > >> it in the first iteration. What I wanted to avoid really (from the
> > original
> > >> design) - again trying to simplify it, limit the changes, and speed up
> > >> implementation. And there is one "complexity" that I wanted to avoid
> > >> specifically - having to have separate , additional configuration per
> > team.
> > >> Not only because it complicates already complex configuration handling
> > (I
> > >> know we have PR for that) but mostly because if it is not needed, we
> can
> > >> simplify documentation and explain to our users easier what they need
> > to do
> > >> to have their own multi-team setup. And I am quite open to keeping
> > >> multiple-executors if we can avoid complicating configuration.
> > >>
> > >> But I think some details of that and whether we really need separate
> > >> configuration might also come as a result of updating the AIP - I am
> not
> > >> quite sure now if we need it, but we can discuss it when we iterate on
> > the
> > >> AIP.
> > >>
> > >> J.
> > >>
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Pavankumar Gopidesu <go...@gmail.com>]

Thanks Jarek, that's a great update on this AIP, now it's much more slim
down.

left a minor comment. :) Overall looking great.

Pavan

On Sat, Jun 21, 2025 at 3:10 PM Jens Scheffler <j_...@gmx.de.invalid>
wrote:

> Thanks for the rework/update of the AIP-72!
>
> Just a few small comments but overall I like it as it is much leaner
> than originally planned and is in a level of complexity that it really
> seems to be a benefit to close the gap as described.
>
> On 21.06.25 14:52, Jarek Potiuk wrote:
> > I updated the AIP - including architecture images and reviewed it (again)
> > and corrected any ambiguities and places where it needed to be changed.
> >
> > I think the current state
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > - nicely describes the proposal.
> >
> > Comparing to the previous one:
> >
> > 1. The DB changes are far less intrusive - no ripple effect on Airflow
> > 2. There is no need to merge configurations and provide different set of
> > configs per team - we can add it later but I do not see why we need it in
> > this simplified version
> > 3. We can still configure a different set of executors per team - that is
> > already implemented (we just need to wire it to the bundle -> team
> mapping).
> >
> > I think it will be way simpler and faster to implement this way and it
> > should serve as MVMT -> Minimum Viable Multi Team that we can give our
> > users so that they can provide feedback.
> >
> > J.
> >
> >
> >
> >
> > On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >>
> >>
> >>> I like this iteration a bit more now for sure, thanks for being
> receptive
> >>> to feedback! :)
> >>>
> >>
> >>> This now becomes quite close to what was proposing before, we now again
> >>> have a team ID (which I think is really needed here, glad to see it
> back)
> >>> and it will be used for auth management, configuration specification,
> etc
> >>> but will be carried by Bundle instead of the dag model. Which as you
> say
> >>> “For that we will need to make sure that both api-server, scheduler and
> >>> triggerer have access to the "bundle definition" (to perform the
> mapping)"
> >>> which honestly doesn’t feel too much different from the original
> proposal
> >>> we had last week of adding it to Dag table and ensuring it’s available
> >>> everywhere. but either way I’m happy to meet in the middle and keep it
> on
> >>> Bundle if everyone else feels that’s a more suitable location.
> >>>
> >> I think the big difference is the "ripple effect" that was discussed in
> >> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c (and I
> >> believe - correct me if I am wrong Ash - important trigger for the
> >> discussion) so far what we wanted is to extend the primary key and it
> would
> >> ripple through all the pieces of Airflow -> models, API, UI etc. ...
> >> However - we already have `bundle_name" and "bundle_version" in the Dag
> >> model. So I think when we add a separate table where we map the bundle
> to
> >> the team, the "ripple effect" will be almost 0. We do not want to change
> >> primary key, we do not want to change UI in any way (except filtering of
> >> DAGs available based on your team - but that will be handled in Auth
> >> Manager and will not impact UI in any way, I think that's a huge
> >> simplification of the implementation, and if we agree to it - i think it
> >> should speed up the implementation significantly. There are only a
> limited
> >> number of times where you need to look up the team_id - so having the
> >> bundle -> team mapping in a separate table and having to look them up
> >> should not be a problem. And it has much less complexity and
> >> "ripple-effect" through the codebase (for example I could imagine 100s
> or
> >> thousands already written tests that would have to be adapted if we
> changed
> >> the primary key - where there will be pretty much zero impact on
> existing
> >> tests if we just add bundle -> team lookup table.
> >>
> >>
> >>> One other thing I’d point out is that I think including executors per
> >>> team is a very easy win and quite possible without much work. I already
> >>> have much of the code written. Executors are already aware of Teams
> that
> >>> own them (merged), I have a PR open to have configuration per team
> (with a
> >>> quite simple and isolated approach, I believe you approved Jarek). The
> last
> >>> piece is updating the scheduling logic to route tasks from a particular
> >>> Bundle to the correct executor, which shouldn’t be much work (though it
> >>> would be easier if the Task models had a column for the team they
> belong
> >>> to, rather than having to look up the Dag and Bundle to get the team) I
> >>> have a branch where I was experimenting with this logic already.
> >>> Any who, long story short, I don’t think we necessarily need to remove
> >>> this piece from the project's scope if it is already partly done and
> not
> >>> too difficult.
> >>>
> >> Yeah. I hear you here again. Certainly I would not want to just
> >> **remove** it from the code. And, yep I totally forgot we have it in.
> And
> >> if we can make it in, easily (which it seems we can) - we can also
> include
> >> it in the first iteration. What I wanted to avoid really (from the
> original
> >> design) - again trying to simplify it, limit the changes, and speed up
> >> implementation. And there is one "complexity" that I wanted to avoid
> >> specifically - having to have separate , additional configuration per
> team.
> >> Not only because it complicates already complex configuration handling
> (I
> >> know we have PR for that) but mostly because if it is not needed, we can
> >> simplify documentation and explain to our users easier what they need
> to do
> >> to have their own multi-team setup. And I am quite open to keeping
> >> multiple-executors if we can avoid complicating configuration.
> >>
> >> But I think some details of that and whether we really need separate
> >> configuration might also come as a result of updating the AIP - I am not
> >> quite sure now if we need it, but we can discuss it when we iterate on
> the
> >> AIP.
> >>
> >> J.
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jens Scheffler <j_...@gmx.de.INVALID>]

Thanks for the rework/update of the AIP-72!

Just a few small comments but overall I like it as it is much leaner 
than originally planned and is in a level of complexity that it really 
seems to be a benefit to close the gap as described.

On 21.06.25 14:52, Jarek Potiuk wrote:
> I updated the AIP - including architecture images and reviewed it (again)
> and corrected any ambiguities and places where it needed to be changed.
>
> I think the current state
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> - nicely describes the proposal.
>
> Comparing to the previous one:
>
> 1. The DB changes are far less intrusive - no ripple effect on Airflow
> 2. There is no need to merge configurations and provide different set of
> configs per team - we can add it later but I do not see why we need it in
> this simplified version
> 3. We can still configure a different set of executors per team - that is
> already implemented (we just need to wire it to the bundle -> team mapping).
>
> I think it will be way simpler and faster to implement this way and it
> should serve as MVMT -> Minimum Viable Multi Team that we can give our
> users so that they can provide feedback.
>
> J.
>
>
>
>
> On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>>
>>
>>> I like this iteration a bit more now for sure, thanks for being receptive
>>> to feedback! :)
>>>
>>
>>> This now becomes quite close to what was proposing before, we now again
>>> have a team ID (which I think is really needed here, glad to see it back)
>>> and it will be used for auth management, configuration specification, etc
>>> but will be carried by Bundle instead of the dag model. Which as you say
>>> “For that we will need to make sure that both api-server, scheduler and
>>> triggerer have access to the "bundle definition" (to perform the mapping)"
>>> which honestly doesn’t feel too much different from the original proposal
>>> we had last week of adding it to Dag table and ensuring it’s available
>>> everywhere. but either way I’m happy to meet in the middle and keep it on
>>> Bundle if everyone else feels that’s a more suitable location.
>>>
>> I think the big difference is the "ripple effect" that was discussed in
>> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c (and I
>> believe - correct me if I am wrong Ash - important trigger for the
>> discussion) so far what we wanted is to extend the primary key and it would
>> ripple through all the pieces of Airflow -> models, API, UI etc. ...
>> However - we already have `bundle_name" and "bundle_version" in the Dag
>> model. So I think when we add a separate table where we map the bundle to
>> the team, the "ripple effect" will be almost 0. We do not want to change
>> primary key, we do not want to change UI in any way (except filtering of
>> DAGs available based on your team - but that will be handled in Auth
>> Manager and will not impact UI in any way, I think that's a huge
>> simplification of the implementation, and if we agree to it - i think it
>> should speed up the implementation significantly. There are only a limited
>> number of times where you need to look up the team_id - so having the
>> bundle -> team mapping in a separate table and having to look them up
>> should not be a problem. And it has much less complexity and
>> "ripple-effect" through the codebase (for example I could imagine 100s or
>> thousands already written tests that would have to be adapted if we changed
>> the primary key - where there will be pretty much zero impact on existing
>> tests if we just add bundle -> team lookup table.
>>
>>
>>> One other thing I’d point out is that I think including executors per
>>> team is a very easy win and quite possible without much work. I already
>>> have much of the code written. Executors are already aware of Teams that
>>> own them (merged), I have a PR open to have configuration per team (with a
>>> quite simple and isolated approach, I believe you approved Jarek). The last
>>> piece is updating the scheduling logic to route tasks from a particular
>>> Bundle to the correct executor, which shouldn’t be much work (though it
>>> would be easier if the Task models had a column for the team they belong
>>> to, rather than having to look up the Dag and Bundle to get the team) I
>>> have a branch where I was experimenting with this logic already.
>>> Any who, long story short, I don’t think we necessarily need to remove
>>> this piece from the project's scope if it is already partly done and not
>>> too difficult.
>>>
>> Yeah. I hear you here again. Certainly I would not want to just
>> **remove** it from the code. And, yep I totally forgot we have it in. And
>> if we can make it in, easily (which it seems we can) - we can also include
>> it in the first iteration. What I wanted to avoid really (from the original
>> design) - again trying to simplify it, limit the changes, and speed up
>> implementation. And there is one "complexity" that I wanted to avoid
>> specifically - having to have separate , additional configuration per team.
>> Not only because it complicates already complex configuration handling  (I
>> know we have PR for that) but mostly because if it is not needed, we can
>> simplify documentation and explain to our users easier what they need to do
>> to have their own multi-team setup. And I am quite open to keeping
>> multiple-executors if we can avoid complicating configuration.
>>
>> But I think some details of that and whether we really need separate
>> configuration might also come as a result of updating the AIP - I am not
>> quite sure now if we need it, but we can discuss it when we iterate on the
>> AIP.
>>
>> J.
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

I updated the AIP - including architecture images and reviewed it (again)
and corrected any ambiguities and places where it needed to be changed.

I think the current state
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
- nicely describes the proposal.

Comparing to the previous one:

1. The DB changes are far less intrusive - no ripple effect on Airflow
2. There is no need to merge configurations and provide different set of
configs per team - we can add it later but I do not see why we need it in
this simplified version
3. We can still configure a different set of executors per team - that is
already implemented (we just need to wire it to the bundle -> team mapping).

I think it will be way simpler and faster to implement this way and it
should serve as MVMT -> Minimum Viable Multi Team that we can give our
users so that they can provide feedback.

J.




On Fri, Jun 20, 2025 at 8:33 AM Jarek Potiuk <ja...@potiuk.com> wrote:

>
>
>
>>
>> I like this iteration a bit more now for sure, thanks for being receptive
>> to feedback! :)
>>
>
>
>> This now becomes quite close to what was proposing before, we now again
>> have a team ID (which I think is really needed here, glad to see it back)
>> and it will be used for auth management, configuration specification, etc
>> but will be carried by Bundle instead of the dag model. Which as you say
>> “For that we will need to make sure that both api-server, scheduler and
>> triggerer have access to the "bundle definition" (to perform the mapping)"
>> which honestly doesn’t feel too much different from the original proposal
>> we had last week of adding it to Dag table and ensuring it’s available
>> everywhere. but either way I’m happy to meet in the middle and keep it on
>> Bundle if everyone else feels that’s a more suitable location.
>>
>
> I think the big difference is the "ripple effect" that was discussed in
> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c (and I
> believe - correct me if I am wrong Ash - important trigger for the
> discussion) so far what we wanted is to extend the primary key and it would
> ripple through all the pieces of Airflow -> models, API, UI etc. ...
> However - we already have `bundle_name" and "bundle_version" in the Dag
> model. So I think when we add a separate table where we map the bundle to
> the team, the "ripple effect" will be almost 0. We do not want to change
> primary key, we do not want to change UI in any way (except filtering of
> DAGs available based on your team - but that will be handled in Auth
> Manager and will not impact UI in any way, I think that's a huge
> simplification of the implementation, and if we agree to it - i think it
> should speed up the implementation significantly. There are only a limited
> number of times where you need to look up the team_id - so having the
> bundle -> team mapping in a separate table and having to look them up
> should not be a problem. And it has much less complexity and
> "ripple-effect" through the codebase (for example I could imagine 100s or
> thousands already written tests that would have to be adapted if we changed
> the primary key - where there will be pretty much zero impact on existing
> tests if we just add bundle -> team lookup table.
>
>
>> One other thing I’d point out is that I think including executors per
>> team is a very easy win and quite possible without much work. I already
>> have much of the code written. Executors are already aware of Teams that
>> own them (merged), I have a PR open to have configuration per team (with a
>> quite simple and isolated approach, I believe you approved Jarek). The last
>> piece is updating the scheduling logic to route tasks from a particular
>> Bundle to the correct executor, which shouldn’t be much work (though it
>> would be easier if the Task models had a column for the team they belong
>> to, rather than having to look up the Dag and Bundle to get the team) I
>> have a branch where I was experimenting with this logic already.
>> Any who, long story short, I don’t think we necessarily need to remove
>> this piece from the project's scope if it is already partly done and not
>> too difficult.
>>
>
> Yeah. I hear you here again. Certainly I would not want to just
> **remove** it from the code. And, yep I totally forgot we have it in. And
> if we can make it in, easily (which it seems we can) - we can also include
> it in the first iteration. What I wanted to avoid really (from the original
> design) - again trying to simplify it, limit the changes, and speed up
> implementation. And there is one "complexity" that I wanted to avoid
> specifically - having to have separate , additional configuration per team.
> Not only because it complicates already complex configuration handling  (I
> know we have PR for that) but mostly because if it is not needed, we can
> simplify documentation and explain to our users easier what they need to do
> to have their own multi-team setup. And I am quite open to keeping
> multiple-executors if we can avoid complicating configuration.
>
> But I think some details of that and whether we really need separate
> configuration might also come as a result of updating the AIP - I am not
> quite sure now if we need it, but we can discuss it when we iterate on the
> AIP.
>
> J.
>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

>
> I like this iteration a bit more now for sure, thanks for being receptive
> to feedback! :)
>


> This now becomes quite close to what was proposing before, we now again
> have a team ID (which I think is really needed here, glad to see it back)
> and it will be used for auth management, configuration specification, etc
> but will be carried by Bundle instead of the dag model. Which as you say
> “For that we will need to make sure that both api-server, scheduler and
> triggerer have access to the "bundle definition" (to perform the mapping)"
> which honestly doesn’t feel too much different from the original proposal
> we had last week of adding it to Dag table and ensuring it’s available
> everywhere. but either way I’m happy to meet in the middle and keep it on
> Bundle if everyone else feels that’s a more suitable location.
>

I think the big difference is the "ripple effect" that was discussed in
https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c (and I
believe - correct me if I am wrong Ash - important trigger for the
discussion) so far what we wanted is to extend the primary key and it would
ripple through all the pieces of Airflow -> models, API, UI etc. ...
However - we already have `bundle_name" and "bundle_version" in the Dag
model. So I think when we add a separate table where we map the bundle to
the team, the "ripple effect" will be almost 0. We do not want to change
primary key, we do not want to change UI in any way (except filtering of
DAGs available based on your team - but that will be handled in Auth
Manager and will not impact UI in any way, I think that's a huge
simplification of the implementation, and if we agree to it - i think it
should speed up the implementation significantly. There are only a limited
number of times where you need to look up the team_id - so having the
bundle -> team mapping in a separate table and having to look them up
should not be a problem. And it has much less complexity and
"ripple-effect" through the codebase (for example I could imagine 100s or
thousands already written tests that would have to be adapted if we changed
the primary key - where there will be pretty much zero impact on existing
tests if we just add bundle -> team lookup table.


> One other thing I’d point out is that I think including executors per team
> is a very easy win and quite possible without much work. I already have
> much of the code written. Executors are already aware of Teams that own
> them (merged), I have a PR open to have configuration per team (with a
> quite simple and isolated approach, I believe you approved Jarek). The last
> piece is updating the scheduling logic to route tasks from a particular
> Bundle to the correct executor, which shouldn’t be much work (though it
> would be easier if the Task models had a column for the team they belong
> to, rather than having to look up the Dag and Bundle to get the team) I
> have a branch where I was experimenting with this logic already.
> Any who, long story short, I don’t think we necessarily need to remove
> this piece from the project's scope if it is already partly done and not
> too difficult.
>

Yeah. I hear you here again. Certainly I would not want to just  **remove**
it from the code. And, yep I totally forgot we have it in. And if we can
make it in, easily (which it seems we can) - we can also include it in the
first iteration. What I wanted to avoid really (from the original design) -
again trying to simplify it, limit the changes, and speed up
implementation. And there is one "complexity" that I wanted to avoid
specifically - having to have separate , additional configuration per team.
Not only because it complicates already complex configuration handling  (I
know we have PR for that) but mostly because if it is not needed, we can
simplify documentation and explain to our users easier what they need to do
to have their own multi-team setup. And I am quite open to keeping
multiple-executors if we can avoid complicating configuration.

But I think some details of that and whether we really need separate
configuration might also come as a result of updating the AIP - I am not
quite sure now if we need it, but we can discuss it when we iterate on the
AIP.

J.

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

["Oliveira, Niko" <on...@amazon.com.INVALID>]

> I think "bundle" is a great "first" approximation. But .. I hear you. It's
> too simple and not future-proof. So ... modifying my proposal: - we can add
> team_id to the bundle definition. And do immediately many-to-1 mapping of
> bundles to the team. And still pass the "team_id" to the auth manager
> rather than bundle. That should respond to your concerns and yes - removes
> the leaky "bundle" abstraction in auth manager. You are quite right it
> would be cumbersome.

>> When users provide configuration for their logical “team” do they specify
>> their Dag bundle(s) ID in the config file or env variables that they use to
>> set config? What would that look like concretely, especially if it’s a
>> compliment of more than one bundle? Also, again, how does this remain
>> stable over time if bundles are added and removed for a single logical
>> team. Does that invalidate the config for a logical team that is composed
>> of one or more bundles? Do we not want something more stable to represent a
>> team? Or do we limit teams to just one bundle?

> I think all that is responded to by adding "team_id" to the bundle
> definition - and it also allows all kinds of dynamic behaviours - moving
> bundles between teams, adding bundles to team etc - if we do the mapping
> dynamically, this should be fine. For that we will need to make sure that
> both api-server, scheduler and triggerer have access to the "bundle
> definition" (to perform the mapping) but that should be rather
> straightforward and we can share it via db. We only need to know which
> bundle dag is coming from and then we can easily map it to the team it
> belongs to. That should be quite easy.

I like this iteration a bit more now for sure, thanks for being receptive to feedback! :)
This now becomes quite close to what was proposing before, we now again have a team ID (which I think is really needed here, glad to see it back) and it will be used for auth management, configuration specification, etc but will be carried by Bundle instead of the dag model. Which as you say “For that we will need to make sure that both api-server, scheduler and triggerer have access to the "bundle definition" (to perform the mapping)" which honestly doesn’t feel too much different from the original proposal we had last week of adding it to Dag table and ensuring it’s available everywhere. but either way I’m happy to meet in the middle and keep it on Bundle if everyone else feels that’s a more suitable location.

One other thing I’d point out is that I think including executors per team is a very easy win and quite possible without much work. I already have much of the code written. Executors are already aware of Teams that own them (merged), I have a PR open to have configuration per team (with a quite simple and isolated approach, I believe you approved Jarek). The last piece is updating the scheduling logic to route tasks from a particular Bundle to the correct executor, which shouldn’t be much work (though it would be easier if the Task models had a column for the team they belong to, rather than having to look up the Dag and Bundle to get the team) I have a branch where I was experimenting with this logic already.
Any who, long story short, I don’t think we necessarily need to remove this piece from the project's scope if it is already partly done and not too difficult.

Cheers,
Niko


________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Thursday, June 19, 2025 1:06:32 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.



Also one small comment. Yes MT is NEEDED. My talk from Monday at Berlin
Buzzwords has just been published:
https://www.youtube.com/watch?v=EyhZOnbwc-4&list=PLq-odUc2x7i8dTff006Wg2r0fsseSGrpJ&index=50
-> and if you watch it (I quite recommend it :) ), I just briefly mentioned
that we are looking at a Multi-Team. And the ONLY question I get was "When
is MT going to be ready? We need it!" .... I answered - with the disclaimer
that it might not be what they expect - but it's quite clear that the
simpler solution we get and the faster we get it at the hands of the users,
the better it will be - because if we don't, we will never find out what
they really need.

On Thu, Jun 19, 2025 at 9:58 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Thanks Jens,  Niko,
>
> *Jens*:
>
> > took a long time digesting all the discussion thread. I think it would
> be good to rewrite details to a new AIP so that it can be compared with
> the old AIP.
>
> Definitely - I am planning to update the AIP and re-cast a vote if we will
> be broadly in support of this simplified version (and soon - see below).
>
> > I think this also could include the extension (or is this planned
> otherwise?) to link multiple Airflow instances via Pub/Sub such that
> dataset vents can be externalized... in some cases might be easier to
> host multiple instances.
>
> This is already really implemented with common.messaging and AIP-82 ->
> this is also what triggered Ash original "Is it still needed after AIP-82"
> ?
>
> > The different Envs as described below are looking good. Besides "Env"
> this could also map to dedicated "executor" profiles e.g. instantiating
> a dedicated executor per team? I think in regards of multiple executors
> we always intended allowing to instantiate the same executor multiple
> times. Then the Env could be mapped to an executor each.
>
> I would like to simplify things first and come up with a design that will
> require minimal changes. What Ash wrote triggered me indeed "is it worth to
> complicate Airflow that much to implement just MT even if we know that
> different people have different expectations and we do not know if we
> respond to them" - so I completely changed my assumptions. Rather than
> thinking far ahead, I thought (with this new design) - what is the minimal
> set of changes that will get "some" multi-team variant that we can give to
> our users quickly and get feedback. So maybe yes - in the future - we might
> want separate executors, but I propose - let's start as simply as possible
> - ideally by 3.1, 3.2 latest.
>
> > Yeah and in this regard the triggerer would also need a feature
> increment to be hosted with multiple instances. It might be worth
> considering this is needed per executor anyway. I assume this is lagging
> a bit, also for Edge Executor there is no async coverage.... so my
> background with the assumption that different providers and tools might
> be needed in triggerer would elan rather top Option 3) as sketched below.
>
> In the AIP I will propose (again) a minimal set of changes to support the
> single "executor set" case. I propose We can revise it later as the next
> AIP.
>
> *Niko:*
>
> > Revisiting the proposal for multi team sounds like a reasonable idea. It
> is the feature request that we receive the most from our users/customers
> and we certainly want to get it right (and in a way that services both
> users and stakeholders as Jarek mentioned). AIP-67 was written with some
> Airflow 3 features in mind (such as Task SDK), so it’s not completely
> outdated but some features like event driving scheduling might be cause for
> reevaluation. But I honestly don't think it's all that out dated personally.
>
> See above - I re-evaluated that under the "simplicity first". Previously
> without the AIPs implemented for Airflow 3 we had to do a lot more, and the
> original design from Airflow 2 is still "leaking" in the current AIP. Maybe
> we will get there eventually with "more" complexity (multiple executor sets
> for example) - but I came to the conclusion that incremental implementation
> with much smaller scope should be good enough to get us from the ground and
> get feedback from the users.
>
> > As for using Dag Bundles as an approximation for teams, I don’t think
> I’m fully sold, but looking forward to hearing/discussing more. Some
> concerns I have with it:
> >  don’t think bundles are a great entity to represent team-ness for all
> Airflow components and features. They are only indirectly related to
> connections and variables for example, or for configuration. Also passing
> bundle ID(s) (presumably?) to the auth manager feels very strange to me and
> seems to be a very leaky abstraction for bundles. How do we handle a
> logical team that is made up of more than one bundle? Or when bundles are
> added/removed from that logical team? Users will be constantly chasing
> their tails to keep their “team” up to date in their auth manager.
>
> I think "bundle" is a great "first" approximation. But .. I hear you. It's
> too simple and not future-proof. So ... modifying my proposal: - we can add
> team_id to the bundle definition. And do immediately many-to-1 mapping of
> bundles to the team. And still pass the "team_id" to the auth manager
> rather than bundle. That should respond to your concerns and yes - removes
> the leaky "bundle" abstraction in auth manager. You are quite right it
> would be cumbersome.
>
> > When users provide configuration for their logical “team” do they
> specify their Dag bundle(s) ID in the config file or env variables that
> they use to set config? What would that look like concretely, especially if
> it’s a compliment of more than one bundle? Also, again, how does this
> remain stable over time if bundles are added and removed for a single
> logical team. Does that invalidate the config for a logical team that is
> composed of one or more bundles? Do we not want something more stable to
> represent a team? Or do we limit teams to just one bundle?
>
> I think all that is responded to by adding "team_id" to the bundle
> definition - and it also allows all kinds of dynamic behaviours - moving
> bundles between teams, adding bundles to team etc - if we do the mapping
> dynamically, this should be fine. For that we will need to make sure that
> both api-server, scheduler  and triggerer have access to the "bundle
> definition" (to perform the mapping) but that should be rather
> straightforward and we can share it via db. We only need to know which
> bundle dag is coming from and then we can easily map it to the team it
> belongs to. That should be quite easy.
>
>
> I would love to conclude that part of the discussion quickly and propose
> AIP modifications if this is something we feel like a good direction. So I
> will keep it open for a few days and if you have any comments/ questions I
> am happy to follow up.
>
> Ash - particularly.
>
> You started that thread which means you care (and I really appreciate your
> perspective). And I would like to avoid the situation that there are still
> "other" concerns that you have not yet formulated. I feel I responded to
> pretty much all your concerns, by simplifying the approach and going
> incrementally and prioritising the simplicity and incremental approach over
> "big picture and complexity". If you have any more concerns - I think it
> would be great to formulate them now rather than after we get updates to
> the AIP and start voting.
>
> J.
>
>
>
> On Tue, Jun 17, 2025 at 11:28 PM Oliveira, Niko
> <on...@amazon.com.invalid> wrote:
>
>> Hey folks,
>>
>> I’ve been OOTO for the last week so I’m just catching up on this
>> discussion. Here are some thoughts:
>>
>> Revisiting the proposal for multi team sounds like a reasonable idea. It
>> is the feature request that we receive the most from our users/customers
>> and we certainly want to get it right (and in a way that services both
>> users and stakeholders as Jarek mentioned). AIP-67 was written with some
>> Airflow 3 features in mind (such as Task SDK), so it’s not completely
>> outdated but some features like event driving scheduling might be cause for
>> reevaluation. But I honestly don't think it's all that out dated personally.
>>
>> As for using Dag Bundles as an approximation for teams, I don’t think I’m
>> fully sold, but looking forward to hearing/discussing more. Some concerns I
>> have with it:
>>
>> I don’t think bundles are a great entity to represent team-ness for all
>> Airflow components and features. They are only indirectly related to
>> connections and variables for example, or for configuration. Also passing
>> bundle ID(s) (presumably?) to the auth manager feels very strange to me and
>> seems to be a very leaky abstraction for bundles. How do we handle a
>> logical team that is made up of more than one bundle? Or when bundles are
>> added/removed from that logical team? Users will be constantly chasing
>> their tails to keep their “team” up to date in their auth manager.
>>
>> When users provide configuration for their logical “team” do they specify
>> their Dag bundle(s) ID in the config file or env variables that they use to
>> set config? What would that look like concretely, especially if it’s a
>> compliment of more than one bundle? Also, again, how does this remain
>> stable over time if bundles are added and removed for a single logical
>> team. Does that invalidate the config for a logical team that is composed
>> of one or more bundles? Do we not want something more stable to represent a
>> team? Or do we limit teams to just one bundle?
>>
>> Overall, I think permeating bundles across Airflow as an approximation
>> for team is not going to scale well or be very future proof or meet
>> user/customer expectations. For the area of Dags and Dag execution it’s not
>> too bad, but for the items above (as well as Trigger as discussed below)
>> and as we continue to build on multi-team (make it more robust, add
>> requested features from users, etc) I think we’re going to find that it
>> doesn’t serve the job very well. I think a more stable and concrete
>> representation of teams in Airflow will be a much better platform to build
>> features off of. It may require some large-ish changes, but we weren’t
>> afraid of making such changes for other recent capabilities (and largely
>> those changes went smoothly) in Airflow and I think that was the right
>> call, both then and for this case as well.
>>
>> Cheers,
>> Niko
>>
>>
>> ________________________________
>> From: Jens Scheffler <j_...@gmx.de.INVALID>
>> Sent: Sunday, June 15, 2025 11:13:48 AM
>> To: dev@airflow.apache.org
>> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External
>> event driven dags) exists
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>>
>>
>>
>> Hi all,
>>
>> took a long time digesting all the discussion thread. I think it would
>> be good to rewrite details to a new AIP so that it can be compared with
>> the old AIP.
>>
>> I think this also could include the extension (or is this planned
>> otherwise?) to link multiple Airflow instances via Pub/Sub such that
>> dataset vents can be externalized... in some cases might be easier to
>> host multiple instances.
>>
>> The different Envs as described below are looking good. Besides "Env"
>> this could also map to dedicated "executor" profiles e.g. instantiating
>> a dedicated executor per team? I think in regards of multiple executors
>> we always intended allowing to instantiate the same executor multiple
>> times. Then the Env could be mapped to an executor each.
>>
>> Yeah and in this regard the triggerer would also need a feature
>> increment to be hosted with multiple instances. It might be worth
>> considering this is needed per executor anyway. I assume this is lagging
>> a bit, also for Edge Executor there is no async coverage.... so my
>> background with the assumption that different providers and tools might
>> be needed in triggerer would elan rather top Option 3) as sketched below.
>>
>> Jens
>>
>> On 14.06.25 08:21, Jarek Potiuk wrote:
>> > On Fri, Jun 13, 2025 at 7:09 PM Vincent Beck <vi...@apache.org>
>> wrote:
>> >
>> >> Thanks, Jarek, for this proposal. Overall, I really like it—it
>> >> significantly simplifies multi-team support and removes the need to
>> deploy
>> >> additional components per team, without compromising on the core needs
>> of
>> >> users (unless I’m missing something).
>> >>
>> > Yep. I think with this "design" iteration, I put "simplicity" and
>> > "maintainability" as the primary goal. Separate configuration per tem
>> goes
>> > out the window, ripple-effect on the DB goes out the window, what's
>> left is
>> > basically the same Airflow we already have with few modifications.
>> >
>> >
>> >>> And if we do it and implement packaging and execution environments
>> (say
>> >> ability of choosing predefined venv to parse and execute DAGs coming
>> from a
>> >> specific bundle_id - the expectation 2) above can be handled well.
>> >>
>> >> Could you elaborate on this part? I’m not entirely clear on how it
>> would
>> >> work in practice. For instance, how would it behave with two teams or
>> >> bundles? Real-world examples would help clarify this, unless it's more
>> >> implementation details that we can flesh out once there's agreement on
>> the
>> >> general approach.
>> >>
>> > Currently with Bundle definition we **just** define where the DAGs are
>> > coming from. But we could (and that was even part of the original
>> design)
>> > add extra "execution environment" configuration. For example when we
>> have
>> > bundle_a and bundle_b each of them could have separate "environment"
>> > specified (say env_a, env_b) and we could map such environment to
>> specific
>> > image (image_a, image_b) or virtualenv in the same image (/venv/a/ ,
>> > /venv/b) that would be predefined in the processor/worker images. (or in
>> > VMs if images are not used). The envs might have different sets of
>> > dependencies (providers and others) installed, and both DAG processor
>> > parsing and "Worker" (in celery or k8s Pod) would be run using that
>> > environment. Initially AIP-67 also discussed defining dependencies in
>> > bundle and installing it dynamically (like Python Venv Operator) - but I
>> > think personally having predefined set of environments rather than
>> > dynamically creating (like ExternalPythonOperator) has much better
>> > maintainability, stability and security properties.
>> >
>> >
>> >> Also, what about the triggerer? Since the triggerer runs user code, the
>> >> original AIP-67 proposal required at least one triggerer per team. How
>> >> would that be handled under this new architecture?
>> >>
>> > That is an excellent question :) .  There are few options - depending on
>> > how much of the point 4) "isolating workload" we want to implement.
>> > Paradoxically - to be honest-  for me, the Triggerer always had the
>> > potential of being less of a problem when it comes to isolation. Yes.
>> All
>> > triggers are (currently) running not only in the same interpreter, but
>> also
>> > in the same event loop (which means that isolation goes out of the
>> window),
>> > but also it's relatively easy to introduce the isolation and we've been
>> > discussing options about it in the past as well. I see quite a few.
>> >
>> > Option 1) simplest operationally - We could add a mode in the Airflow
>> that
>> > would resemble Timetables. All Triggers would have to be exposed via the
>> > plugin interface (we could easily expose all triggers this way from all
>> our
>> > providers in a bulk way). This means that deployment manager will have
>> > control on what is run in Triggerrer - effectively limiting it
>> similarly as
>> > Scheduler code today. That would prevent some of the other cases we
>> > discussed recently (such as sending "notification" serialized methods to
>> > triggerer to execute), but that's mostly optimization, and they could be
>> > sent as worker tasks instead in this case).
>> >
>> > Option 2) Semi-isolation - for a number of our users just separating
>> > processes might be "enough" (especially if we add cgroups to isolate
>> > the processes - we had that in the past). Not "perfect" and does not
>> have
>> > all security properties, but for a number of our users it might be "good
>> > enough" because they will trust their teams enough to not worry about
>> > potential "malicious actions". In this case a single Triggerrer could
>> run
>> > several event loops - one per bundle, each of them in a separate,
>> isolated
>> > - process and the only change that we would have to do is to route the
>> > triggers to the right loop based on bundle id. Almost no operational
>> > complexity increases, but isolation is greatly improved. Again following
>> > the bundle -> environment mapping each of those processes could be run
>> > using a specific "per-bundle" environment where all necessary
>> dependencies
>> > would be installed. And here the limit of arbitrary code execution
>> coming
>> > from DAG can be lifted.
>> >
>> > Option 3) Full isolation -> simply run one triggerer per bundle. That
>> is a
>> > bit more like the original proposal, because we will then have an extra
>> > triggerer for each bundle/team (or group of bundles - it does not have
>> to
>> > be 1-to-1 mapping, could be many-to-1) . But it should provide full
>> > "security" properties with isolation and separation of workload, each
>> > triggerer could run completely in the same environment as defined in the
>> > bundle. It increases operational complexity - but just a bit. Rainbows
>> and
>> > unicorns - we have it all.
>> >
>> > Also one more thing.
>> >
>> > We usually discuss technical aspects here in develist and rarely talk
>> > about "business". But I think this is in some cases wrong - including a
>> > multi-team that has the potential of either supporting or undermining
>> some
>> > of the business our stakeholders do with Airflow.
>> >
>> > I would like to - really - make a collaborative effort to come up with a
>> > multi-team approach with all the stakeholders here - Amazon, Google,
>> > Astronomer especially should all be on-board with it. We know our users
>> > need it (survey and a number of talks about multi-team/tenancy that were
>> > submitted this year for Summit speak for themselves - we had ~10
>> sessions
>> > submitted about it this year, 30% of survey respondents want it -
>> though of
>> > course as Ash correctly pointed out many of those people have different
>> > expectations). Again multi-team has the potential of either killing or
>> > supporting some of the business models our stakeholders might implement
>> in
>> > their offering. And while here we do not "care" too much about those
>> > models, we should care about our stakeholders sustainability  - as they
>> are
>> > the ones who are fueling Airflow in many ways - so it would be stupid
>> if we
>> > do not consider their expectations and needs and - yes - sustainability
>> of
>> > their businesses. Here in the community we mostly add features that can
>> be
>> > used by everyone - whether in "as a service" or "on-prem" environment.
>> And
>> > we cannot "know" what the business is being planned or is possible or is
>> > good for our stakeholders. But we can collaboratively design the feature
>> > that might be usable on-prem - but one that we know is good for everyone
>> > and they can continue making business (or even better - provide better
>> > offerings to their users building on top of it.
>> >
>> > Let's do it. If there are things we can improve/make better here, I
>> want to
>> > hear - from everyone - Ash, Vikram, Raj, Vincent, Rafał, Michał - if
>> there
>> > is any idea how to improve it and make it better also for you - I think
>> > it's a good time to discuss it.
>> >
>> > J.
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

Also one small comment. Yes MT is NEEDED. My talk from Monday at Berlin
Buzzwords has just been published:
https://www.youtube.com/watch?v=EyhZOnbwc-4&list=PLq-odUc2x7i8dTff006Wg2r0fsseSGrpJ&index=50
-> and if you watch it (I quite recommend it :) ), I just briefly mentioned
that we are looking at a Multi-Team. And the ONLY question I get was "When
is MT going to be ready? We need it!" .... I answered - with the disclaimer
that it might not be what they expect - but it's quite clear that the
simpler solution we get and the faster we get it at the hands of the users,
the better it will be - because if we don't, we will never find out what
they really need.

On Thu, Jun 19, 2025 at 9:58 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Thanks Jens,  Niko,
>
> *Jens*:
>
> > took a long time digesting all the discussion thread. I think it would
> be good to rewrite details to a new AIP so that it can be compared with
> the old AIP.
>
> Definitely - I am planning to update the AIP and re-cast a vote if we will
> be broadly in support of this simplified version (and soon - see below).
>
> > I think this also could include the extension (or is this planned
> otherwise?) to link multiple Airflow instances via Pub/Sub such that
> dataset vents can be externalized... in some cases might be easier to
> host multiple instances.
>
> This is already really implemented with common.messaging and AIP-82 ->
> this is also what triggered Ash original "Is it still needed after AIP-82"
> ?
>
> > The different Envs as described below are looking good. Besides "Env"
> this could also map to dedicated "executor" profiles e.g. instantiating
> a dedicated executor per team? I think in regards of multiple executors
> we always intended allowing to instantiate the same executor multiple
> times. Then the Env could be mapped to an executor each.
>
> I would like to simplify things first and come up with a design that will
> require minimal changes. What Ash wrote triggered me indeed "is it worth to
> complicate Airflow that much to implement just MT even if we know that
> different people have different expectations and we do not know if we
> respond to them" - so I completely changed my assumptions. Rather than
> thinking far ahead, I thought (with this new design) - what is the minimal
> set of changes that will get "some" multi-team variant that we can give to
> our users quickly and get feedback. So maybe yes - in the future - we might
> want separate executors, but I propose - let's start as simply as possible
> - ideally by 3.1, 3.2 latest.
>
> > Yeah and in this regard the triggerer would also need a feature
> increment to be hosted with multiple instances. It might be worth
> considering this is needed per executor anyway. I assume this is lagging
> a bit, also for Edge Executor there is no async coverage.... so my
> background with the assumption that different providers and tools might
> be needed in triggerer would elan rather top Option 3) as sketched below.
>
> In the AIP I will propose (again) a minimal set of changes to support the
> single "executor set" case. I propose We can revise it later as the next
> AIP.
>
> *Niko:*
>
> > Revisiting the proposal for multi team sounds like a reasonable idea. It
> is the feature request that we receive the most from our users/customers
> and we certainly want to get it right (and in a way that services both
> users and stakeholders as Jarek mentioned). AIP-67 was written with some
> Airflow 3 features in mind (such as Task SDK), so it’s not completely
> outdated but some features like event driving scheduling might be cause for
> reevaluation. But I honestly don't think it's all that out dated personally.
>
> See above - I re-evaluated that under the "simplicity first". Previously
> without the AIPs implemented for Airflow 3 we had to do a lot more, and the
> original design from Airflow 2 is still "leaking" in the current AIP. Maybe
> we will get there eventually with "more" complexity (multiple executor sets
> for example) - but I came to the conclusion that incremental implementation
> with much smaller scope should be good enough to get us from the ground and
> get feedback from the users.
>
> > As for using Dag Bundles as an approximation for teams, I don’t think
> I’m fully sold, but looking forward to hearing/discussing more. Some
> concerns I have with it:
> >  don’t think bundles are a great entity to represent team-ness for all
> Airflow components and features. They are only indirectly related to
> connections and variables for example, or for configuration. Also passing
> bundle ID(s) (presumably?) to the auth manager feels very strange to me and
> seems to be a very leaky abstraction for bundles. How do we handle a
> logical team that is made up of more than one bundle? Or when bundles are
> added/removed from that logical team? Users will be constantly chasing
> their tails to keep their “team” up to date in their auth manager.
>
> I think "bundle" is a great "first" approximation. But .. I hear you. It's
> too simple and not future-proof. So ... modifying my proposal: - we can add
> team_id to the bundle definition. And do immediately many-to-1 mapping of
> bundles to the team. And still pass the "team_id" to the auth manager
> rather than bundle. That should respond to your concerns and yes - removes
> the leaky "bundle" abstraction in auth manager. You are quite right it
> would be cumbersome.
>
> > When users provide configuration for their logical “team” do they
> specify their Dag bundle(s) ID in the config file or env variables that
> they use to set config? What would that look like concretely, especially if
> it’s a compliment of more than one bundle? Also, again, how does this
> remain stable over time if bundles are added and removed for a single
> logical team. Does that invalidate the config for a logical team that is
> composed of one or more bundles? Do we not want something more stable to
> represent a team? Or do we limit teams to just one bundle?
>
> I think all that is responded to by adding "team_id" to the bundle
> definition - and it also allows all kinds of dynamic behaviours - moving
> bundles between teams, adding bundles to team etc - if we do the mapping
> dynamically, this should be fine. For that we will need to make sure that
> both api-server, scheduler  and triggerer have access to the "bundle
> definition" (to perform the mapping) but that should be rather
> straightforward and we can share it via db. We only need to know which
> bundle dag is coming from and then we can easily map it to the team it
> belongs to. That should be quite easy.
>
>
> I would love to conclude that part of the discussion quickly and propose
> AIP modifications if this is something we feel like a good direction. So I
> will keep it open for a few days and if you have any comments/ questions I
> am happy to follow up.
>
> Ash - particularly.
>
> You started that thread which means you care (and I really appreciate your
> perspective). And I would like to avoid the situation that there are still
> "other" concerns that you have not yet formulated. I feel I responded to
> pretty much all your concerns, by simplifying the approach and going
> incrementally and prioritising the simplicity and incremental approach over
> "big picture and complexity". If you have any more concerns - I think it
> would be great to formulate them now rather than after we get updates to
> the AIP and start voting.
>
> J.
>
>
>
> On Tue, Jun 17, 2025 at 11:28 PM Oliveira, Niko
> <on...@amazon.com.invalid> wrote:
>
>> Hey folks,
>>
>> I’ve been OOTO for the last week so I’m just catching up on this
>> discussion. Here are some thoughts:
>>
>> Revisiting the proposal for multi team sounds like a reasonable idea. It
>> is the feature request that we receive the most from our users/customers
>> and we certainly want to get it right (and in a way that services both
>> users and stakeholders as Jarek mentioned). AIP-67 was written with some
>> Airflow 3 features in mind (such as Task SDK), so it’s not completely
>> outdated but some features like event driving scheduling might be cause for
>> reevaluation. But I honestly don't think it's all that out dated personally.
>>
>> As for using Dag Bundles as an approximation for teams, I don’t think I’m
>> fully sold, but looking forward to hearing/discussing more. Some concerns I
>> have with it:
>>
>> I don’t think bundles are a great entity to represent team-ness for all
>> Airflow components and features. They are only indirectly related to
>> connections and variables for example, or for configuration. Also passing
>> bundle ID(s) (presumably?) to the auth manager feels very strange to me and
>> seems to be a very leaky abstraction for bundles. How do we handle a
>> logical team that is made up of more than one bundle? Or when bundles are
>> added/removed from that logical team? Users will be constantly chasing
>> their tails to keep their “team” up to date in their auth manager.
>>
>> When users provide configuration for their logical “team” do they specify
>> their Dag bundle(s) ID in the config file or env variables that they use to
>> set config? What would that look like concretely, especially if it’s a
>> compliment of more than one bundle? Also, again, how does this remain
>> stable over time if bundles are added and removed for a single logical
>> team. Does that invalidate the config for a logical team that is composed
>> of one or more bundles? Do we not want something more stable to represent a
>> team? Or do we limit teams to just one bundle?
>>
>> Overall, I think permeating bundles across Airflow as an approximation
>> for team is not going to scale well or be very future proof or meet
>> user/customer expectations. For the area of Dags and Dag execution it’s not
>> too bad, but for the items above (as well as Trigger as discussed below)
>> and as we continue to build on multi-team (make it more robust, add
>> requested features from users, etc) I think we’re going to find that it
>> doesn’t serve the job very well. I think a more stable and concrete
>> representation of teams in Airflow will be a much better platform to build
>> features off of. It may require some large-ish changes, but we weren’t
>> afraid of making such changes for other recent capabilities (and largely
>> those changes went smoothly) in Airflow and I think that was the right
>> call, both then and for this case as well.
>>
>> Cheers,
>> Niko
>>
>>
>> ________________________________
>> From: Jens Scheffler <j_...@gmx.de.INVALID>
>> Sent: Sunday, June 15, 2025 11:13:48 AM
>> To: dev@airflow.apache.org
>> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External
>> event driven dags) exists
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>>
>>
>>
>> Hi all,
>>
>> took a long time digesting all the discussion thread. I think it would
>> be good to rewrite details to a new AIP so that it can be compared with
>> the old AIP.
>>
>> I think this also could include the extension (or is this planned
>> otherwise?) to link multiple Airflow instances via Pub/Sub such that
>> dataset vents can be externalized... in some cases might be easier to
>> host multiple instances.
>>
>> The different Envs as described below are looking good. Besides "Env"
>> this could also map to dedicated "executor" profiles e.g. instantiating
>> a dedicated executor per team? I think in regards of multiple executors
>> we always intended allowing to instantiate the same executor multiple
>> times. Then the Env could be mapped to an executor each.
>>
>> Yeah and in this regard the triggerer would also need a feature
>> increment to be hosted with multiple instances. It might be worth
>> considering this is needed per executor anyway. I assume this is lagging
>> a bit, also for Edge Executor there is no async coverage.... so my
>> background with the assumption that different providers and tools might
>> be needed in triggerer would elan rather top Option 3) as sketched below.
>>
>> Jens
>>
>> On 14.06.25 08:21, Jarek Potiuk wrote:
>> > On Fri, Jun 13, 2025 at 7:09 PM Vincent Beck <vi...@apache.org>
>> wrote:
>> >
>> >> Thanks, Jarek, for this proposal. Overall, I really like it—it
>> >> significantly simplifies multi-team support and removes the need to
>> deploy
>> >> additional components per team, without compromising on the core needs
>> of
>> >> users (unless I’m missing something).
>> >>
>> > Yep. I think with this "design" iteration, I put "simplicity" and
>> > "maintainability" as the primary goal. Separate configuration per tem
>> goes
>> > out the window, ripple-effect on the DB goes out the window, what's
>> left is
>> > basically the same Airflow we already have with few modifications.
>> >
>> >
>> >>> And if we do it and implement packaging and execution environments
>> (say
>> >> ability of choosing predefined venv to parse and execute DAGs coming
>> from a
>> >> specific bundle_id - the expectation 2) above can be handled well.
>> >>
>> >> Could you elaborate on this part? I’m not entirely clear on how it
>> would
>> >> work in practice. For instance, how would it behave with two teams or
>> >> bundles? Real-world examples would help clarify this, unless it's more
>> >> implementation details that we can flesh out once there's agreement on
>> the
>> >> general approach.
>> >>
>> > Currently with Bundle definition we **just** define where the DAGs are
>> > coming from. But we could (and that was even part of the original
>> design)
>> > add extra "execution environment" configuration. For example when we
>> have
>> > bundle_a and bundle_b each of them could have separate "environment"
>> > specified (say env_a, env_b) and we could map such environment to
>> specific
>> > image (image_a, image_b) or virtualenv in the same image (/venv/a/ ,
>> > /venv/b) that would be predefined in the processor/worker images. (or in
>> > VMs if images are not used). The envs might have different sets of
>> > dependencies (providers and others) installed, and both DAG processor
>> > parsing and "Worker" (in celery or k8s Pod) would be run using that
>> > environment. Initially AIP-67 also discussed defining dependencies in
>> > bundle and installing it dynamically (like Python Venv Operator) - but I
>> > think personally having predefined set of environments rather than
>> > dynamically creating (like ExternalPythonOperator) has much better
>> > maintainability, stability and security properties.
>> >
>> >
>> >> Also, what about the triggerer? Since the triggerer runs user code, the
>> >> original AIP-67 proposal required at least one triggerer per team. How
>> >> would that be handled under this new architecture?
>> >>
>> > That is an excellent question :) .  There are few options - depending on
>> > how much of the point 4) "isolating workload" we want to implement.
>> > Paradoxically - to be honest-  for me, the Triggerer always had the
>> > potential of being less of a problem when it comes to isolation. Yes.
>> All
>> > triggers are (currently) running not only in the same interpreter, but
>> also
>> > in the same event loop (which means that isolation goes out of the
>> window),
>> > but also it's relatively easy to introduce the isolation and we've been
>> > discussing options about it in the past as well. I see quite a few.
>> >
>> > Option 1) simplest operationally - We could add a mode in the Airflow
>> that
>> > would resemble Timetables. All Triggers would have to be exposed via the
>> > plugin interface (we could easily expose all triggers this way from all
>> our
>> > providers in a bulk way). This means that deployment manager will have
>> > control on what is run in Triggerrer - effectively limiting it
>> similarly as
>> > Scheduler code today. That would prevent some of the other cases we
>> > discussed recently (such as sending "notification" serialized methods to
>> > triggerer to execute), but that's mostly optimization, and they could be
>> > sent as worker tasks instead in this case).
>> >
>> > Option 2) Semi-isolation - for a number of our users just separating
>> > processes might be "enough" (especially if we add cgroups to isolate
>> > the processes - we had that in the past). Not "perfect" and does not
>> have
>> > all security properties, but for a number of our users it might be "good
>> > enough" because they will trust their teams enough to not worry about
>> > potential "malicious actions". In this case a single Triggerrer could
>> run
>> > several event loops - one per bundle, each of them in a separate,
>> isolated
>> > - process and the only change that we would have to do is to route the
>> > triggers to the right loop based on bundle id. Almost no operational
>> > complexity increases, but isolation is greatly improved. Again following
>> > the bundle -> environment mapping each of those processes could be run
>> > using a specific "per-bundle" environment where all necessary
>> dependencies
>> > would be installed. And here the limit of arbitrary code execution
>> coming
>> > from DAG can be lifted.
>> >
>> > Option 3) Full isolation -> simply run one triggerer per bundle. That
>> is a
>> > bit more like the original proposal, because we will then have an extra
>> > triggerer for each bundle/team (or group of bundles - it does not have
>> to
>> > be 1-to-1 mapping, could be many-to-1) . But it should provide full
>> > "security" properties with isolation and separation of workload, each
>> > triggerer could run completely in the same environment as defined in the
>> > bundle. It increases operational complexity - but just a bit. Rainbows
>> and
>> > unicorns - we have it all.
>> >
>> > Also one more thing.
>> >
>> > We usually discuss technical aspects here in develist and rarely talk
>> > about "business". But I think this is in some cases wrong - including a
>> > multi-team that has the potential of either supporting or undermining
>> some
>> > of the business our stakeholders do with Airflow.
>> >
>> > I would like to - really - make a collaborative effort to come up with a
>> > multi-team approach with all the stakeholders here - Amazon, Google,
>> > Astronomer especially should all be on-board with it. We know our users
>> > need it (survey and a number of talks about multi-team/tenancy that were
>> > submitted this year for Summit speak for themselves - we had ~10
>> sessions
>> > submitted about it this year, 30% of survey respondents want it -
>> though of
>> > course as Ash correctly pointed out many of those people have different
>> > expectations). Again multi-team has the potential of either killing or
>> > supporting some of the business models our stakeholders might implement
>> in
>> > their offering. And while here we do not "care" too much about those
>> > models, we should care about our stakeholders sustainability  - as they
>> are
>> > the ones who are fueling Airflow in many ways - so it would be stupid
>> if we
>> > do not consider their expectations and needs and - yes - sustainability
>> of
>> > their businesses. Here in the community we mostly add features that can
>> be
>> > used by everyone - whether in "as a service" or "on-prem" environment.
>> And
>> > we cannot "know" what the business is being planned or is possible or is
>> > good for our stakeholders. But we can collaboratively design the feature
>> > that might be usable on-prem - but one that we know is good for everyone
>> > and they can continue making business (or even better - provide better
>> > offerings to their users building on top of it.
>> >
>> > Let's do it. If there are things we can improve/make better here, I
>> want to
>> > hear - from everyone - Ash, Vikram, Raj, Vincent, Rafał, Michał - if
>> there
>> > is any idea how to improve it and make it better also for you - I think
>> > it's a good time to discuss it.
>> >
>> > J.
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

Thanks Jens,  Niko,

*Jens*:

> took a long time digesting all the discussion thread. I think it would
be good to rewrite details to a new AIP so that it can be compared with
the old AIP.

Definitely - I am planning to update the AIP and re-cast a vote if we will
be broadly in support of this simplified version (and soon - see below).

> I think this also could include the extension (or is this planned
otherwise?) to link multiple Airflow instances via Pub/Sub such that
dataset vents can be externalized... in some cases might be easier to
host multiple instances.

This is already really implemented with common.messaging and AIP-82 -> this
is also what triggered Ash original "Is it still needed after AIP-82" ?

> The different Envs as described below are looking good. Besides "Env"
this could also map to dedicated "executor" profiles e.g. instantiating
a dedicated executor per team? I think in regards of multiple executors
we always intended allowing to instantiate the same executor multiple
times. Then the Env could be mapped to an executor each.

I would like to simplify things first and come up with a design that will
require minimal changes. What Ash wrote triggered me indeed "is it worth to
complicate Airflow that much to implement just MT even if we know that
different people have different expectations and we do not know if we
respond to them" - so I completely changed my assumptions. Rather than
thinking far ahead, I thought (with this new design) - what is the minimal
set of changes that will get "some" multi-team variant that we can give to
our users quickly and get feedback. So maybe yes - in the future - we might
want separate executors, but I propose - let's start as simply as possible
- ideally by 3.1, 3.2 latest.

> Yeah and in this regard the triggerer would also need a feature
increment to be hosted with multiple instances. It might be worth
considering this is needed per executor anyway. I assume this is lagging
a bit, also for Edge Executor there is no async coverage.... so my
background with the assumption that different providers and tools might
be needed in triggerer would elan rather top Option 3) as sketched below.

In the AIP I will propose (again) a minimal set of changes to support the
single "executor set" case. I propose We can revise it later as the next
AIP.

*Niko:*

> Revisiting the proposal for multi team sounds like a reasonable idea. It
is the feature request that we receive the most from our users/customers
and we certainly want to get it right (and in a way that services both
users and stakeholders as Jarek mentioned). AIP-67 was written with some
Airflow 3 features in mind (such as Task SDK), so it’s not completely
outdated but some features like event driving scheduling might be cause for
reevaluation. But I honestly don't think it's all that out dated personally.

See above - I re-evaluated that under the "simplicity first". Previously
without the AIPs implemented for Airflow 3 we had to do a lot more, and the
original design from Airflow 2 is still "leaking" in the current AIP. Maybe
we will get there eventually with "more" complexity (multiple executor sets
for example) - but I came to the conclusion that incremental implementation
with much smaller scope should be good enough to get us from the ground and
get feedback from the users.

> As for using Dag Bundles as an approximation for teams, I don’t think I’m
fully sold, but looking forward to hearing/discussing more. Some concerns I
have with it:
>  don’t think bundles are a great entity to represent team-ness for all
Airflow components and features. They are only indirectly related to
connections and variables for example, or for configuration. Also passing
bundle ID(s) (presumably?) to the auth manager feels very strange to me and
seems to be a very leaky abstraction for bundles. How do we handle a
logical team that is made up of more than one bundle? Or when bundles are
added/removed from that logical team? Users will be constantly chasing
their tails to keep their “team” up to date in their auth manager.

I think "bundle" is a great "first" approximation. But .. I hear you. It's
too simple and not future-proof. So ... modifying my proposal: - we can add
team_id to the bundle definition. And do immediately many-to-1 mapping of
bundles to the team. And still pass the "team_id" to the auth manager
rather than bundle. That should respond to your concerns and yes - removes
the leaky "bundle" abstraction in auth manager. You are quite right it
would be cumbersome.

> When users provide configuration for their logical “team” do they specify
their Dag bundle(s) ID in the config file or env variables that they use to
set config? What would that look like concretely, especially if it’s a
compliment of more than one bundle? Also, again, how does this remain
stable over time if bundles are added and removed for a single logical
team. Does that invalidate the config for a logical team that is composed
of one or more bundles? Do we not want something more stable to represent a
team? Or do we limit teams to just one bundle?

I think all that is responded to by adding "team_id" to the bundle
definition - and it also allows all kinds of dynamic behaviours - moving
bundles between teams, adding bundles to team etc - if we do the mapping
dynamically, this should be fine. For that we will need to make sure that
both api-server, scheduler  and triggerer have access to the "bundle
definition" (to perform the mapping) but that should be rather
straightforward and we can share it via db. We only need to know which
bundle dag is coming from and then we can easily map it to the team it
belongs to. That should be quite easy.


I would love to conclude that part of the discussion quickly and propose
AIP modifications if this is something we feel like a good direction. So I
will keep it open for a few days and if you have any comments/ questions I
am happy to follow up.

Ash - particularly.

You started that thread which means you care (and I really appreciate your
perspective). And I would like to avoid the situation that there are still
"other" concerns that you have not yet formulated. I feel I responded to
pretty much all your concerns, by simplifying the approach and going
incrementally and prioritising the simplicity and incremental approach over
"big picture and complexity". If you have any more concerns - I think it
would be great to formulate them now rather than after we get updates to
the AIP and start voting.

J.



On Tue, Jun 17, 2025 at 11:28 PM Oliveira, Niko <on...@amazon.com.invalid>
wrote:

> Hey folks,
>
> I’ve been OOTO for the last week so I’m just catching up on this
> discussion. Here are some thoughts:
>
> Revisiting the proposal for multi team sounds like a reasonable idea. It
> is the feature request that we receive the most from our users/customers
> and we certainly want to get it right (and in a way that services both
> users and stakeholders as Jarek mentioned). AIP-67 was written with some
> Airflow 3 features in mind (such as Task SDK), so it’s not completely
> outdated but some features like event driving scheduling might be cause for
> reevaluation. But I honestly don't think it's all that out dated personally.
>
> As for using Dag Bundles as an approximation for teams, I don’t think I’m
> fully sold, but looking forward to hearing/discussing more. Some concerns I
> have with it:
>
> I don’t think bundles are a great entity to represent team-ness for all
> Airflow components and features. They are only indirectly related to
> connections and variables for example, or for configuration. Also passing
> bundle ID(s) (presumably?) to the auth manager feels very strange to me and
> seems to be a very leaky abstraction for bundles. How do we handle a
> logical team that is made up of more than one bundle? Or when bundles are
> added/removed from that logical team? Users will be constantly chasing
> their tails to keep their “team” up to date in their auth manager.
>
> When users provide configuration for their logical “team” do they specify
> their Dag bundle(s) ID in the config file or env variables that they use to
> set config? What would that look like concretely, especially if it’s a
> compliment of more than one bundle? Also, again, how does this remain
> stable over time if bundles are added and removed for a single logical
> team. Does that invalidate the config for a logical team that is composed
> of one or more bundles? Do we not want something more stable to represent a
> team? Or do we limit teams to just one bundle?
>
> Overall, I think permeating bundles across Airflow as an approximation for
> team is not going to scale well or be very future proof or meet
> user/customer expectations. For the area of Dags and Dag execution it’s not
> too bad, but for the items above (as well as Trigger as discussed below)
> and as we continue to build on multi-team (make it more robust, add
> requested features from users, etc) I think we’re going to find that it
> doesn’t serve the job very well. I think a more stable and concrete
> representation of teams in Airflow will be a much better platform to build
> features off of. It may require some large-ish changes, but we weren’t
> afraid of making such changes for other recent capabilities (and largely
> those changes went smoothly) in Airflow and I think that was the right
> call, both then and for this case as well.
>
> Cheers,
> Niko
>
>
> ________________________________
> From: Jens Scheffler <j_...@gmx.de.INVALID>
> Sent: Sunday, June 15, 2025 11:13:48 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External
> event driven dags) exists
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> Hi all,
>
> took a long time digesting all the discussion thread. I think it would
> be good to rewrite details to a new AIP so that it can be compared with
> the old AIP.
>
> I think this also could include the extension (or is this planned
> otherwise?) to link multiple Airflow instances via Pub/Sub such that
> dataset vents can be externalized... in some cases might be easier to
> host multiple instances.
>
> The different Envs as described below are looking good. Besides "Env"
> this could also map to dedicated "executor" profiles e.g. instantiating
> a dedicated executor per team? I think in regards of multiple executors
> we always intended allowing to instantiate the same executor multiple
> times. Then the Env could be mapped to an executor each.
>
> Yeah and in this regard the triggerer would also need a feature
> increment to be hosted with multiple instances. It might be worth
> considering this is needed per executor anyway. I assume this is lagging
> a bit, also for Edge Executor there is no async coverage.... so my
> background with the assumption that different providers and tools might
> be needed in triggerer would elan rather top Option 3) as sketched below.
>
> Jens
>
> On 14.06.25 08:21, Jarek Potiuk wrote:
> > On Fri, Jun 13, 2025 at 7:09 PM Vincent Beck <vi...@apache.org>
> wrote:
> >
> >> Thanks, Jarek, for this proposal. Overall, I really like it—it
> >> significantly simplifies multi-team support and removes the need to
> deploy
> >> additional components per team, without compromising on the core needs
> of
> >> users (unless I’m missing something).
> >>
> > Yep. I think with this "design" iteration, I put "simplicity" and
> > "maintainability" as the primary goal. Separate configuration per tem
> goes
> > out the window, ripple-effect on the DB goes out the window, what's left
> is
> > basically the same Airflow we already have with few modifications.
> >
> >
> >>> And if we do it and implement packaging and execution environments (say
> >> ability of choosing predefined venv to parse and execute DAGs coming
> from a
> >> specific bundle_id - the expectation 2) above can be handled well.
> >>
> >> Could you elaborate on this part? I’m not entirely clear on how it would
> >> work in practice. For instance, how would it behave with two teams or
> >> bundles? Real-world examples would help clarify this, unless it's more
> >> implementation details that we can flesh out once there's agreement on
> the
> >> general approach.
> >>
> > Currently with Bundle definition we **just** define where the DAGs are
> > coming from. But we could (and that was even part of the original design)
> > add extra "execution environment" configuration. For example when we have
> > bundle_a and bundle_b each of them could have separate "environment"
> > specified (say env_a, env_b) and we could map such environment to
> specific
> > image (image_a, image_b) or virtualenv in the same image (/venv/a/ ,
> > /venv/b) that would be predefined in the processor/worker images. (or in
> > VMs if images are not used). The envs might have different sets of
> > dependencies (providers and others) installed, and both DAG processor
> > parsing and "Worker" (in celery or k8s Pod) would be run using that
> > environment. Initially AIP-67 also discussed defining dependencies in
> > bundle and installing it dynamically (like Python Venv Operator) - but I
> > think personally having predefined set of environments rather than
> > dynamically creating (like ExternalPythonOperator) has much better
> > maintainability, stability and security properties.
> >
> >
> >> Also, what about the triggerer? Since the triggerer runs user code, the
> >> original AIP-67 proposal required at least one triggerer per team. How
> >> would that be handled under this new architecture?
> >>
> > That is an excellent question :) .  There are few options - depending on
> > how much of the point 4) "isolating workload" we want to implement.
> > Paradoxically - to be honest-  for me, the Triggerer always had the
> > potential of being less of a problem when it comes to isolation. Yes. All
> > triggers are (currently) running not only in the same interpreter, but
> also
> > in the same event loop (which means that isolation goes out of the
> window),
> > but also it's relatively easy to introduce the isolation and we've been
> > discussing options about it in the past as well. I see quite a few.
> >
> > Option 1) simplest operationally - We could add a mode in the Airflow
> that
> > would resemble Timetables. All Triggers would have to be exposed via the
> > plugin interface (we could easily expose all triggers this way from all
> our
> > providers in a bulk way). This means that deployment manager will have
> > control on what is run in Triggerrer - effectively limiting it similarly
> as
> > Scheduler code today. That would prevent some of the other cases we
> > discussed recently (such as sending "notification" serialized methods to
> > triggerer to execute), but that's mostly optimization, and they could be
> > sent as worker tasks instead in this case).
> >
> > Option 2) Semi-isolation - for a number of our users just separating
> > processes might be "enough" (especially if we add cgroups to isolate
> > the processes - we had that in the past). Not "perfect" and does not have
> > all security properties, but for a number of our users it might be "good
> > enough" because they will trust their teams enough to not worry about
> > potential "malicious actions". In this case a single Triggerrer could run
> > several event loops - one per bundle, each of them in a separate,
> isolated
> > - process and the only change that we would have to do is to route the
> > triggers to the right loop based on bundle id. Almost no operational
> > complexity increases, but isolation is greatly improved. Again following
> > the bundle -> environment mapping each of those processes could be run
> > using a specific "per-bundle" environment where all necessary
> dependencies
> > would be installed. And here the limit of arbitrary code execution coming
> > from DAG can be lifted.
> >
> > Option 3) Full isolation -> simply run one triggerer per bundle. That is
> a
> > bit more like the original proposal, because we will then have an extra
> > triggerer for each bundle/team (or group of bundles - it does not have to
> > be 1-to-1 mapping, could be many-to-1) . But it should provide full
> > "security" properties with isolation and separation of workload, each
> > triggerer could run completely in the same environment as defined in the
> > bundle. It increases operational complexity - but just a bit. Rainbows
> and
> > unicorns - we have it all.
> >
> > Also one more thing.
> >
> > We usually discuss technical aspects here in develist and rarely talk
> > about "business". But I think this is in some cases wrong - including a
> > multi-team that has the potential of either supporting or undermining
> some
> > of the business our stakeholders do with Airflow.
> >
> > I would like to - really - make a collaborative effort to come up with a
> > multi-team approach with all the stakeholders here - Amazon, Google,
> > Astronomer especially should all be on-board with it. We know our users
> > need it (survey and a number of talks about multi-team/tenancy that were
> > submitted this year for Summit speak for themselves - we had ~10 sessions
> > submitted about it this year, 30% of survey respondents want it - though
> of
> > course as Ash correctly pointed out many of those people have different
> > expectations). Again multi-team has the potential of either killing or
> > supporting some of the business models our stakeholders might implement
> in
> > their offering. And while here we do not "care" too much about those
> > models, we should care about our stakeholders sustainability  - as they
> are
> > the ones who are fueling Airflow in many ways - so it would be stupid if
> we
> > do not consider their expectations and needs and - yes - sustainability
> of
> > their businesses. Here in the community we mostly add features that can
> be
> > used by everyone - whether in "as a service" or "on-prem" environment.
> And
> > we cannot "know" what the business is being planned or is possible or is
> > good for our stakeholders. But we can collaboratively design the feature
> > that might be usable on-prem - but one that we know is good for everyone
> > and they can continue making business (or even better - provide better
> > offerings to their users building on top of it.
> >
> > Let's do it. If there are things we can improve/make better here, I want
> to
> > hear - from everyone - Ash, Vikram, Raj, Vincent, Rafał, Michał - if
> there
> > is any idea how to improve it and make it better also for you - I think
> > it's a good time to discuss it.
> >
> > J.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

["Oliveira, Niko" <on...@amazon.com.INVALID>]

Hey folks,

I’ve been OOTO for the last week so I’m just catching up on this discussion. Here are some thoughts:

Revisiting the proposal for multi team sounds like a reasonable idea. It is the feature request that we receive the most from our users/customers and we certainly want to get it right (and in a way that services both users and stakeholders as Jarek mentioned). AIP-67 was written with some Airflow 3 features in mind (such as Task SDK), so it’s not completely outdated but some features like event driving scheduling might be cause for reevaluation. But I honestly don't think it's all that out dated personally.

As for using Dag Bundles as an approximation for teams, I don’t think I’m fully sold, but looking forward to hearing/discussing more. Some concerns I have with it:

I don’t think bundles are a great entity to represent team-ness for all Airflow components and features. They are only indirectly related to connections and variables for example, or for configuration. Also passing bundle ID(s) (presumably?) to the auth manager feels very strange to me and seems to be a very leaky abstraction for bundles. How do we handle a logical team that is made up of more than one bundle? Or when bundles are added/removed from that logical team? Users will be constantly chasing their tails to keep their “team” up to date in their auth manager.

When users provide configuration for their logical “team” do they specify their Dag bundle(s) ID in the config file or env variables that they use to set config? What would that look like concretely, especially if it’s a compliment of more than one bundle? Also, again, how does this remain stable over time if bundles are added and removed for a single logical team. Does that invalidate the config for a logical team that is composed of one or more bundles? Do we not want something more stable to represent a team? Or do we limit teams to just one bundle?

Overall, I think permeating bundles across Airflow as an approximation for team is not going to scale well or be very future proof or meet user/customer expectations. For the area of Dags and Dag execution it’s not too bad, but for the items above (as well as Trigger as discussed below) and as we continue to build on multi-team (make it more robust, add requested features from users, etc) I think we’re going to find that it doesn’t serve the job very well. I think a more stable and concrete representation of teams in Airflow will be a much better platform to build features off of. It may require some large-ish changes, but we weren’t afraid of making such changes for other recent capabilities (and largely those changes went smoothly) in Airflow and I think that was the right call, both then and for this case as well.

Cheers,
Niko


________________________________
From: Jens Scheffler <j_...@gmx.de.INVALID>
Sent: Sunday, June 15, 2025 11:13:48 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.



Hi all,

took a long time digesting all the discussion thread. I think it would
be good to rewrite details to a new AIP so that it can be compared with
the old AIP.

I think this also could include the extension (or is this planned
otherwise?) to link multiple Airflow instances via Pub/Sub such that
dataset vents can be externalized... in some cases might be easier to
host multiple instances.

The different Envs as described below are looking good. Besides "Env"
this could also map to dedicated "executor" profiles e.g. instantiating
a dedicated executor per team? I think in regards of multiple executors
we always intended allowing to instantiate the same executor multiple
times. Then the Env could be mapped to an executor each.

Yeah and in this regard the triggerer would also need a feature
increment to be hosted with multiple instances. It might be worth
considering this is needed per executor anyway. I assume this is lagging
a bit, also for Edge Executor there is no async coverage.... so my
background with the assumption that different providers and tools might
be needed in triggerer would elan rather top Option 3) as sketched below.

Jens

On 14.06.25 08:21, Jarek Potiuk wrote:
> On Fri, Jun 13, 2025 at 7:09 PM Vincent Beck <vi...@apache.org> wrote:
>
>> Thanks, Jarek, for this proposal. Overall, I really like it—it
>> significantly simplifies multi-team support and removes the need to deploy
>> additional components per team, without compromising on the core needs of
>> users (unless I’m missing something).
>>
> Yep. I think with this "design" iteration, I put "simplicity" and
> "maintainability" as the primary goal. Separate configuration per tem goes
> out the window, ripple-effect on the DB goes out the window, what's left is
> basically the same Airflow we already have with few modifications.
>
>
>>> And if we do it and implement packaging and execution environments (say
>> ability of choosing predefined venv to parse and execute DAGs coming from a
>> specific bundle_id - the expectation 2) above can be handled well.
>>
>> Could you elaborate on this part? I’m not entirely clear on how it would
>> work in practice. For instance, how would it behave with two teams or
>> bundles? Real-world examples would help clarify this, unless it's more
>> implementation details that we can flesh out once there's agreement on the
>> general approach.
>>
> Currently with Bundle definition we **just** define where the DAGs are
> coming from. But we could (and that was even part of the original design)
> add extra "execution environment" configuration. For example when we have
> bundle_a and bundle_b each of them could have separate "environment"
> specified (say env_a, env_b) and we could map such environment to specific
> image (image_a, image_b) or virtualenv in the same image (/venv/a/ ,
> /venv/b) that would be predefined in the processor/worker images. (or in
> VMs if images are not used). The envs might have different sets of
> dependencies (providers and others) installed, and both DAG processor
> parsing and "Worker" (in celery or k8s Pod) would be run using that
> environment. Initially AIP-67 also discussed defining dependencies in
> bundle and installing it dynamically (like Python Venv Operator) - but I
> think personally having predefined set of environments rather than
> dynamically creating (like ExternalPythonOperator) has much better
> maintainability, stability and security properties.
>
>
>> Also, what about the triggerer? Since the triggerer runs user code, the
>> original AIP-67 proposal required at least one triggerer per team. How
>> would that be handled under this new architecture?
>>
> That is an excellent question :) .  There are few options - depending on
> how much of the point 4) "isolating workload" we want to implement.
> Paradoxically - to be honest-  for me, the Triggerer always had the
> potential of being less of a problem when it comes to isolation. Yes. All
> triggers are (currently) running not only in the same interpreter, but also
> in the same event loop (which means that isolation goes out of the window),
> but also it's relatively easy to introduce the isolation and we've been
> discussing options about it in the past as well. I see quite a few.
>
> Option 1) simplest operationally - We could add a mode in the Airflow that
> would resemble Timetables. All Triggers would have to be exposed via the
> plugin interface (we could easily expose all triggers this way from all our
> providers in a bulk way). This means that deployment manager will have
> control on what is run in Triggerrer - effectively limiting it similarly as
> Scheduler code today. That would prevent some of the other cases we
> discussed recently (such as sending "notification" serialized methods to
> triggerer to execute), but that's mostly optimization, and they could be
> sent as worker tasks instead in this case).
>
> Option 2) Semi-isolation - for a number of our users just separating
> processes might be "enough" (especially if we add cgroups to isolate
> the processes - we had that in the past). Not "perfect" and does not have
> all security properties, but for a number of our users it might be "good
> enough" because they will trust their teams enough to not worry about
> potential "malicious actions". In this case a single Triggerrer could run
> several event loops - one per bundle, each of them in a separate, isolated
> - process and the only change that we would have to do is to route the
> triggers to the right loop based on bundle id. Almost no operational
> complexity increases, but isolation is greatly improved. Again following
> the bundle -> environment mapping each of those processes could be run
> using a specific "per-bundle" environment where all necessary dependencies
> would be installed. And here the limit of arbitrary code execution coming
> from DAG can be lifted.
>
> Option 3) Full isolation -> simply run one triggerer per bundle. That is a
> bit more like the original proposal, because we will then have an extra
> triggerer for each bundle/team (or group of bundles - it does not have to
> be 1-to-1 mapping, could be many-to-1) . But it should provide full
> "security" properties with isolation and separation of workload, each
> triggerer could run completely in the same environment as defined in the
> bundle. It increases operational complexity - but just a bit. Rainbows and
> unicorns - we have it all.
>
> Also one more thing.
>
> We usually discuss technical aspects here in develist and rarely talk
> about "business". But I think this is in some cases wrong - including a
> multi-team that has the potential of either supporting or undermining some
> of the business our stakeholders do with Airflow.
>
> I would like to - really - make a collaborative effort to come up with a
> multi-team approach with all the stakeholders here - Amazon, Google,
> Astronomer especially should all be on-board with it. We know our users
> need it (survey and a number of talks about multi-team/tenancy that were
> submitted this year for Summit speak for themselves - we had ~10 sessions
> submitted about it this year, 30% of survey respondents want it - though of
> course as Ash correctly pointed out many of those people have different
> expectations). Again multi-team has the potential of either killing or
> supporting some of the business models our stakeholders might implement in
> their offering. And while here we do not "care" too much about those
> models, we should care about our stakeholders sustainability  - as they are
> the ones who are fueling Airflow in many ways - so it would be stupid if we
> do not consider their expectations and needs and - yes - sustainability of
> their businesses. Here in the community we mostly add features that can be
> used by everyone - whether in "as a service" or "on-prem" environment.  And
> we cannot "know" what the business is being planned or is possible or is
> good for our stakeholders. But we can collaboratively design the feature
> that might be usable on-prem - but one that we know is good for everyone
> and they can continue making business (or even better - provide better
> offerings to their users building on top of it.
>
> Let's do it. If there are things we can improve/make better here, I want to
> hear - from everyone - Ash, Vikram, Raj, Vincent, Rafał, Michał - if there
> is any idea how to improve it and make it better also for you - I think
> it's a good time to discuss it.
>
> J.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jens Scheffler <j_...@gmx.de.INVALID>]

Hi all,

took a long time digesting all the discussion thread. I think it would 
be good to rewrite details to a new AIP so that it can be compared with 
the old AIP.

I think this also could include the extension (or is this planned 
otherwise?) to link multiple Airflow instances via Pub/Sub such that 
dataset vents can be externalized... in some cases might be easier to 
host multiple instances.

The different Envs as described below are looking good. Besides "Env" 
this could also map to dedicated "executor" profiles e.g. instantiating 
a dedicated executor per team? I think in regards of multiple executors 
we always intended allowing to instantiate the same executor multiple 
times. Then the Env could be mapped to an executor each.

Yeah and in this regard the triggerer would also need a feature 
increment to be hosted with multiple instances. It might be worth 
considering this is needed per executor anyway. I assume this is lagging 
a bit, also for Edge Executor there is no async coverage.... so my 
background with the assumption that different providers and tools might 
be needed in triggerer would elan rather top Option 3) as sketched below.

Jens

On 14.06.25 08:21, Jarek Potiuk wrote:
> On Fri, Jun 13, 2025 at 7:09 PM Vincent Beck <vi...@apache.org> wrote:
>
>> Thanks, Jarek, for this proposal. Overall, I really like it—it
>> significantly simplifies multi-team support and removes the need to deploy
>> additional components per team, without compromising on the core needs of
>> users (unless I’m missing something).
>>
> Yep. I think with this "design" iteration, I put "simplicity" and
> "maintainability" as the primary goal. Separate configuration per tem goes
> out the window, ripple-effect on the DB goes out the window, what's left is
> basically the same Airflow we already have with few modifications.
>
>
>>> And if we do it and implement packaging and execution environments (say
>> ability of choosing predefined venv to parse and execute DAGs coming from a
>> specific bundle_id - the expectation 2) above can be handled well.
>>
>> Could you elaborate on this part? I’m not entirely clear on how it would
>> work in practice. For instance, how would it behave with two teams or
>> bundles? Real-world examples would help clarify this, unless it's more
>> implementation details that we can flesh out once there's agreement on the
>> general approach.
>>
> Currently with Bundle definition we **just** define where the DAGs are
> coming from. But we could (and that was even part of the original design)
> add extra "execution environment" configuration. For example when we have
> bundle_a and bundle_b each of them could have separate "environment"
> specified (say env_a, env_b) and we could map such environment to specific
> image (image_a, image_b) or virtualenv in the same image (/venv/a/ ,
> /venv/b) that would be predefined in the processor/worker images. (or in
> VMs if images are not used). The envs might have different sets of
> dependencies (providers and others) installed, and both DAG processor
> parsing and "Worker" (in celery or k8s Pod) would be run using that
> environment. Initially AIP-67 also discussed defining dependencies in
> bundle and installing it dynamically (like Python Venv Operator) - but I
> think personally having predefined set of environments rather than
> dynamically creating (like ExternalPythonOperator) has much better
> maintainability, stability and security properties.
>
>
>> Also, what about the triggerer? Since the triggerer runs user code, the
>> original AIP-67 proposal required at least one triggerer per team. How
>> would that be handled under this new architecture?
>>
> That is an excellent question :) .  There are few options - depending on
> how much of the point 4) "isolating workload" we want to implement.
> Paradoxically - to be honest-  for me, the Triggerer always had the
> potential of being less of a problem when it comes to isolation. Yes. All
> triggers are (currently) running not only in the same interpreter, but also
> in the same event loop (which means that isolation goes out of the window),
> but also it's relatively easy to introduce the isolation and we've been
> discussing options about it in the past as well. I see quite a few.
>
> Option 1) simplest operationally - We could add a mode in the Airflow that
> would resemble Timetables. All Triggers would have to be exposed via the
> plugin interface (we could easily expose all triggers this way from all our
> providers in a bulk way). This means that deployment manager will have
> control on what is run in Triggerrer - effectively limiting it similarly as
> Scheduler code today. That would prevent some of the other cases we
> discussed recently (such as sending "notification" serialized methods to
> triggerer to execute), but that's mostly optimization, and they could be
> sent as worker tasks instead in this case).
>
> Option 2) Semi-isolation - for a number of our users just separating
> processes might be "enough" (especially if we add cgroups to isolate
> the processes - we had that in the past). Not "perfect" and does not have
> all security properties, but for a number of our users it might be "good
> enough" because they will trust their teams enough to not worry about
> potential "malicious actions". In this case a single Triggerrer could run
> several event loops - one per bundle, each of them in a separate, isolated
> - process and the only change that we would have to do is to route the
> triggers to the right loop based on bundle id. Almost no operational
> complexity increases, but isolation is greatly improved. Again following
> the bundle -> environment mapping each of those processes could be run
> using a specific "per-bundle" environment where all necessary dependencies
> would be installed. And here the limit of arbitrary code execution coming
> from DAG can be lifted.
>
> Option 3) Full isolation -> simply run one triggerer per bundle. That is a
> bit more like the original proposal, because we will then have an extra
> triggerer for each bundle/team (or group of bundles - it does not have to
> be 1-to-1 mapping, could be many-to-1) . But it should provide full
> "security" properties with isolation and separation of workload, each
> triggerer could run completely in the same environment as defined in the
> bundle. It increases operational complexity - but just a bit. Rainbows and
> unicorns - we have it all.
>
> Also one more thing.
>
> We usually discuss technical aspects here in develist and rarely talk
> about "business". But I think this is in some cases wrong - including a
> multi-team that has the potential of either supporting or undermining some
> of the business our stakeholders do with Airflow.
>
> I would like to - really - make a collaborative effort to come up with a
> multi-team approach with all the stakeholders here - Amazon, Google,
> Astronomer especially should all be on-board with it. We know our users
> need it (survey and a number of talks about multi-team/tenancy that were
> submitted this year for Summit speak for themselves - we had ~10 sessions
> submitted about it this year, 30% of survey respondents want it - though of
> course as Ash correctly pointed out many of those people have different
> expectations). Again multi-team has the potential of either killing or
> supporting some of the business models our stakeholders might implement in
> their offering. And while here we do not "care" too much about those
> models, we should care about our stakeholders sustainability  - as they are
> the ones who are fueling Airflow in many ways - so it would be stupid if we
> do not consider their expectations and needs and - yes - sustainability of
> their businesses. Here in the community we mostly add features that can be
> used by everyone - whether in "as a service" or "on-prem" environment.  And
> we cannot "know" what the business is being planned or is possible or is
> good for our stakeholders. But we can collaboratively design the feature
> that might be usable on-prem - but one that we know is good for everyone
> and they can continue making business (or even better - provide better
> offerings to their users building on top of it.
>
> Let's do it. If there are things we can improve/make better here, I want to
> hear - from everyone - Ash, Vikram, Raj, Vincent, Rafał, Michał - if there
> is any idea how to improve it and make it better also for you - I think
> it's a good time to discuss it.
>
> J.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

On Fri, Jun 13, 2025 at 7:09 PM Vincent Beck <vi...@apache.org> wrote:

> Thanks, Jarek, for this proposal. Overall, I really like it—it
> significantly simplifies multi-team support and removes the need to deploy
> additional components per team, without compromising on the core needs of
> users (unless I’m missing something).
>

Yep. I think with this "design" iteration, I put "simplicity" and
"maintainability" as the primary goal. Separate configuration per tem goes
out the window, ripple-effect on the DB goes out the window, what's left is
basically the same Airflow we already have with few modifications.


> > And if we do it and implement packaging and execution environments (say
> ability of choosing predefined venv to parse and execute DAGs coming from a
> specific bundle_id - the expectation 2) above can be handled well.
>
> Could you elaborate on this part? I’m not entirely clear on how it would
> work in practice. For instance, how would it behave with two teams or
> bundles? Real-world examples would help clarify this, unless it's more
> implementation details that we can flesh out once there's agreement on the
> general approach.
>

Currently with Bundle definition we **just** define where the DAGs are
coming from. But we could (and that was even part of the original design)
add extra "execution environment" configuration. For example when we have
bundle_a and bundle_b each of them could have separate "environment"
specified (say env_a, env_b) and we could map such environment to specific
image (image_a, image_b) or virtualenv in the same image (/venv/a/ ,
/venv/b) that would be predefined in the processor/worker images. (or in
VMs if images are not used). The envs might have different sets of
dependencies (providers and others) installed, and both DAG processor
parsing and "Worker" (in celery or k8s Pod) would be run using that
environment. Initially AIP-67 also discussed defining dependencies in
bundle and installing it dynamically (like Python Venv Operator) - but I
think personally having predefined set of environments rather than
dynamically creating (like ExternalPythonOperator) has much better
maintainability, stability and security properties.


> Also, what about the triggerer? Since the triggerer runs user code, the
> original AIP-67 proposal required at least one triggerer per team. How
> would that be handled under this new architecture?
>

That is an excellent question :) .  There are few options - depending on
how much of the point 4) "isolating workload" we want to implement.
Paradoxically - to be honest-  for me, the Triggerer always had the
potential of being less of a problem when it comes to isolation. Yes. All
triggers are (currently) running not only in the same interpreter, but also
in the same event loop (which means that isolation goes out of the window),
but also it's relatively easy to introduce the isolation and we've been
discussing options about it in the past as well. I see quite a few.

Option 1) simplest operationally - We could add a mode in the Airflow that
would resemble Timetables. All Triggers would have to be exposed via the
plugin interface (we could easily expose all triggers this way from all our
providers in a bulk way). This means that deployment manager will have
control on what is run in Triggerrer - effectively limiting it similarly as
Scheduler code today. That would prevent some of the other cases we
discussed recently (such as sending "notification" serialized methods to
triggerer to execute), but that's mostly optimization, and they could be
sent as worker tasks instead in this case).

Option 2) Semi-isolation - for a number of our users just separating
processes might be "enough" (especially if we add cgroups to isolate
the processes - we had that in the past). Not "perfect" and does not have
all security properties, but for a number of our users it might be "good
enough" because they will trust their teams enough to not worry about
potential "malicious actions". In this case a single Triggerrer could run
several event loops - one per bundle, each of them in a separate, isolated
- process and the only change that we would have to do is to route the
triggers to the right loop based on bundle id. Almost no operational
complexity increases, but isolation is greatly improved. Again following
the bundle -> environment mapping each of those processes could be run
using a specific "per-bundle" environment where all necessary dependencies
would be installed. And here the limit of arbitrary code execution coming
from DAG can be lifted.

Option 3) Full isolation -> simply run one triggerer per bundle. That is a
bit more like the original proposal, because we will then have an extra
triggerer for each bundle/team (or group of bundles - it does not have to
be 1-to-1 mapping, could be many-to-1) . But it should provide full
"security" properties with isolation and separation of workload, each
triggerer could run completely in the same environment as defined in the
bundle. It increases operational complexity - but just a bit. Rainbows and
unicorns - we have it all.

Also one more thing.

We usually discuss technical aspects here in develist and rarely talk
about "business". But I think this is in some cases wrong - including a
multi-team that has the potential of either supporting or undermining some
of the business our stakeholders do with Airflow.

I would like to - really - make a collaborative effort to come up with a
multi-team approach with all the stakeholders here - Amazon, Google,
Astronomer especially should all be on-board with it. We know our users
need it (survey and a number of talks about multi-team/tenancy that were
submitted this year for Summit speak for themselves - we had ~10 sessions
submitted about it this year, 30% of survey respondents want it - though of
course as Ash correctly pointed out many of those people have different
expectations). Again multi-team has the potential of either killing or
supporting some of the business models our stakeholders might implement in
their offering. And while here we do not "care" too much about those
models, we should care about our stakeholders sustainability  - as they are
the ones who are fueling Airflow in many ways - so it would be stupid if we
do not consider their expectations and needs and - yes - sustainability of
their businesses. Here in the community we mostly add features that can be
used by everyone - whether in "as a service" or "on-prem" environment.  And
we cannot "know" what the business is being planned or is possible or is
good for our stakeholders. But we can collaboratively design the feature
that might be usable on-prem - but one that we know is good for everyone
and they can continue making business (or even better - provide better
offerings to their users building on top of it.

Let's do it. If there are things we can improve/make better here, I want to
hear - from everyone - Ash, Vikram, Raj, Vincent, Rafał, Michał - if there
is any idea how to improve it and make it better also for you - I think
it's a good time to discuss it.

J.

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Vincent Beck <vi...@apache.org>]

Thanks, Jarek, for this proposal. Overall, I really like it—it significantly simplifies multi-team support and removes the need to deploy additional components per team, without compromising on the core needs of users (unless I’m missing something).

> And if we do it and implement packaging and execution environments (say ability of choosing predefined venv to parse and execute DAGs coming from a specific bundle_id - the expectation 2) above can be handled well.

Could you elaborate on this part? I’m not entirely clear on how it would work in practice. For instance, how would it behave with two teams or bundles? Real-world examples would help clarify this, unless it's more implementation details that we can flesh out once there's agreement on the general approach.

Also, what about the triggerer? Since the triggerer runs user code, the original AIP-67 proposal required at least one triggerer per team. How would that be handled under this new architecture?

Vincent

On 2025/06/13 07:12:07 Jarek Potiuk wrote:
> OK. I spent some time thinking about it, I had to go back to the
> drawing board - which required me to clean my head of many of the previous
> assumptions and I think I figured out how we can provide "9+%" of what - at
> least I heard from the users who asked for it without adding the complexity
> that ripples through the whole airflow DB design.
> 
> *Cleaned assumptions:*
> 
> I think what the users **really** want is:
> 
> 1) first of all (and when I think about it now - this is more important) -
> to give the ability to control who of the dag authors have access to which
> Connections and Variables (hence our proposal was to add team_id to those
> in the DB). Especially connections because their credentials are the most
> sensitive part of Airflow installation.
> 2) have a group of DAGs use different set of dependencies installed when
> the DAGs are executed (most important) but also parsed (due to the way how
> in Airflow DAG parsing and task definition are all in single Python files).
> 3) have a way to limit access to see and monitor and operate on the DAGs
> via UI by different users (but this is far less important)
> 4) have a way to isolate workloads executed in Celery executor especially -
> so that tasks from a single "team" are not executed in the same
> machine/interpreter as the tasks for another team.
> 
> *Airflow 3.1+ environment and my new thinking*
> 
> With the few AIPs implemented (not yet fully and we might build on top of
> those) I think we might approach to solve the problems above a bit
> differently than having separate deployments of workers, triggers and dag
> processors (which as Ash pointed out is almost as complex as having whole
> Airflow deployment - or actually even more complex, because complexity of
> the whole airflow increases. So I put the question - can we leverage the
> currently implemented AIPs and changes in Airflow 3 to achieve those three
> goals without changes rippling through Airflow DB. I think we can.
> 
> Instead of introducing a team id concept, we could leverage AIP-67 - Dag
> Bundles and parsing to achieve pretty much the same. While currently DAG
> Bundles were designed to just provide different backends to retrieve them
> from, the initial concept and more complete version of those included
> "packaging" which could also be connected with "execution and parsing
> environment". I think we could treat "bundle_id" as "team_id". And if we do
> it and implement packaging and execution environments (say ability of
> choosing predefined venv to parse and execute DAGs coming from a specific
> bundle_id - the expectation 2) above can be handled well.
> 
> The same bundle id can be used to namespace connections and variables.
> There are several ways we can achieve that, but I believe we could now with
> Task SDK be cryptographically sure (with JWT tokens) sure that the requests
> to api server are coming from specific dag and find the bundle it comes
> from (even if not currently we can add specific claims to the tokens that
> we can verify). Which means that the API server can know which bundle id
> the DAG comes from when it asks for a connection and variables. Which means
> that we could extend Bundle definition and either explicitly - listing the
> connections and variables or implicitly - for example by prefix or adding
> just "bundle_id" to connections and variables - limit which connections can
> be retrieved when the task is being executed. That will achieve 1)
> 
> Same bundle_id could be passed as one of the auth manager "fields" instead
> of team_id - and make it the base for writing "bundle_id" aware auth
> managers (addressing 3) )
> 
> Similarly we could link bundle_id with celery queue_id and "hard link"
> those - i.e. do not allow to choose queue id to a different set of queues
> than the one defined for the bundle the dag comes from. That would allow
> the users to configure separate queues (and workers) for each DAG bundle.
> 
> I think with this approach all 4 points above can be achieved this way
> without a ripple effect on Airflow DB and complicating Airflow deployment.
> 
> J.
> 
> 
> 
> 
> On Thu, Jun 12, 2025 at 6:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> 
> > > As far as I can work out, and again, please correct me if I'm wrong, the
> > only real difference to users fro the multi-team solution over running
> > multiple Airflows is the ability to “communicate” via Datasets/Assets.
> > (Variables aren’t shared, Connections aren’t shared, workers aren’t shared.
> > Webserver and Scheduler could be/are shared but reducing resource
> > consumption of a deployment is explicitly not a goal)
> >
> > One other thing is the execution environment - i.e. set of dependencies
> > used to parse and execute Airflow DAGs. Which of course will become less of
> > a problem when we have task-sdk full separation but still is a concern of
> > users.
> >
> > J.
> >
> >
> > On Thu, Jun 12, 2025 at 5:51 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> >> > Other than the feature being a consistent request on our Airflow
> >> surveys, we have a number of users that have asked and continue to ask when
> >> a multi-team solution would be available in Airflow
> >>
> >> This is precisely one of my points. I don’t believe that AIP-67 as in the
> >> wiki page will address the need of many users asking for multi team. (Ask
> >> three Airflow users what they want from multi team, and you’ll get 5
> >> different answers)
> >>
> >> As far as I can work out, and again, please correct me if I'm wrong, the
> >> only real difference to users fro the multi-team solution over running
> >> multiple Airflows is the ability to “communicate” via Datasets/Assets.
> >> (Variables aren’t shared, Connections aren’t shared, workers aren’t shared.
> >> Webserver and Scheduler could be/are shared but reducing resource
> >> consumption of a deployment is explicitly not a goal)
> >>
> >> And if that is really all this AIP delivers to us, then my hypothesis is
> >> that we’ll a) miss the mark on what many users actually want from
> >> multi-team, and b) that we could already get the “communicate between two
> >> teams DAGs” benfit today with no changes to Airflow by using the
> >> AssetWatcher that Vincent already added to 3.0.0.
> >>
> >> -ash
> >>
> >>
> >> > On 12 Jun 2025, at 16:24, Bishundeo, Rajeshwar
> >> <rb...@amazon.com.INVALID> wrote:
> >> >
> >> > Ash, you've raised some good points on the need to re-evaluate AIP-67,
> >> although I'm a bit confused on how AIP-82 factors into a multi-team
> >> solution. It's fair to have the discussion on how Airflow has changed and
> >> perhaps either redefining what AIP-67 means...or a set of new AIP's solving
> >> a subset of a larger need.
> >> > We have seen talks at previous ( and even at the upcoming summit) where
> >> users have demonstrated their implementation of multi-team. I can't help
> >> but feel that they are being creative with some of those solutions (not
> >> that there's anything wrong with that), but because one doesn't exist in
> >> Airflow. Other than the feature being a consistent request on our Airflow
> >> surveys, we have a number of users that have asked and continue to ask when
> >> a multi-team solution would be available in Airflow.
> >> > I think the next dev call (06/26) is great time to dive into this
> >> further.
> >> >
> >> > -- Rajesh
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On 2025-06-12, 9:34 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> >> jarek@potiuk.com> <ma...@potiuk.com>> wrote:
> >> >
> >> >
> >> > CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you can confirm the sender and know
> >> the content is safe.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> >> certain que le contenu ne présente aucun risque.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Yep. It's a valid point that we should re-evaluate things now after
> >> Airflow
> >> > 3 is out - the reason why we delayed it was that we wanted to get more
> >> > clarity on what implementation and scope of the Airflow 3 changes will
> >> be
> >> > and see how it fits.
> >> >
> >> >
> >> > I wonder what others - especially those who run Airflow at scale and
> >> > hear the users asking for different forms of multi-team - would say to
> >> the
> >> > expectations they have and how they map to Airflow 3 - maybe indeed we
> >> > might come up with a simpler way of achieving those expectations.
> >> >
> >> >
> >> > Definitely worth discussing it.
> >> >
> >> >
> >> > J.
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Jun 12, 2025 at 2:15 PM Ash Berlin-Taylor <ash@apache.org
> >> <ma...@apache.org> <ma...@apache.org>> wrote:
> >> >
> >> >
> >> >> Hi everyone,
> >> >>
> >> >> One thing I’ve been struggling with while reading the other thread
> >> about
> >> >> multi-team DB changes[0] is what is the end-user problem we are trying
> >> to
> >> >> address with it.
> >> >>
> >> >> The main impetus for opening this discussion is that a lot has changed
> >> in
> >> >> Airflow since this AIP was created in early 2024 and voted on
> >> mid-2024, and
> >> >> I'm wondering if those changes are big enough to invalidate the design
> >> and
> >> >> assumptions made at the time.
> >> >>
> >> >> Reading the DB changes thread I see that that changes are far reaching
> >> and
> >> >> necessarily have to touch most of the Airflow object models, and this
> >> got
> >> >> me thinking about what value do we actually get with the change, since
> >> as
> >> >> stated in the AIP some of the non-goals are[1] (slightly edited here
> >> for
> >> >> brevity with the “[…]"):
> >> >>
> >> >>
> >> >>> • Sharing broker/backend for celery executors between teams. This MAY
> >> be
> >> >> covered by future AIPs
> >> >>> • Implementation of FAB-based multi-team Auth Manager. […]
> >> >>> • Per-team concurrency and prioritization of tasks. […].
> >> >>> • Resource allocation per-executor. In the current proposal, executors
> >> >> are run as sub-processes of Scheduler and we have very little control
> >> over
> >> >> their individual resource usage. […]
> >> >>> • Turn-key multi-team Deployment of Airflow (for example via Helm
> >> >> chart). This is unlikely to happen.[…]
> >> >>> • team management tools (creation, removal, rename etc.). […]
> >> >>> • Combining "global" execution with "team" execution. While it should
> >> be
> >> >> possible in the proposed architecture to have a "team" execution and
> >> >> "global" execution in a single instance of Airflow, this has it's own
> >> >> unique set of challenges and assumption is that Airflow Deployment is
> >> >> either "global" (today) or "multi-team" (After this AIP is
> >> implemented) -
> >> >> but it cannot be combined (yet). This is possible to be implemented in
> >> the
> >> >> future.
> >> >>> • Running multiple schedulers - one-per team. While it should be
> >> >> possible if we add support to select DAGs "per team" per scheduler,
> >> this is
> >> >> not implemented in this AIP and left for the future
> >> >>
> >> >> And also Design Non-goals from the AIP [2]:
> >> >>
> >> >>> • It’s not a primary goal of this proposal to significantly decrease
> >> >> resource consumption for Airflow installation compared to the current
> >> ways
> >> >> of achieving “multi-tenant” setup. […]
> >> >>> • It’s not a goal of the proposal to provide a one-stop installation
> >> >> mechanism for “Multi-team” Airflow. […]
> >> >>> • It’s not a goal to decrease the overall maintenance effort involved
> >> in
> >> >> responding to needs of different teams, […]
> >> >>
> >> >> The main pain point that we seem to be addressing with this AIP is
> >> this[3]:
> >> >>
> >> >>> The main reason for having multi-team deployment of Airflow is
> >> achieving
> >> >> security and isolation between the teams, coupled with ability of the
> >> >> isolated teams to collaborate via shared Datasets.
> >> >>
> >> >>
> >> >> So what’s changed since we collectively (myself included) voted on and
> >> >> accepted this AIP? Well, we now have AIP-82 — External event driven
> >> dags.
> >> >> That could be used to achieve this goal right now in 3.0 with no
> >> changes to
> >> >> Airflow itself, and is perhaps a more robust mechanism of doing it too.
> >> >>
> >> >> So my main question, given the wide reaching code changes need for
> >> AIP-67,
> >> >> and (IMO) the imperfect/limited scope of team completion I wonder if
> >> using
> >> >> AIP-82 would not be a better solution to the problem.
> >> >>
> >> >> 1. It’s much simpler from a code level, as nothing need to change
> >> >> 2. It’s not _that_ much more complex from an operational point of view
> >> >> (you have to run an extra scheduler and web server, but those would
> >> likely
> >> >> need scaling up.)
> >> >> 3. We won’t disappoint people by not implementing the part of
> >> multi-team
> >> >> that they want (Someone being part of multiple teams, sharing
> >> >> connections/vars between teams)
> >> >>
> >> >> And using this mechanism (of external dataset/asset polling) also
> >> negates
> >> >> one of the biggest cons of the AIP-67, that of the tight coupling of
> >> >> Airflow versions between the teams. In larger companies this is a
> >> _huge_
> >> >> problem already, and this would only make it worse.
> >> >>
> >> >> So what’s my idea (and at this stage is it only an idea for
> >> discussion) is
> >> >> that we re-evalute AIP-67 in light of what exists in Airflow 3.0 now
> >> and
> >> >> decide if it’s still worth the added complexity of DB, code and
> >> operational
> >> >> overhead, and decided if we still want it.
> >> >>
> >> >> Please, please, please point out if there are other benefits that I
> >> have
> >> >> missed, I'm not trying to be selective and get my way, I'm trying to
> >> make
> >> >> sure Airflow continues to meet the need of users, and can also
> >> continue to
> >> >> evolve (where I worry that complexity of code/datamodel materially
> >> hurts
> >> >> that final point)
> >> >>
> >> >> Thoughts?
> >> >>
> >> >> [0]: https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
> >> <https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c>
> >> >> [1]:
> >> >>
> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope
> >> <
> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope
> >> >
> >> >> ?
> >> >> [2]:
> >> >>
> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals
> >> <
> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals
> >> >
> >> >> [3]:
> >> >>
> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded
> >> <
> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded
> >> >
> >> >> ?
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> >> dev-unsubscribe@airflow.apache.org> <mailto:
> >> dev-unsubscribe@airflow.apache.org>
> >> >> For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> >> dev-help@airflow.apache.org> <ma...@airflow.apache.org>
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> >> dev-unsubscribe@airflow.apache.org>
> >> > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> >> dev-help@airflow.apache.org>
> >>
> >
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

OK. I spent some time thinking about it, I had to go back to the
drawing board - which required me to clean my head of many of the previous
assumptions and I think I figured out how we can provide "9+%" of what - at
least I heard from the users who asked for it without adding the complexity
that ripples through the whole airflow DB design.

*Cleaned assumptions:*

I think what the users **really** want is:

1) first of all (and when I think about it now - this is more important) -
to give the ability to control who of the dag authors have access to which
Connections and Variables (hence our proposal was to add team_id to those
in the DB). Especially connections because their credentials are the most
sensitive part of Airflow installation.
2) have a group of DAGs use different set of dependencies installed when
the DAGs are executed (most important) but also parsed (due to the way how
in Airflow DAG parsing and task definition are all in single Python files).
3) have a way to limit access to see and monitor and operate on the DAGs
via UI by different users (but this is far less important)
4) have a way to isolate workloads executed in Celery executor especially -
so that tasks from a single "team" are not executed in the same
machine/interpreter as the tasks for another team.

*Airflow 3.1+ environment and my new thinking*

With the few AIPs implemented (not yet fully and we might build on top of
those) I think we might approach to solve the problems above a bit
differently than having separate deployments of workers, triggers and dag
processors (which as Ash pointed out is almost as complex as having whole
Airflow deployment - or actually even more complex, because complexity of
the whole airflow increases. So I put the question - can we leverage the
currently implemented AIPs and changes in Airflow 3 to achieve those three
goals without changes rippling through Airflow DB. I think we can.

Instead of introducing a team id concept, we could leverage AIP-67 - Dag
Bundles and parsing to achieve pretty much the same. While currently DAG
Bundles were designed to just provide different backends to retrieve them
from, the initial concept and more complete version of those included
"packaging" which could also be connected with "execution and parsing
environment". I think we could treat "bundle_id" as "team_id". And if we do
it and implement packaging and execution environments (say ability of
choosing predefined venv to parse and execute DAGs coming from a specific
bundle_id - the expectation 2) above can be handled well.

The same bundle id can be used to namespace connections and variables.
There are several ways we can achieve that, but I believe we could now with
Task SDK be cryptographically sure (with JWT tokens) sure that the requests
to api server are coming from specific dag and find the bundle it comes
from (even if not currently we can add specific claims to the tokens that
we can verify). Which means that the API server can know which bundle id
the DAG comes from when it asks for a connection and variables. Which means
that we could extend Bundle definition and either explicitly - listing the
connections and variables or implicitly - for example by prefix or adding
just "bundle_id" to connections and variables - limit which connections can
be retrieved when the task is being executed. That will achieve 1)

Same bundle_id could be passed as one of the auth manager "fields" instead
of team_id - and make it the base for writing "bundle_id" aware auth
managers (addressing 3) )

Similarly we could link bundle_id with celery queue_id and "hard link"
those - i.e. do not allow to choose queue id to a different set of queues
than the one defined for the bundle the dag comes from. That would allow
the users to configure separate queues (and workers) for each DAG bundle.

I think with this approach all 4 points above can be achieved this way
without a ripple effect on Airflow DB and complicating Airflow deployment.

J.




On Thu, Jun 12, 2025 at 6:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> > As far as I can work out, and again, please correct me if I'm wrong, the
> only real difference to users fro the multi-team solution over running
> multiple Airflows is the ability to “communicate” via Datasets/Assets.
> (Variables aren’t shared, Connections aren’t shared, workers aren’t shared.
> Webserver and Scheduler could be/are shared but reducing resource
> consumption of a deployment is explicitly not a goal)
>
> One other thing is the execution environment - i.e. set of dependencies
> used to parse and execute Airflow DAGs. Which of course will become less of
> a problem when we have task-sdk full separation but still is a concern of
> users.
>
> J.
>
>
> On Thu, Jun 12, 2025 at 5:51 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> > Other than the feature being a consistent request on our Airflow
>> surveys, we have a number of users that have asked and continue to ask when
>> a multi-team solution would be available in Airflow
>>
>> This is precisely one of my points. I don’t believe that AIP-67 as in the
>> wiki page will address the need of many users asking for multi team. (Ask
>> three Airflow users what they want from multi team, and you’ll get 5
>> different answers)
>>
>> As far as I can work out, and again, please correct me if I'm wrong, the
>> only real difference to users fro the multi-team solution over running
>> multiple Airflows is the ability to “communicate” via Datasets/Assets.
>> (Variables aren’t shared, Connections aren’t shared, workers aren’t shared.
>> Webserver and Scheduler could be/are shared but reducing resource
>> consumption of a deployment is explicitly not a goal)
>>
>> And if that is really all this AIP delivers to us, then my hypothesis is
>> that we’ll a) miss the mark on what many users actually want from
>> multi-team, and b) that we could already get the “communicate between two
>> teams DAGs” benfit today with no changes to Airflow by using the
>> AssetWatcher that Vincent already added to 3.0.0.
>>
>> -ash
>>
>>
>> > On 12 Jun 2025, at 16:24, Bishundeo, Rajeshwar
>> <rb...@amazon.com.INVALID> wrote:
>> >
>> > Ash, you've raised some good points on the need to re-evaluate AIP-67,
>> although I'm a bit confused on how AIP-82 factors into a multi-team
>> solution. It's fair to have the discussion on how Airflow has changed and
>> perhaps either redefining what AIP-67 means...or a set of new AIP's solving
>> a subset of a larger need.
>> > We have seen talks at previous ( and even at the upcoming summit) where
>> users have demonstrated their implementation of multi-team. I can't help
>> but feel that they are being creative with some of those solutions (not
>> that there's anything wrong with that), but because one doesn't exist in
>> Airflow. Other than the feature being a consistent request on our Airflow
>> surveys, we have a number of users that have asked and continue to ask when
>> a multi-team solution would be available in Airflow.
>> > I think the next dev call (06/26) is great time to dive into this
>> further.
>> >
>> > -- Rajesh
>> >
>> >
>> >
>> >
>> >
>> >
>> > On 2025-06-12, 9:34 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
>> jarek@potiuk.com> <ma...@potiuk.com>> wrote:
>> >
>> >
>> > CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>> >
>> >
>> >
>> >
>> >
>> >
>> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>> certain que le contenu ne présente aucun risque.
>> >
>> >
>> >
>> >
>> >
>> >
>> > Yep. It's a valid point that we should re-evaluate things now after
>> Airflow
>> > 3 is out - the reason why we delayed it was that we wanted to get more
>> > clarity on what implementation and scope of the Airflow 3 changes will
>> be
>> > and see how it fits.
>> >
>> >
>> > I wonder what others - especially those who run Airflow at scale and
>> > hear the users asking for different forms of multi-team - would say to
>> the
>> > expectations they have and how they map to Airflow 3 - maybe indeed we
>> > might come up with a simpler way of achieving those expectations.
>> >
>> >
>> > Definitely worth discussing it.
>> >
>> >
>> > J.
>> >
>> >
>> >
>> >
>> > On Thu, Jun 12, 2025 at 2:15 PM Ash Berlin-Taylor <ash@apache.org
>> <ma...@apache.org> <ma...@apache.org>> wrote:
>> >
>> >
>> >> Hi everyone,
>> >>
>> >> One thing I’ve been struggling with while reading the other thread
>> about
>> >> multi-team DB changes[0] is what is the end-user problem we are trying
>> to
>> >> address with it.
>> >>
>> >> The main impetus for opening this discussion is that a lot has changed
>> in
>> >> Airflow since this AIP was created in early 2024 and voted on
>> mid-2024, and
>> >> I'm wondering if those changes are big enough to invalidate the design
>> and
>> >> assumptions made at the time.
>> >>
>> >> Reading the DB changes thread I see that that changes are far reaching
>> and
>> >> necessarily have to touch most of the Airflow object models, and this
>> got
>> >> me thinking about what value do we actually get with the change, since
>> as
>> >> stated in the AIP some of the non-goals are[1] (slightly edited here
>> for
>> >> brevity with the “[…]"):
>> >>
>> >>
>> >>> • Sharing broker/backend for celery executors between teams. This MAY
>> be
>> >> covered by future AIPs
>> >>> • Implementation of FAB-based multi-team Auth Manager. […]
>> >>> • Per-team concurrency and prioritization of tasks. […].
>> >>> • Resource allocation per-executor. In the current proposal, executors
>> >> are run as sub-processes of Scheduler and we have very little control
>> over
>> >> their individual resource usage. […]
>> >>> • Turn-key multi-team Deployment of Airflow (for example via Helm
>> >> chart). This is unlikely to happen.[…]
>> >>> • team management tools (creation, removal, rename etc.). […]
>> >>> • Combining "global" execution with "team" execution. While it should
>> be
>> >> possible in the proposed architecture to have a "team" execution and
>> >> "global" execution in a single instance of Airflow, this has it's own
>> >> unique set of challenges and assumption is that Airflow Deployment is
>> >> either "global" (today) or "multi-team" (After this AIP is
>> implemented) -
>> >> but it cannot be combined (yet). This is possible to be implemented in
>> the
>> >> future.
>> >>> • Running multiple schedulers - one-per team. While it should be
>> >> possible if we add support to select DAGs "per team" per scheduler,
>> this is
>> >> not implemented in this AIP and left for the future
>> >>
>> >> And also Design Non-goals from the AIP [2]:
>> >>
>> >>> • It’s not a primary goal of this proposal to significantly decrease
>> >> resource consumption for Airflow installation compared to the current
>> ways
>> >> of achieving “multi-tenant” setup. […]
>> >>> • It’s not a goal of the proposal to provide a one-stop installation
>> >> mechanism for “Multi-team” Airflow. […]
>> >>> • It’s not a goal to decrease the overall maintenance effort involved
>> in
>> >> responding to needs of different teams, […]
>> >>
>> >> The main pain point that we seem to be addressing with this AIP is
>> this[3]:
>> >>
>> >>> The main reason for having multi-team deployment of Airflow is
>> achieving
>> >> security and isolation between the teams, coupled with ability of the
>> >> isolated teams to collaborate via shared Datasets.
>> >>
>> >>
>> >> So what’s changed since we collectively (myself included) voted on and
>> >> accepted this AIP? Well, we now have AIP-82 — External event driven
>> dags.
>> >> That could be used to achieve this goal right now in 3.0 with no
>> changes to
>> >> Airflow itself, and is perhaps a more robust mechanism of doing it too.
>> >>
>> >> So my main question, given the wide reaching code changes need for
>> AIP-67,
>> >> and (IMO) the imperfect/limited scope of team completion I wonder if
>> using
>> >> AIP-82 would not be a better solution to the problem.
>> >>
>> >> 1. It’s much simpler from a code level, as nothing need to change
>> >> 2. It’s not _that_ much more complex from an operational point of view
>> >> (you have to run an extra scheduler and web server, but those would
>> likely
>> >> need scaling up.)
>> >> 3. We won’t disappoint people by not implementing the part of
>> multi-team
>> >> that they want (Someone being part of multiple teams, sharing
>> >> connections/vars between teams)
>> >>
>> >> And using this mechanism (of external dataset/asset polling) also
>> negates
>> >> one of the biggest cons of the AIP-67, that of the tight coupling of
>> >> Airflow versions between the teams. In larger companies this is a
>> _huge_
>> >> problem already, and this would only make it worse.
>> >>
>> >> So what’s my idea (and at this stage is it only an idea for
>> discussion) is
>> >> that we re-evalute AIP-67 in light of what exists in Airflow 3.0 now
>> and
>> >> decide if it’s still worth the added complexity of DB, code and
>> operational
>> >> overhead, and decided if we still want it.
>> >>
>> >> Please, please, please point out if there are other benefits that I
>> have
>> >> missed, I'm not trying to be selective and get my way, I'm trying to
>> make
>> >> sure Airflow continues to meet the need of users, and can also
>> continue to
>> >> evolve (where I worry that complexity of code/datamodel materially
>> hurts
>> >> that final point)
>> >>
>> >> Thoughts?
>> >>
>> >> [0]: https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
>> <https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c>
>> >> [1]:
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope
>> <
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope
>> >
>> >> ?
>> >> [2]:
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals
>> <
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals
>> >
>> >> [3]:
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded
>> <
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded
>> >
>> >> ?
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
>> dev-unsubscribe@airflow.apache.org> <mailto:
>> dev-unsubscribe@airflow.apache.org>
>> >> For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
>> dev-help@airflow.apache.org> <ma...@airflow.apache.org>
>> >>
>> >>
>> >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
>> dev-unsubscribe@airflow.apache.org>
>> > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
>> dev-help@airflow.apache.org>
>>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

> As far as I can work out, and again, please correct me if I'm wrong, the
only real difference to users fro the multi-team solution over running
multiple Airflows is the ability to “communicate” via Datasets/Assets.
(Variables aren’t shared, Connections aren’t shared, workers aren’t shared.
Webserver and Scheduler could be/are shared but reducing resource
consumption of a deployment is explicitly not a goal)

One other thing is the execution environment - i.e. set of dependencies
used to parse and execute Airflow DAGs. Which of course will become less of
a problem when we have task-sdk full separation but still is a concern of
users.

J.


On Thu, Jun 12, 2025 at 5:51 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> > Other than the feature being a consistent request on our Airflow
> surveys, we have a number of users that have asked and continue to ask when
> a multi-team solution would be available in Airflow
>
> This is precisely one of my points. I don’t believe that AIP-67 as in the
> wiki page will address the need of many users asking for multi team. (Ask
> three Airflow users what they want from multi team, and you’ll get 5
> different answers)
>
> As far as I can work out, and again, please correct me if I'm wrong, the
> only real difference to users fro the multi-team solution over running
> multiple Airflows is the ability to “communicate” via Datasets/Assets.
> (Variables aren’t shared, Connections aren’t shared, workers aren’t shared.
> Webserver and Scheduler could be/are shared but reducing resource
> consumption of a deployment is explicitly not a goal)
>
> And if that is really all this AIP delivers to us, then my hypothesis is
> that we’ll a) miss the mark on what many users actually want from
> multi-team, and b) that we could already get the “communicate between two
> teams DAGs” benfit today with no changes to Airflow by using the
> AssetWatcher that Vincent already added to 3.0.0.
>
> -ash
>
>
> > On 12 Jun 2025, at 16:24, Bishundeo, Rajeshwar
> <rb...@amazon.com.INVALID> wrote:
> >
> > Ash, you've raised some good points on the need to re-evaluate AIP-67,
> although I'm a bit confused on how AIP-82 factors into a multi-team
> solution. It's fair to have the discussion on how Airflow has changed and
> perhaps either redefining what AIP-67 means...or a set of new AIP's solving
> a subset of a larger need.
> > We have seen talks at previous ( and even at the upcoming summit) where
> users have demonstrated their implementation of multi-team. I can't help
> but feel that they are being creative with some of those solutions (not
> that there's anything wrong with that), but because one doesn't exist in
> Airflow. Other than the feature being a consistent request on our Airflow
> surveys, we have a number of users that have asked and continue to ask when
> a multi-team solution would be available in Airflow.
> > I think the next dev call (06/26) is great time to dive into this
> further.
> >
> > -- Rajesh
> >
> >
> >
> >
> >
> >
> > On 2025-06-12, 9:34 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> jarek@potiuk.com> <ma...@potiuk.com>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain que le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > Yep. It's a valid point that we should re-evaluate things now after
> Airflow
> > 3 is out - the reason why we delayed it was that we wanted to get more
> > clarity on what implementation and scope of the Airflow 3 changes will be
> > and see how it fits.
> >
> >
> > I wonder what others - especially those who run Airflow at scale and
> > hear the users asking for different forms of multi-team - would say to
> the
> > expectations they have and how they map to Airflow 3 - maybe indeed we
> > might come up with a simpler way of achieving those expectations.
> >
> >
> > Definitely worth discussing it.
> >
> >
> > J.
> >
> >
> >
> >
> > On Thu, Jun 12, 2025 at 2:15 PM Ash Berlin-Taylor <ash@apache.org
> <ma...@apache.org> <ma...@apache.org>> wrote:
> >
> >
> >> Hi everyone,
> >>
> >> One thing I’ve been struggling with while reading the other thread about
> >> multi-team DB changes[0] is what is the end-user problem we are trying
> to
> >> address with it.
> >>
> >> The main impetus for opening this discussion is that a lot has changed
> in
> >> Airflow since this AIP was created in early 2024 and voted on mid-2024,
> and
> >> I'm wondering if those changes are big enough to invalidate the design
> and
> >> assumptions made at the time.
> >>
> >> Reading the DB changes thread I see that that changes are far reaching
> and
> >> necessarily have to touch most of the Airflow object models, and this
> got
> >> me thinking about what value do we actually get with the change, since
> as
> >> stated in the AIP some of the non-goals are[1] (slightly edited here for
> >> brevity with the “[…]"):
> >>
> >>
> >>> • Sharing broker/backend for celery executors between teams. This MAY
> be
> >> covered by future AIPs
> >>> • Implementation of FAB-based multi-team Auth Manager. […]
> >>> • Per-team concurrency and prioritization of tasks. […].
> >>> • Resource allocation per-executor. In the current proposal, executors
> >> are run as sub-processes of Scheduler and we have very little control
> over
> >> their individual resource usage. […]
> >>> • Turn-key multi-team Deployment of Airflow (for example via Helm
> >> chart). This is unlikely to happen.[…]
> >>> • team management tools (creation, removal, rename etc.). […]
> >>> • Combining "global" execution with "team" execution. While it should
> be
> >> possible in the proposed architecture to have a "team" execution and
> >> "global" execution in a single instance of Airflow, this has it's own
> >> unique set of challenges and assumption is that Airflow Deployment is
> >> either "global" (today) or "multi-team" (After this AIP is implemented)
> -
> >> but it cannot be combined (yet). This is possible to be implemented in
> the
> >> future.
> >>> • Running multiple schedulers - one-per team. While it should be
> >> possible if we add support to select DAGs "per team" per scheduler,
> this is
> >> not implemented in this AIP and left for the future
> >>
> >> And also Design Non-goals from the AIP [2]:
> >>
> >>> • It’s not a primary goal of this proposal to significantly decrease
> >> resource consumption for Airflow installation compared to the current
> ways
> >> of achieving “multi-tenant” setup. […]
> >>> • It’s not a goal of the proposal to provide a one-stop installation
> >> mechanism for “Multi-team” Airflow. […]
> >>> • It’s not a goal to decrease the overall maintenance effort involved
> in
> >> responding to needs of different teams, […]
> >>
> >> The main pain point that we seem to be addressing with this AIP is
> this[3]:
> >>
> >>> The main reason for having multi-team deployment of Airflow is
> achieving
> >> security and isolation between the teams, coupled with ability of the
> >> isolated teams to collaborate via shared Datasets.
> >>
> >>
> >> So what’s changed since we collectively (myself included) voted on and
> >> accepted this AIP? Well, we now have AIP-82 — External event driven
> dags.
> >> That could be used to achieve this goal right now in 3.0 with no
> changes to
> >> Airflow itself, and is perhaps a more robust mechanism of doing it too.
> >>
> >> So my main question, given the wide reaching code changes need for
> AIP-67,
> >> and (IMO) the imperfect/limited scope of team completion I wonder if
> using
> >> AIP-82 would not be a better solution to the problem.
> >>
> >> 1. It’s much simpler from a code level, as nothing need to change
> >> 2. It’s not _that_ much more complex from an operational point of view
> >> (you have to run an extra scheduler and web server, but those would
> likely
> >> need scaling up.)
> >> 3. We won’t disappoint people by not implementing the part of multi-team
> >> that they want (Someone being part of multiple teams, sharing
> >> connections/vars between teams)
> >>
> >> And using this mechanism (of external dataset/asset polling) also
> negates
> >> one of the biggest cons of the AIP-67, that of the tight coupling of
> >> Airflow versions between the teams. In larger companies this is a _huge_
> >> problem already, and this would only make it worse.
> >>
> >> So what’s my idea (and at this stage is it only an idea for discussion)
> is
> >> that we re-evalute AIP-67 in light of what exists in Airflow 3.0 now and
> >> decide if it’s still worth the added complexity of DB, code and
> operational
> >> overhead, and decided if we still want it.
> >>
> >> Please, please, please point out if there are other benefits that I have
> >> missed, I'm not trying to be selective and get my way, I'm trying to
> make
> >> sure Airflow continues to meet the need of users, and can also continue
> to
> >> evolve (where I worry that complexity of code/datamodel materially hurts
> >> that final point)
> >>
> >> Thoughts?
> >>
> >> [0]: https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c <
> https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c>
> >> [1]:
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope
> <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope
> >
> >> ?
> >> [2]:
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals
> <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals
> >
> >> [3]:
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded
> <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded
> >
> >> ?
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org> <mailto:
> dev-unsubscribe@airflow.apache.org>
> >> For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org> <ma...@airflow.apache.org>
> >>
> >>
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>
> > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Ash Berlin-Taylor <as...@apache.org>]

> Other than the feature being a consistent request on our Airflow surveys, we have a number of users that have asked and continue to ask when a multi-team solution would be available in Airflow

This is precisely one of my points. I don’t believe that AIP-67 as in the wiki page will address the need of many users asking for multi team. (Ask three Airflow users what they want from multi team, and you’ll get 5 different answers)

As far as I can work out, and again, please correct me if I'm wrong, the only real difference to users fro the multi-team solution over running multiple Airflows is the ability to “communicate” via Datasets/Assets. (Variables aren’t shared, Connections aren’t shared, workers aren’t shared. Webserver and Scheduler could be/are shared but reducing resource consumption of a deployment is explicitly not a goal)

And if that is really all this AIP delivers to us, then my hypothesis is that we’ll a) miss the mark on what many users actually want from multi-team, and b) that we could already get the “communicate between two teams DAGs” benfit today with no changes to Airflow by using the AssetWatcher that Vincent already added to 3.0.0. 

-ash


> On 12 Jun 2025, at 16:24, Bishundeo, Rajeshwar <rb...@amazon.com.INVALID> wrote:
> 
> Ash, you've raised some good points on the need to re-evaluate AIP-67, although I'm a bit confused on how AIP-82 factors into a multi-team solution. It's fair to have the discussion on how Airflow has changed and perhaps either redefining what AIP-67 means...or a set of new AIP's solving a subset of a larger need. 
> We have seen talks at previous ( and even at the upcoming summit) where users have demonstrated their implementation of multi-team. I can't help but feel that they are being creative with some of those solutions (not that there's anything wrong with that), but because one doesn't exist in Airflow. Other than the feature being a consistent request on our Airflow surveys, we have a number of users that have asked and continue to ask when a multi-team solution would be available in Airflow.
> I think the next dev call (06/26) is great time to dive into this further. 
> 
> -- Rajesh 
> 
> 
> 
> 
> 
> 
> On 2025-06-12, 9:34 AM, "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com> <ma...@potiuk.com>> wrote:
> 
> 
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> 
> 
> 
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
> 
> 
> 
> 
> 
> 
> Yep. It's a valid point that we should re-evaluate things now after Airflow
> 3 is out - the reason why we delayed it was that we wanted to get more
> clarity on what implementation and scope of the Airflow 3 changes will be
> and see how it fits.
> 
> 
> I wonder what others - especially those who run Airflow at scale and
> hear the users asking for different forms of multi-team - would say to the
> expectations they have and how they map to Airflow 3 - maybe indeed we
> might come up with a simpler way of achieving those expectations.
> 
> 
> Definitely worth discussing it.
> 
> 
> J.
> 
> 
> 
> 
> On Thu, Jun 12, 2025 at 2:15 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org> <ma...@apache.org>> wrote:
> 
> 
>> Hi everyone,
>> 
>> One thing I’ve been struggling with while reading the other thread about
>> multi-team DB changes[0] is what is the end-user problem we are trying to
>> address with it.
>> 
>> The main impetus for opening this discussion is that a lot has changed in
>> Airflow since this AIP was created in early 2024 and voted on mid-2024, and
>> I'm wondering if those changes are big enough to invalidate the design and
>> assumptions made at the time.
>> 
>> Reading the DB changes thread I see that that changes are far reaching and
>> necessarily have to touch most of the Airflow object models, and this got
>> me thinking about what value do we actually get with the change, since as
>> stated in the AIP some of the non-goals are[1] (slightly edited here for
>> brevity with the “[…]"):
>> 
>> 
>>> • Sharing broker/backend for celery executors between teams. This MAY be
>> covered by future AIPs
>>> • Implementation of FAB-based multi-team Auth Manager. […]
>>> • Per-team concurrency and prioritization of tasks. […].
>>> • Resource allocation per-executor. In the current proposal, executors
>> are run as sub-processes of Scheduler and we have very little control over
>> their individual resource usage. […]
>>> • Turn-key multi-team Deployment of Airflow (for example via Helm
>> chart). This is unlikely to happen.[…]
>>> • team management tools (creation, removal, rename etc.). […]
>>> • Combining "global" execution with "team" execution. While it should be
>> possible in the proposed architecture to have a "team" execution and
>> "global" execution in a single instance of Airflow, this has it's own
>> unique set of challenges and assumption is that Airflow Deployment is
>> either "global" (today) or "multi-team" (After this AIP is implemented) -
>> but it cannot be combined (yet). This is possible to be implemented in the
>> future.
>>> • Running multiple schedulers - one-per team. While it should be
>> possible if we add support to select DAGs "per team" per scheduler, this is
>> not implemented in this AIP and left for the future
>> 
>> And also Design Non-goals from the AIP [2]:
>> 
>>> • It’s not a primary goal of this proposal to significantly decrease
>> resource consumption for Airflow installation compared to the current ways
>> of achieving “multi-tenant” setup. […]
>>> • It’s not a goal of the proposal to provide a one-stop installation
>> mechanism for “Multi-team” Airflow. […]
>>> • It’s not a goal to decrease the overall maintenance effort involved in
>> responding to needs of different teams, […]
>> 
>> The main pain point that we seem to be addressing with this AIP is this[3]:
>> 
>>> The main reason for having multi-team deployment of Airflow is achieving
>> security and isolation between the teams, coupled with ability of the
>> isolated teams to collaborate via shared Datasets.
>> 
>> 
>> So what’s changed since we collectively (myself included) voted on and
>> accepted this AIP? Well, we now have AIP-82 — External event driven dags.
>> That could be used to achieve this goal right now in 3.0 with no changes to
>> Airflow itself, and is perhaps a more robust mechanism of doing it too.
>> 
>> So my main question, given the wide reaching code changes need for AIP-67,
>> and (IMO) the imperfect/limited scope of team completion I wonder if using
>> AIP-82 would not be a better solution to the problem.
>> 
>> 1. It’s much simpler from a code level, as nothing need to change
>> 2. It’s not _that_ much more complex from an operational point of view
>> (you have to run an extra scheduler and web server, but those would likely
>> need scaling up.)
>> 3. We won’t disappoint people by not implementing the part of multi-team
>> that they want (Someone being part of multiple teams, sharing
>> connections/vars between teams)
>> 
>> And using this mechanism (of external dataset/asset polling) also negates
>> one of the biggest cons of the AIP-67, that of the tight coupling of
>> Airflow versions between the teams. In larger companies this is a _huge_
>> problem already, and this would only make it worse.
>> 
>> So what’s my idea (and at this stage is it only an idea for discussion) is
>> that we re-evalute AIP-67 in light of what exists in Airflow 3.0 now and
>> decide if it’s still worth the added complexity of DB, code and operational
>> overhead, and decided if we still want it.
>> 
>> Please, please, please point out if there are other benefits that I have
>> missed, I'm not trying to be selective and get my way, I'm trying to make
>> sure Airflow continues to meet the need of users, and can also continue to
>> evolve (where I worry that complexity of code/datamodel materially hurts
>> that final point)
>> 
>> Thoughts?
>> 
>> [0]: https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c <https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c>
>> [1]:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope>
>> ?
>> [2]:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals>
>> [3]:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded>
>> ?
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <ma...@airflow.apache.org>
>> For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org> <ma...@airflow.apache.org>
>> 
>> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
> For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

["Bishundeo, Rajeshwar" <rb...@amazon.com.INVALID>]

Ash, you've raised some good points on the need to re-evaluate AIP-67, although I'm a bit confused on how AIP-82 factors into a multi-team solution. It's fair to have the discussion on how Airflow has changed and perhaps either redefining what AIP-67 means...or a set of new AIP's solving a subset of a larger need. 
We have seen talks at previous ( and even at the upcoming summit) where users have demonstrated their implementation of multi-team. I can't help but feel that they are being creative with some of those solutions (not that there's anything wrong with that), but because one doesn't exist in Airflow. Other than the feature being a consistent request on our Airflow surveys, we have a number of users that have asked and continue to ask when a multi-team solution would be available in Airflow.
I think the next dev call (06/26) is great time to dive into this further. 

-- Rajesh 






On 2025-06-12, 9:34 AM, "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>> wrote:


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.






AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.






Yep. It's a valid point that we should re-evaluate things now after Airflow
3 is out - the reason why we delayed it was that we wanted to get more
clarity on what implementation and scope of the Airflow 3 changes will be
and see how it fits.


I wonder what others - especially those who run Airflow at scale and
hear the users asking for different forms of multi-team - would say to the
expectations they have and how they map to Airflow 3 - maybe indeed we
might come up with a simpler way of achieving those expectations.


Definitely worth discussing it.


J.




On Thu, Jun 12, 2025 at 2:15 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> wrote:


> Hi everyone,
>
> One thing I’ve been struggling with while reading the other thread about
> multi-team DB changes[0] is what is the end-user problem we are trying to
> address with it.
>
> The main impetus for opening this discussion is that a lot has changed in
> Airflow since this AIP was created in early 2024 and voted on mid-2024, and
> I'm wondering if those changes are big enough to invalidate the design and
> assumptions made at the time.
>
> Reading the DB changes thread I see that that changes are far reaching and
> necessarily have to touch most of the Airflow object models, and this got
> me thinking about what value do we actually get with the change, since as
> stated in the AIP some of the non-goals are[1] (slightly edited here for
> brevity with the “[…]"):
>
>
> > • Sharing broker/backend for celery executors between teams. This MAY be
> covered by future AIPs
> > • Implementation of FAB-based multi-team Auth Manager. […]
> > • Per-team concurrency and prioritization of tasks. […].
> > • Resource allocation per-executor. In the current proposal, executors
> are run as sub-processes of Scheduler and we have very little control over
> their individual resource usage. […]
> > • Turn-key multi-team Deployment of Airflow (for example via Helm
> chart). This is unlikely to happen.[…]
> > • team management tools (creation, removal, rename etc.). […]
> > • Combining "global" execution with "team" execution. While it should be
> possible in the proposed architecture to have a "team" execution and
> "global" execution in a single instance of Airflow, this has it's own
> unique set of challenges and assumption is that Airflow Deployment is
> either "global" (today) or "multi-team" (After this AIP is implemented) -
> but it cannot be combined (yet). This is possible to be implemented in the
> future.
> > • Running multiple schedulers - one-per team. While it should be
> possible if we add support to select DAGs "per team" per scheduler, this is
> not implemented in this AIP and left for the future
>
> And also Design Non-goals from the AIP [2]:
>
> > • It’s not a primary goal of this proposal to significantly decrease
> resource consumption for Airflow installation compared to the current ways
> of achieving “multi-tenant” setup. […]
> > • It’s not a goal of the proposal to provide a one-stop installation
> mechanism for “Multi-team” Airflow. […]
> > • It’s not a goal to decrease the overall maintenance effort involved in
> responding to needs of different teams, […]
>
> The main pain point that we seem to be addressing with this AIP is this[3]:
>
> > The main reason for having multi-team deployment of Airflow is achieving
> security and isolation between the teams, coupled with ability of the
> isolated teams to collaborate via shared Datasets.
>
>
> So what’s changed since we collectively (myself included) voted on and
> accepted this AIP? Well, we now have AIP-82 — External event driven dags.
> That could be used to achieve this goal right now in 3.0 with no changes to
> Airflow itself, and is perhaps a more robust mechanism of doing it too.
>
> So my main question, given the wide reaching code changes need for AIP-67,
> and (IMO) the imperfect/limited scope of team completion I wonder if using
> AIP-82 would not be a better solution to the problem.
>
> 1. It’s much simpler from a code level, as nothing need to change
> 2. It’s not _that_ much more complex from an operational point of view
> (you have to run an extra scheduler and web server, but those would likely
> need scaling up.)
> 3. We won’t disappoint people by not implementing the part of multi-team
> that they want (Someone being part of multiple teams, sharing
> connections/vars between teams)
>
> And using this mechanism (of external dataset/asset polling) also negates
> one of the biggest cons of the AIP-67, that of the tight coupling of
> Airflow versions between the teams. In larger companies this is a _huge_
> problem already, and this would only make it worse.
>
> So what’s my idea (and at this stage is it only an idea for discussion) is
> that we re-evalute AIP-67 in light of what exists in Airflow 3.0 now and
> decide if it’s still worth the added complexity of DB, code and operational
> overhead, and decided if we still want it.
>
> Please, please, please point out if there are other benefits that I have
> missed, I'm not trying to be selective and get my way, I'm trying to make
> sure Airflow continues to meet the need of users, and can also continue to
> evolve (where I worry that complexity of code/datamodel materially hurts
> that final point)
>
> Thoughts?
>
> [0]: https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c <https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c>
> [1]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope>
> ?
> [2]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals>
> [3]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded>
> ?
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
> For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>
>
>

Re: Discuss: AIP-67 (multi team) now that AIP-82 (External event driven dags) exists

[Jarek Potiuk <ja...@potiuk.com>]

Yep. It's a valid point that we should re-evaluate things now after Airflow
3 is out - the reason why we delayed it was that we wanted to get more
clarity on what implementation and scope of the Airflow 3 changes will be
and see how it fits.

I wonder what others - especially those who run Airflow at scale and
hear the users asking for different forms of multi-team - would say to the
expectations they have and how they map to Airflow 3 - maybe indeed we
might come up with a simpler way of achieving those expectations.

Definitely worth discussing it.

J.


On Thu, Jun 12, 2025 at 2:15 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> Hi everyone,
>
> One thing I’ve been struggling with while reading the other thread about
> multi-team DB changes[0] is what is the end-user problem we are trying to
> address with it.
>
> The main impetus for opening this discussion is that a lot has changed in
> Airflow since this AIP was created in early 2024 and voted on mid-2024, and
> I'm wondering if those changes are big enough to invalidate the design and
> assumptions made at the time.
>
> Reading the DB changes thread I see that that changes are far reaching and
> necessarily have to touch most of the Airflow object models, and this got
> me thinking about what value do we actually get with the change, since as
> stated in the AIP some of the non-goals are[1] (slightly edited here for
> brevity with the “[…]"):
>
>
> > • Sharing broker/backend for celery executors between teams. This MAY be
> covered by future AIPs
> > • Implementation of FAB-based multi-team Auth Manager. […]
> > • Per-team concurrency and prioritization of tasks. […].
> > • Resource allocation per-executor. In the current proposal, executors
> are run as sub-processes of Scheduler and we have very little control over
> their individual resource usage. […]
> > • Turn-key multi-team Deployment of Airflow (for example via Helm
> chart). This is unlikely to happen.[…]
> > • team management tools (creation, removal, rename etc.). […]
> > • Combining "global" execution with "team" execution. While it should be
> possible in the proposed architecture to have a "team" execution and
> "global" execution in a single instance of Airflow, this has it's own
> unique set of challenges and assumption is that Airflow Deployment is
> either "global" (today) or "multi-team" (After this AIP is implemented) -
> but it cannot be combined (yet). This is possible to be implemented in the
> future.
> > • Running multiple schedulers - one-per team. While it should be
> possible if we add support to select DAGs "per team" per scheduler, this is
> not implemented in this AIP and left for the future
>
> And also Design Non-goals from the AIP [2]:
>
> > • It’s not a primary goal of this proposal to significantly decrease
> resource consumption for Airflow installation compared to the current ways
> of achieving “multi-tenant” setup. […]
> > • It’s not a goal of the proposal to provide a one-stop installation
> mechanism for “Multi-team” Airflow. […]
> > • It’s not a goal to decrease the overall maintenance effort involved in
> responding to needs of different teams, […]
>
> The main pain point that we seem to be addressing with this AIP is this[3]:
>
> > The main reason for having multi-team deployment of Airflow is achieving
> security and isolation between the teams, coupled with ability of the
> isolated teams to collaborate via shared Datasets.
>
>
> So what’s changed since we collectively (myself included) voted on and
> accepted this AIP? Well, we now have AIP-82 — External event driven dags.
> That could be used to achieve this goal right now in 3.0 with no changes to
> Airflow itself, and is perhaps a more robust mechanism of doing it too.
>
> So my main question, given the wide reaching code changes need for AIP-67,
> and (IMO) the imperfect/limited scope of team completion I wonder if using
> AIP-82 would not be a better solution to the problem.
>
> 1. It’s much simpler from a code level, as nothing need to change
> 2. It’s not _that_ much more complex from an operational point of view
> (you have to run an extra scheduler and web server,  but those would likely
> need scaling up.)
> 3. We won’t disappoint people by not implementing the part of multi-team
> that they want (Someone being part of multiple teams, sharing
> connections/vars between teams)
>
> And using this mechanism (of external dataset/asset polling) also negates
> one of the biggest cons of the AIP-67, that of the tight coupling of
> Airflow versions between the teams. In larger companies this is a _huge_
> problem already, and this would only make it worse.
>
> So what’s my idea (and at this stage is it only an idea for discussion) is
> that we re-evalute AIP-67 in light of what exists in Airflow 3.0 now and
> decide if it’s still worth the added complexity of DB, code and operational
> overhead, and decided if we still want it.
>
> Please, please, please point out if there are other benefits that I have
> missed, I'm not trying to be selective and get my way, I'm trying to make
> sure Airflow continues to meet the need of users, and can also continue to
> evolve (where I worry that complexity of code/datamodel materially hurts
> that final point)
>
> Thoughts?
>
> [0]: https://lists.apache.org/thread/78vndnybgpp705j6sm77l1t6xbrtnt5c
> [1]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whatisexcludedfromthescope
> ?
> [2]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-DesignNonGoals
> [3]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=294816378#AIP67MultiteamdeploymentofAirflowcomponents-Whyisitneeded
> ?
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>