You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2024/07/09 14:48:13 UTC
[DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Hello Everyone,
I would like to resume discussion on AIP-67. After going through a
number of discussions and clarifications about the scope of Airflow 3,
I rewrote the proposal for AIP-67 with the assumption that we will do
it for Airflow 3 only - and that it will be based on the new proposed
AIP-72 (Task Execution Interface) rather than Airflow 2-only AIP-44
Internal API.
The updated proposal is here
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
Feel free to comment there in-line or raise your "big" comments" here,
but here is the impact of changing the target to Airflow 3:
1) I proposed to change configuration of Airflow to use more
structured TOML than plain "ini" - toml is a successor of "ini" and is
largely compatible, but it has arrays, tables and nesting, has good
support in Python and is "de-facto" standard for configuration now
(pyproject.toml and the like). This was far too big of a change for
Airflow 2 but with Airflow 3 it seems very appropriate.
2) On a popular request I added "team_id" as a database field - this
has quite a few far-reaching implications and it's ripple-effect on
Airflow 2 would be far too big for the "limited" multi-team setup -
but since we are going to do full versioning including DB changes in
Airflow 3, this is an opportunity to do it well. The implementation
detail of it will however depend on our choice of supported databases
so there is a little dependency on other decisions here. If we stick
with both Postgres and MySQL we will likely have to restructure the DB
to have synthetic UUID identifiers in order to add both versioning and
multi-team (because of MySQL index limitations).
3) The "proper" team identifier also allows to expand the scope of
multi-team to also allow "per-team" connections and variables. Again
for Airflow 2 case we could limit it to only the case where
connections and variables comes only from "per-team" secrets - but
since we are going to have DB identifiers and we are going to - anyhow
- reimplement Connections and Variables UI to get rid of FAB models
and implement them in reactive technology, it's only a bit more
complex to add "per-team" access there.
4) AIP-72 due to its "task" isolation, allows dropping the idea about
the "--team" flag from the components. With AIP-72 routing tasks to
particular "team" executors" is enough and there is no need to pass
the team information via "--team" flag that was originally supposed to
limit access of the components to only a single team. For Airflow 2
and AIP-44 that was a nice "hack" so that we do not have to carry the
"authorization" information together with the task. But since part of
AIP-72 is to carry the verifiable meta-data that will allow us to
cryptographically verify task provenance, we can drop this hack and
rely on AIP-72 implementation.
5) since DB isolation is "given" by AIP-72, we do not have to split
the delivery of AIP-67 into two phases (with and without DB isolation)
- it will be delivered as a single "with DB isolation" stage.
Those are the major differences vs. the proposal from May ( and as you
might see it is quite a different scope - and this is really why I
insisted on having Airflow 2/ Airflow 3 discussion before we conclude
the vote on it.
I will go through the proposal on Thursday during our call as planned
- but feel free to start discussions and comments before.
J.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hey Daniel,
> This is not a veto threat, but just expressing concern, I remain a bit
skeptical of multitenancy for a few reasons.
Just to bring it back to the right track - this is not a multitenancy,
really. It's multi-team. We've been going over that topic and I started to
like the "team" approach - and we definitely are not going to run multiple
"tenants" just multiple "teams" - in the same organization - where
typically today those teams share the same instance of Airflow and have
common "pipeline dependencies" between them, while they have different
expectations for dependencies (both for parsing and execution) and want to
be able to manage the execution resources (not scheduling) separately.
> 1. a lot of complexity will be introduced.
Somewhat yes. If it's a lot compared to all the other changes we are
implementing - I don't think so. The concept evolved "to stand on the
shoulders" of all the other changes for Airflow - and I'd say that if they
can be done, this one is mostly a "description on how to do team-deployment
as part of organization deployment" using existing components and "enabling
the concept by introducing shared team_id" that can bind together dag
parsing and execution and UI.
2. it's already not terribly uncommon that airflow users find scheduler
performance / concurrency insufficient and this would tend to make that
issue worse.
If someone will try to use it as "multi-tenancy", yes - but again - for a
long time this is not a goal (and I understood that tenancy means more for
a lot of people, that's why I adopted the "team" and I am going to explain
it over and over). This is not going to make it "easy" to manage and add
new tenants. Quite the opposite - this should be deliberate effort where
organization's manager delegates part of their responsibility (setting up
the environment and managing execution resources) to team deployment
managers. There will be no "one-click-add-new-tenant" feature built in. it
will require reconfiguring your airflow to add a new team.
3. usefulness would seem to be somewhat limited by the requirement that
users be on same version of airflow.
Again - yes, if you want to run a multi-tenant Airflow. But if you have
several teams to manage in your organization, you want to keep the "core
airflow" upgraded while giving certain freedom to your teams to manage
their environment.
J.
On Tue, Jul 30, 2024 at 6:04 PM Daniel Standish
<da...@astronomer.io.invalid> wrote:
> This is not a veto threat, but just expressing concern, I remain a bit
> skeptical of multitenancy for a few reasons.
>
> 1. a lot of complexity will be introduced.
> 2. it's already not terribly uncommon that airflow users find scheduler
> performance / concurrency insufficient and this would tend to make that
> issue worse.
> 3. usefulness would seem to be somewhat limited by the requirement that
> users be on same version of airflow.
>
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
This is not a veto threat, but just expressing concern, I remain a bit
skeptical of multitenancy for a few reasons.
1. a lot of complexity will be introduced.
2. it's already not terribly uncommon that airflow users find scheduler
performance / concurrency insufficient and this would tend to make that
issue worse.
3. usefulness would seem to be somewhat limited by the requirement that
users be on same version of airflow.
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Updated the description to:
> This is "airflow configuration" as defined in the config file. Each team
can also have access to separate Configuration and variables (identified by
the team_id) but there are also common configurations and variables that
might be shared between the teams (with null team-id). Uniqueness is
maintained across all connections and variable ids between the teams
(regardless if they have team_id or not).
Actually I realised now that this also stresses how much "mullti-team" vs.
"multi-tenant" this proposal is -where the team within the same
organisation will have to coordinate the connections/variables used so that
they do not clash. Which might make sense - because then the convention of
connection/variable ids might be set at the "organisation" level.
J.
On Mon, Jul 29, 2024 at 8:23 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Thanks Vikram.
>
>>
>> I think I understand your concept of team very clearly and the concept of
>> "team configuration".
>> I did find two "per-team configuration principles" somewhat at conflict
>> though:
>> 1. *Each team configuration SHOULD be a separate configuration file or
>> separate set of environment variables, holding team-specific configuration
>> needed by DAG file processor, Workers and Triggerer*
>> 2. In multi-team deployment, Connections and Variables have (nullable)
>> team_id field - which makes them either belonging to a specific team or
>> "globally available".
>>
>> Yes. That one will need a bit of clarification, It's been changed from
> the original design where connections and variables were totally separated
> - and point 1) is a bit of left-over. I will rephrase it slightly to add
> (with exception of common configuration defined at "global level" - that
> was a result of other comments where common connections and variables
> shared between teams were seen as an important feature, also original
> design assumed no connection and variables in database, which has been
> changed after we moved to Airlfow 3- only, because now the database
> structure modification is all but "given" by other changes.
>
>
>> Personally, I would have thought that you would want to standardize on
>> either explicit configuration required or implicit allowed throughout
>> rather than mixing. But, you probably have reasons for why a mixed
>> approach
>> is suitable here.
>>
>
> Yes. Those comments made me think this is a pretty valid case for the
> "mult-team" case. If we did multi-tenant, clear separation would be indeed
> better. But with multi-team, I can easily imagine some shared connections
> and variables that are "organisation" standard and shared between teams
> (Say SMTP connection that is used by all teams to send messages and
> notifications).
>
>>
>>
>> I assume that the DAG Parsing per-team environment is where you see the
>> need to integrate with AIP-66 (DAG Bundles ...)
>> Is that correct?
>>
>>
> Correct. Also as discussed with AIP-66 initial proposal from Jed - "per
> bundle" environment based exclusively on `pip install --target` zipped
> archive has many technical traps and does not define the "management" part
> of it and "security scope". So here the difference is that both -
> separation of dag parsing per team (separate DAG file processor) and common
> environment (dependencies) for DAG parsing for team can be still defined
> and maintained by the "Team Deployment Manager". And the environment
> separation can be done in various ways - same as today (separate image,
> separate virtualenv) and it can be done in a completely separated security
> perimeter from other teams. Basically - it makes it easy to make sure that
> parsing and execution of DAGs belonging to the same team can have the same
> dependencies and environment (potentially different from dependencies and
> environment of the other teams).
>
> In a way it is very similar to what has been already proposed (but is
> abandoned now) in AIP-46 (Runtime isolation for airlfow tasks and dag
> parsing) -
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-46+Runtime+isolation+for+airflow+tasks+and+dag+parsing
> that has been proposed by Ping Zhang, but unlike AIP-46 it does not make
> any assumptions on technology used (virtualenv, images) how the environment
> is prepared. The only assumption is that each team has it's own, separate
> environment managed by the Team Deployment Manager.
>
> J.
>
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Thanks Vikram.
>
> I think I understand your concept of team very clearly and the concept of
> "team configuration".
> I did find two "per-team configuration principles" somewhat at conflict
> though:
> 1. *Each team configuration SHOULD be a separate configuration file or
> separate set of environment variables, holding team-specific configuration
> needed by DAG file processor, Workers and Triggerer*
> 2. In multi-team deployment, Connections and Variables have (nullable)
> team_id field - which makes them either belonging to a specific team or
> "globally available".
>
> Yes. That one will need a bit of clarification, It's been changed from the
original design where connections and variables were totally separated -
and point 1) is a bit of left-over. I will rephrase it slightly to add
(with exception of common configuration defined at "global level" - that
was a result of other comments where common connections and variables
shared between teams were seen as an important feature, also original
design assumed no connection and variables in database, which has been
changed after we moved to Airlfow 3- only, because now the database
structure modification is all but "given" by other changes.
> Personally, I would have thought that you would want to standardize on
> either explicit configuration required or implicit allowed throughout
> rather than mixing. But, you probably have reasons for why a mixed approach
> is suitable here.
>
Yes. Those comments made me think this is a pretty valid case for the
"mult-team" case. If we did multi-tenant, clear separation would be indeed
better. But with multi-team, I can easily imagine some shared connections
and variables that are "organisation" standard and shared between teams
(Say SMTP connection that is used by all teams to send messages and
notifications).
>
>
> I assume that the DAG Parsing per-team environment is where you see the
> need to integrate with AIP-66 (DAG Bundles ...)
> Is that correct?
>
>
Correct. Also as discussed with AIP-66 initial proposal from Jed - "per
bundle" environment based exclusively on `pip install --target` zipped
archive has many technical traps and does not define the "management" part
of it and "security scope". So here the difference is that both -
separation of dag parsing per team (separate DAG file processor) and common
environment (dependencies) for DAG parsing for team can be still defined
and maintained by the "Team Deployment Manager". And the environment
separation can be done in various ways - same as today (separate image,
separate virtualenv) and it can be done in a completely separated security
perimeter from other teams. Basically - it makes it easy to make sure that
parsing and execution of DAGs belonging to the same team can have the same
dependencies and environment (potentially different from dependencies and
environment of the other teams).
In a way it is very similar to what has been already proposed (but is
abandoned now) in AIP-46 (Runtime isolation for airlfow tasks and dag
parsing) -
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-46+Runtime+isolation+for+airflow+tasks+and+dag+parsing
that has been proposed by Ping Zhang, but unlike AIP-46 it does not make
any assumptions on technology used (virtualenv, images) how the environment
is prepared. The only assumption is that each team has it's own, separate
environment managed by the Team Deployment Manager.
J.
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
Thanks Jarek, both the comments you added above make sense and help me
understand the steps here.
I was definitely struggling with the dependencies of this AIP on AIP-72
(task isolation) and AIP-66(DAG Bundles).
Specifically, there were a couple of references which were key in my mind.
- Workload isolation which I do think is very clearly understood in AIP-72
- DAGFileProcessor changes which I hadn't quite thought through in the
context of multi-team, which may touch both AIP-66 and AIP-72.
I think I understand your concept of team very clearly and the concept of
"team configuration".
I did find two "per-team configuration principles" somewhat at conflict
though:
1. *Each team configuration SHOULD be a separate configuration file or
separate set of environment variables, holding team-specific configuration
needed by DAG file processor, Workers and Triggerer*
2. In multi-team deployment, Connections and Variables have (nullable)
team_id field - which makes them either belonging to a specific team or
"globally available".
Personally, I would have thought that you would want to standardize on
either explicit configuration required or implicit allowed throughout
rather than mixing. But, you probably have reasons for why a mixed approach
is suitable here.
I also agree with your comment on Scheduling fairness for now
I assume that the DAG Parsing per-team environment is where you see the
need to integrate with AIP-66 (DAG Bundles ...)
Is that correct?
On Thu, Jul 25, 2024 at 5:13 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> One more thing to add - if there will be no more comments, I will start a
> vote soon - but maybe some clarifications might help - I spoke to few other
> people about it:
>
> * this one is heavily based on other AIPs that are also part of Airlfow 3.
> And while some parts are independent, AIP-72 (Task Isolation) and AIP-62
> (Parsing and bundling) and some details of those might have impact on some
> of the details - and maybe what will happen those dependencies will **not**
> complete in time for Airflow 3.0, So I am happy to mark it as Airflow 3.0
> (or 3.1 depending on dependencies)
> * the AIP is not trying to address "huge" installations with multiple
> tenants. Those are better served by writing extra UI layers and
> more-or-less "hiding" airflow effectively behind the corporate "solution"
> for those who can afford it. It's mostly for "mid-size" deployments, where
> there are enough separate (but interdependent) teams to manage but where
> the users won't have the capacity to make their own "uber-airflow" on top
> and where Airlfow UI might be useful to manage all the pipelines from those
> teams together. I might want to clarify it if that helps to convince those
> unconvinced.
>
> J
>
>
> On Tue, Jul 23, 2024 at 9:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I responded to some comments and had a long discussion on Slack with Ash
> > and I would love to hear even more comments and see if my responses are
> > satisfactory (so I would love to get confirmation/further comments on
> those
> > threads opened as discussion points - before I put it to a vote). There
> are
> > 18 inline comments now that wait for "yeah looks good" or "no I have
> still
> > doubts":
> >
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP
> > -67+Multi-team+deployment+of+Airflow+components
> >
> > Things that I wanted to address as general "big points" and explain them
> > here - so that others can make their comments on it here as well.
> >
> > 1) (Mostly Ash)* Is it worth it? Should it be part of the community
> > effort ?* a) On one hand AIP-72 gives a lot of isolation, b) AIP-66
> > Bundles and Parsing **might** introduce separate execution environments.
> c)
> > On the other hand users can wrap around multiple Airflow instances if the
> > only "real" benefit is a single "UI/management" endpoint.
> >
> > My answer - I think it still is.
> >
> > a) AIP-72 does give isolation (same as AIP-44 only more optimized and
> > designed for scratch), but it does not allow separation between teams for
> > DAG file processors. AIP-67 effectively introduces it by allowing each
> team
> > to have separate DAG file processors - this is possible even today - but
> > then it also allows them to effectively use a per-team environment -
> > because via team_id specific executors all the tasks of all the DAGs of
> the
> > team might be executing in their dedicated environment (so for example
> > whole team might use the same container image for processors and all the
> > workers they have in Celery. This is currently not part of AIP-72
> >
> > b) While AIP-66 had initially the concept of a per-bundle environment, I
> > thought having it defined by the DAG author and the way how it could be
> > managed is problematic. Especially if we allow to have "any" binary
> > environment (whether it's container image or zip containing venv) it has
> > security issue (unreviewable dependencies) and performance issues for
> > zipped venv (installing such a bundle takes time, and environmental
> issues
> > involving shared libraries cases and architectural differences are
> > basically unsurmountable - for example that would make it very difficult
> to
> > test same DAGs across different architectures and even different versions
> > of the same OS. Also it was not clear who and how would manage those
> > environments (i.e. roles of actors involved). While this is a great
> idea, I
> > think it's too granular to be effectively managed "per team" - while it
> is
> > great for some exceptional cases where we can say "this DAG is so special
> > that it needs a separate environment". I strongly believe the "per team"
> > environment is a much more common case (and it coincides with "working
> as a
> > team" - where the whole team should be able to easily run and develop
> > various DAGs. Conway's law in full swing. AIP-67 is addressing all that -
> > various environments per team, Team Deployment Manager is introduced as
> > Actor who manages those environments and team configuration.
> >
> > c) while yes, it's possible to wrap around multiple Airflows with an
> extra
> > UI layer and have them managed in common - but it either has to provide a
> > version of Airflow UI "clone" where you can see several DAGs - possibly
> > interacting with each other, or you have to have some weird ways of easy
> > switching or showing multiple Airflow UIs at the same time. First is
> going
> > to be super-costly to manage over time as Airflow will evolve, second is
> a
> > band-aid really IMHO. And when I think about it - I think mostly about
> > on-premise users, not managed-services users. Yes managed services could
> > afford such extra layers and it could be thought of as a nice "paid"
> > feature of Airflow services. So maybe indeed we should think about it as
> > "not in community" ? But I know a number of users who have on-premise
> > Airflow (and for good reasons cannot move to managed Airflow) and we cut
> > all of them off from being able to have a number of their teams having
> > pipelines isolated from environment and security point of view, while
> > centrally monitored and managed.
> >
> > 2) (Vikram, Constance and TP ): *Permissions for Data Assets (related to
> > AIP-73 + sub-aips)* - I proposed that each team's Data Assets are
> > separated (same as for DAGs automatically team_id is assigned when
> parsing
> > the DAGs by DAG File Processor) - and that they are effectively "separate
> > namespaces". We have no proposal for permissions of Data Assets (yet) -
> it
> > might come later - that's why I do not want to interfere with it and
> > proposed a simple solution where Data Assets can be matched by URIs
> (which
> > become optional in AIP-73) and that the DAG using Data Asset with same
> URI
> > might get triggered by a different Data Set from another team but the
> same
> > URI only if the "triggered" DAG explicitly hase `allowed_triggering_by =
> [
> > "team_2", "team 3"]` (or similar).
> >
> > 3) (Niko, Amogh, TP) - *Scope of "access" to team DAGs/folders/managing
> > access* - I assume from DAG file processing POV "folder(s)" is currently
> > the right abstraction (and already handled by standalone
> DAGFileProcessor).
> > The nice thing about it is that control over the folder "access" can (and
> > should) be done outside of Airflow - we delegate it out. Similarly like
> > Auth Manager access - we do not want to record team members who have
> access
> > - but we want to delegate it out. We do not want to keep information
> about
> > who is in which team nor handle team changes - this should all be handled
> > in the corporate identity system. This way both sides are delegated out -
> > DAG authoring and DAG management. Airflow only knows "team_id", nothing
> > else once the DAG is parsed (DAG and DAG Assets created by parsing the
> DAG
> > have team_id assigned automatically). And In the future we might expand
> it
> > - when we have - for example - declarative DAGs submitted via API, the
> API
> > can have "team_id" added or when it is created via some kind of factory -
> > that factory might assign their ids. The "folder" that we have now is
> > simply to make use of DAG file processor --subdir flag - but actually it
> > does not matter - once the DAG gets serialized - the DAG and it's assets
> > will just have "team_id" assigned, and they do not have to come from the
> > same subdir.
> >
> > Looking forward to closing those discussions and putting it finally up to
> > a vote.
> >
> > J.
> >
> >
> >
> >
> >
> > On Fri, Jul 19, 2024 at 10:49 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >>
> >> 1. The roles and responsibilities of what you have called "Organization
> >>> Deployment Managers" vs. "Team Deployment Managers". This is somewhat
> >>> different than the model I have seen in practice, so trying to
> reconcile
> >>> this in my head. If this is a terminology difference or a role
> difference
> >>> or where exactly the line is drawn.
> >>>
> >>>
> >> This is what is not possible in current practice, because we have no
> such
> >> distinction. Currently there is only one deployment manager role, but at
> >> least with my talks to some users and with the approach I proposed, it's
> >> possible that the "oganization deployment manager" (a.k.a Data Platform
> >> Team) prepares and manages Airflow as a "whole". - i.e. Running
> >> airflow scheduler, webserver, connection to the organisation identity
> >> management system. But then - those people are struggling when different
> >> teams have different expectations for what dependencies, OS, queues, GPU
> >> access etc. and Data Platform "Organisation Deployment Manager" wants to
> >> delegate that to each team's deployment manager - such "team DM" should
> be
> >> able - for example to manage their own K8S cluster where their team's
> job
> >> will be running - with appropriate dependencies/hardware resources etc.
> >> And important aspect is that each team can manage it without
> "bothering"
> >> the Platform team (but still will be trusted enough to allow installing
> >> arbitrary packages),- and the platform team will mostly worry about
> airflow
> >> itself.
> >>
> >>
> >>> 2. DAG File Processing
> >>> I am quite perplexed by this and wondering about the overlap between
> the
> >>> "execution bundles" concept which Jed had defined as part of AIP-66 and
> >>> since deleted from there.
> >>>
> >>
> >> Actually that was also the result of discussion we had in the doc with
> >> Ash and Jed. And I see the potential of joining it. The AIP-66
> >> defined bundle environment that could be specified - but the problem
> with
> >> that was that a) it assumed it is based on "pip install --target" venvs
> >> (which has a host of potential problems I explained) and also b) it
> skipped
> >> over on the management part of those - i.e. who prepares those
> >> environments and manages them - is this a DAG author ? if so, this
> opens a
> >> host of security issues - because those environments are essentially
> >> "binary" and "not reviewable" - so there must be some role responsible
> for
> >> it in our security model (i.e... "bundle deployment manager"? ) - which
> >> sounds suspiciously similar "team deployment manager". One of the
> important
> >> differences of AIP-67 here is that it explicitly explains and allows to
> >> separate all three "vulnerable" components per "team" - > DAG file
> >> Processor,. Triggerer,. Worker. AIP-66 does not mention that. Implicitly
> >> assuming that running the code from different bundles can be done on
> shared
> >> machines (DAG file processor/Schediuler) or even in the same process
> >> (Triggerer). AIP-67 adds an explicit, strong isolation of code
> execution
> >> between the teams - so that the code from different teams are not
> executed
> >> on the same machines, containers, but they can be easily separated (by
> >> deployment options). Without AIP-67 it is impossible - or very
> difficult
> >> and error-prone - to make sure that the code of one "bundle" cannot
> leak to
> >> the other "bundle". We seem to silently skip over the fact that both DAG
> >> File Processor and Triggerrer can execute the code in the same machine
> or
> >> process and that isolation in such a case is next to impossible. And
> this
> >> is a very important aspect of "enterprise security isolation". In some
> ways
> >> even more important than isolating access to the same DB.
> >>
> >> And I actually am quite open to joining it - and have either "bundle =
> >> team" or "bundle belongs to team".
> >>
> >>
> >>
> >>> I will read this doc again for sure.
> >>> It has grown and evolved so much that at least for me it was quite
> >>> challenging to grasp. Thanks for working through this.
> >>>
> >>> Vikram
> >>>
> >>>
> >>> On Thu, Jul 18, 2024 at 9:39 PM Amogh Desai <am...@gmail.com>
> >>> wrote:
> >>>
> >>> > Nice, thanks for clarifying all this!
> >>> >
> >>> > Now that I read the new proposal, it is adding up to me why certain
> >>> > decisions were made.
> >>> > The decision to separate the "common" part from the "per team" part
> >>> adds up
> >>> > now. It
> >>> > is a traditional paradigm of separating "control plane" from
> "compute".
> >>> >
> >>> > Thanks & Regards,
> >>> > Amogh Desai
> >>> >
> >>> >
> >>> > On Mon, Jul 15, 2024 at 8:53 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >>> >
> >>> > > I got the transcript and chat from the last call (thanks Kaxil!)
> and
> >>> it
> >>> > > allowed me to answer a few questions that were asked during my
> >>> > presentation
> >>> > > about AIP-67. I updated the AIP document but here is summary:
> >>> > >
> >>> > > 1) What about Pools (asked by Elad and Jed, Jorrick): I thought
> >>> about it
> >>> > > and I propose that pools could have (optional) team_id added. This
> >>> will
> >>> > > allow users to keep common pools (no team_id assigned) and have
> >>> > > team-specific ones. DAG file processor specific for each team will
> >>> fail
> >>> > DAG
> >>> > > if it tries using a pool that is not common and belonging to "other
> >>> > team".
> >>> > > Also each team will be able to have their own "default_pool"
> >>> configured.
> >>> > > This will give enough flexibility on 'common vs. team exclusive"
> use
> >>> of
> >>> > > pools.
> >>> > >
> >>> > > 2) Isolation for connections (John, Filip, Elad, Kaxil, Amogh,
> Ash):
> >>> yes.
> >>> > > That is part of the design. The connections and variables can be
> >>> accessed
> >>> > > per team - AIP-72 will only provide the tasks with connections that
> >>> > belong
> >>> > > to the team. Ash mentioned OPA (which might be used for that
> >>> purpose).
> >>> > It's
> >>> > > not defined how exactly it will be implemented in AIP-72, it's not
> >>> > detailed
> >>> > > enough, but it can use the very mechanisms that AIP-72 - by only
> >>> allowing
> >>> > > "global" connections and "my team" connections to be passed by
> >>> AIP-72 API
> >>> > > to the task and DAG file processor.
> >>> > >
> >>> > > 3) Whether "Team=deployment" - Igor / Vikram ? -> depends on what
> you
> >>> > > understand by deployment. i'd say "sub-deployment" - each
> deployment
> >>> in a
> >>> > > "multi-team" environment will consist of the "common" part and each
> >>> team
> >>> > > will have their own part (where configuration and management of
> such
> >>> team
> >>> > > deployment parts will be delegated to the team deployment manager).
> >>> For
> >>> > > example such deployment managers will be able to build and publish
> >>> the
> >>> > > environment (for example container images) used by team A to run
> >>> Airflow.
> >>> > > Or change "team" specific configuration.
> >>> > >
> >>> > > 4) "This seems like quite a lot of work to share a scheduler and a
> >>> web
> >>> > > server. What’s the net benefit of this complexity?" -> Ash, John,
> >>> Amogh,
> >>> > > Maciej: Yes. I absolutely see it as a valuable option. It reflects
> >>> > > organizational structure and needs of many of our users, where they
> >>> want
> >>> > to
> >>> > > manage part of the environment, monitoring of what's going in all
> of
> >>> > their
> >>> > > teams centrally (and manage things like upgrades of Airflow,
> security
> >>> > > centrally), while they want to delegate control of environments and
> >>> > > resources down to their teams. This is the need that I've heard
> from
> >>> many
> >>> > > users who have a "data platform team" that makes Airflow available
> to
> >>> > their
> >>> > > several teams. I think the proposal I have is a nice middle ground
> >>> that
> >>> > > follows Conway's law - that architecture of your system should
> >>> reflect
> >>> > your
> >>> > > organizational structure - and what I separated out as "common"
> >>> parts is
> >>> > > precisely what "data platform team" would like to manage, where
> "team
> >>> > > environment" is something that data platform should (and want to)
> >>> > delegate
> >>> > > to their teams.
> >>> > >
> >>> > > 5) "I am a little surprised by a shared dataset" - Vikram/Elad :
> The
> >>> > > datasets are defined by their URLs and as such - they don't have
> >>> > > "ownership". As I see it - It's really important who can trigger a
> >>> DAG
> >>> > and
> >>> > > the controls I proposed allow the DAG author to specify "In this
> DAG
> >>> it's
> >>> > > also ok when a different team (specified) triggered the dataset
> >>> event".
> >>> > But
> >>> > > I left a note that it is AIP-73-dependent "Expanded Data Awareness"
> >>> and
> >>> > > once we get that explained/clarified I am happy to coordinate with
> >>> > > Constance and see if we need to do more. Happy to hear more
> comments
> >>> on
> >>> > > that one.
> >>> > >
> >>> > > I reflected the 2 points and 5) in the AIP. Looking forward to
> more
> >>> > > comments on the proposal - in the AIP or here.
> >>> > >
> >>> > > J.
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Tue, Jul 9, 2024 at 4:48 PM Jarek Potiuk <ja...@potiuk.com>
> >>> wrote:
> >>> > >
> >>> > > > Hello Everyone,
> >>> > > >
> >>> > > > I would like to resume discussion on AIP-67. After going through
> a
> >>> > > > number of discussions and clarifications about the scope of
> >>> Airflow 3,
> >>> > > > I rewrote the proposal for AIP-67 with the assumption that we
> will
> >>> do
> >>> > > > it for Airflow 3 only - and that it will be based on the new
> >>> proposed
> >>> > > > AIP-72 (Task Execution Interface) rather than Airflow 2-only
> AIP-44
> >>> > > > Internal API.
> >>> > > >
> >>> > > > The updated proposal is here
> >>> > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >>> > > >
> >>> > > > Feel free to comment there in-line or raise your "big" comments"
> >>> here,
> >>> > > > but here is the impact of changing the target to Airflow 3:
> >>> > > >
> >>> > > > 1) I proposed to change configuration of Airflow to use more
> >>> > > > structured TOML than plain "ini" - toml is a successor of "ini"
> >>> and is
> >>> > > > largely compatible, but it has arrays, tables and nesting, has
> good
> >>> > > > support in Python and is "de-facto" standard for configuration
> now
> >>> > > > (pyproject.toml and the like). This was far too big of a change
> for
> >>> > > > Airflow 2 but with Airflow 3 it seems very appropriate.
> >>> > > >
> >>> > > > 2) On a popular request I added "team_id" as a database field -
> >>> this
> >>> > > > has quite a few far-reaching implications and it's ripple-effect
> on
> >>> > > > Airflow 2 would be far too big for the "limited" multi-team
> setup -
> >>> > > > but since we are going to do full versioning including DB changes
> >>> in
> >>> > > > Airflow 3, this is an opportunity to do it well. The
> implementation
> >>> > > > detail of it will however depend on our choice of supported
> >>> databases
> >>> > > > so there is a little dependency on other decisions here. If we
> >>> stick
> >>> > > > with both Postgres and MySQL we will likely have to restructure
> >>> the DB
> >>> > > > to have synthetic UUID identifiers in order to add both
> versioning
> >>> and
> >>> > > > multi-team (because of MySQL index limitations).
> >>> > > >
> >>> > > > 3) The "proper" team identifier also allows to expand the scope
> of
> >>> > > > multi-team to also allow "per-team" connections and variables.
> >>> Again
> >>> > > > for Airflow 2 case we could limit it to only the case where
> >>> > > > connections and variables comes only from "per-team" secrets -
> but
> >>> > > > since we are going to have DB identifiers and we are going to -
> >>> anyhow
> >>> > > > - reimplement Connections and Variables UI to get rid of FAB
> models
> >>> > > > and implement them in reactive technology, it's only a bit more
> >>> > > > complex to add "per-team" access there.
> >>> > > >
> >>> > > > 4) AIP-72 due to its "task" isolation, allows dropping the idea
> >>> about
> >>> > > > the "--team" flag from the components. With AIP-72 routing tasks
> to
> >>> > > > particular "team" executors" is enough and there is no need to
> pass
> >>> > > > the team information via "--team" flag that was originally
> >>> supposed to
> >>> > > > limit access of the components to only a single team. For
> Airflow 2
> >>> > > > and AIP-44 that was a nice "hack" so that we do not have to carry
> >>> the
> >>> > > > "authorization" information together with the task. But since
> part
> >>> of
> >>> > > > AIP-72 is to carry the verifiable meta-data that will allow us to
> >>> > > > cryptographically verify task provenance, we can drop this hack
> and
> >>> > > > rely on AIP-72 implementation.
> >>> > > >
> >>> > > > 5) since DB isolation is "given" by AIP-72, we do not have to
> split
> >>> > > > the delivery of AIP-67 into two phases (with and without DB
> >>> isolation)
> >>> > > > - it will be delivered as a single "with DB isolation" stage.
> >>> > > >
> >>> > > > Those are the major differences vs. the proposal from May ( and
> as
> >>> you
> >>> > > > might see it is quite a different scope - and this is really why
> I
> >>> > > > insisted on having Airflow 2/ Airflow 3 discussion before we
> >>> conclude
> >>> > > > the vote on it.
> >>> > > >
> >>> > > > I will go through the proposal on Thursday during our call as
> >>> planned
> >>> > > > - but feel free to start discussions and comments before.
> >>> > > >
> >>> > > > J.
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Jarek Potiuk <ja...@potiuk.com>.
One more thing to add - if there will be no more comments, I will start a
vote soon - but maybe some clarifications might help - I spoke to few other
people about it:
* this one is heavily based on other AIPs that are also part of Airlfow 3.
And while some parts are independent, AIP-72 (Task Isolation) and AIP-62
(Parsing and bundling) and some details of those might have impact on some
of the details - and maybe what will happen those dependencies will **not**
complete in time for Airflow 3.0, So I am happy to mark it as Airflow 3.0
(or 3.1 depending on dependencies)
* the AIP is not trying to address "huge" installations with multiple
tenants. Those are better served by writing extra UI layers and
more-or-less "hiding" airflow effectively behind the corporate "solution"
for those who can afford it. It's mostly for "mid-size" deployments, where
there are enough separate (but interdependent) teams to manage but where
the users won't have the capacity to make their own "uber-airflow" on top
and where Airlfow UI might be useful to manage all the pipelines from those
teams together. I might want to clarify it if that helps to convince those
unconvinced.
J
On Tue, Jul 23, 2024 at 9:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> I responded to some comments and had a long discussion on Slack with Ash
> and I would love to hear even more comments and see if my responses are
> satisfactory (so I would love to get confirmation/further comments on those
> threads opened as discussion points - before I put it to a vote). There are
> 18 inline comments now that wait for "yeah looks good" or "no I have still
> doubts":
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP
> -67+Multi-team+deployment+of+Airflow+components
>
> Things that I wanted to address as general "big points" and explain them
> here - so that others can make their comments on it here as well.
>
> 1) (Mostly Ash)* Is it worth it? Should it be part of the community
> effort ?* a) On one hand AIP-72 gives a lot of isolation, b) AIP-66
> Bundles and Parsing **might** introduce separate execution environments. c)
> On the other hand users can wrap around multiple Airflow instances if the
> only "real" benefit is a single "UI/management" endpoint.
>
> My answer - I think it still is.
>
> a) AIP-72 does give isolation (same as AIP-44 only more optimized and
> designed for scratch), but it does not allow separation between teams for
> DAG file processors. AIP-67 effectively introduces it by allowing each team
> to have separate DAG file processors - this is possible even today - but
> then it also allows them to effectively use a per-team environment -
> because via team_id specific executors all the tasks of all the DAGs of the
> team might be executing in their dedicated environment (so for example
> whole team might use the same container image for processors and all the
> workers they have in Celery. This is currently not part of AIP-72
>
> b) While AIP-66 had initially the concept of a per-bundle environment, I
> thought having it defined by the DAG author and the way how it could be
> managed is problematic. Especially if we allow to have "any" binary
> environment (whether it's container image or zip containing venv) it has
> security issue (unreviewable dependencies) and performance issues for
> zipped venv (installing such a bundle takes time, and environmental issues
> involving shared libraries cases and architectural differences are
> basically unsurmountable - for example that would make it very difficult to
> test same DAGs across different architectures and even different versions
> of the same OS. Also it was not clear who and how would manage those
> environments (i.e. roles of actors involved). While this is a great idea, I
> think it's too granular to be effectively managed "per team" - while it is
> great for some exceptional cases where we can say "this DAG is so special
> that it needs a separate environment". I strongly believe the "per team"
> environment is a much more common case (and it coincides with "working as a
> team" - where the whole team should be able to easily run and develop
> various DAGs. Conway's law in full swing. AIP-67 is addressing all that -
> various environments per team, Team Deployment Manager is introduced as
> Actor who manages those environments and team configuration.
>
> c) while yes, it's possible to wrap around multiple Airflows with an extra
> UI layer and have them managed in common - but it either has to provide a
> version of Airflow UI "clone" where you can see several DAGs - possibly
> interacting with each other, or you have to have some weird ways of easy
> switching or showing multiple Airflow UIs at the same time. First is going
> to be super-costly to manage over time as Airflow will evolve, second is a
> band-aid really IMHO. And when I think about it - I think mostly about
> on-premise users, not managed-services users. Yes managed services could
> afford such extra layers and it could be thought of as a nice "paid"
> feature of Airflow services. So maybe indeed we should think about it as
> "not in community" ? But I know a number of users who have on-premise
> Airflow (and for good reasons cannot move to managed Airflow) and we cut
> all of them off from being able to have a number of their teams having
> pipelines isolated from environment and security point of view, while
> centrally monitored and managed.
>
> 2) (Vikram, Constance and TP ): *Permissions for Data Assets (related to
> AIP-73 + sub-aips)* - I proposed that each team's Data Assets are
> separated (same as for DAGs automatically team_id is assigned when parsing
> the DAGs by DAG File Processor) - and that they are effectively "separate
> namespaces". We have no proposal for permissions of Data Assets (yet) - it
> might come later - that's why I do not want to interfere with it and
> proposed a simple solution where Data Assets can be matched by URIs (which
> become optional in AIP-73) and that the DAG using Data Asset with same URI
> might get triggered by a different Data Set from another team but the same
> URI only if the "triggered" DAG explicitly hase `allowed_triggering_by = [
> "team_2", "team 3"]` (or similar).
>
> 3) (Niko, Amogh, TP) - *Scope of "access" to team DAGs/folders/managing
> access* - I assume from DAG file processing POV "folder(s)" is currently
> the right abstraction (and already handled by standalone DAGFileProcessor).
> The nice thing about it is that control over the folder "access" can (and
> should) be done outside of Airflow - we delegate it out. Similarly like
> Auth Manager access - we do not want to record team members who have access
> - but we want to delegate it out. We do not want to keep information about
> who is in which team nor handle team changes - this should all be handled
> in the corporate identity system. This way both sides are delegated out -
> DAG authoring and DAG management. Airflow only knows "team_id", nothing
> else once the DAG is parsed (DAG and DAG Assets created by parsing the DAG
> have team_id assigned automatically). And In the future we might expand it
> - when we have - for example - declarative DAGs submitted via API, the API
> can have "team_id" added or when it is created via some kind of factory -
> that factory might assign their ids. The "folder" that we have now is
> simply to make use of DAG file processor --subdir flag - but actually it
> does not matter - once the DAG gets serialized - the DAG and it's assets
> will just have "team_id" assigned, and they do not have to come from the
> same subdir.
>
> Looking forward to closing those discussions and putting it finally up to
> a vote.
>
> J.
>
>
>
>
>
> On Fri, Jul 19, 2024 at 10:49 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>>
>> 1. The roles and responsibilities of what you have called "Organization
>>> Deployment Managers" vs. "Team Deployment Managers". This is somewhat
>>> different than the model I have seen in practice, so trying to reconcile
>>> this in my head. If this is a terminology difference or a role difference
>>> or where exactly the line is drawn.
>>>
>>>
>> This is what is not possible in current practice, because we have no such
>> distinction. Currently there is only one deployment manager role, but at
>> least with my talks to some users and with the approach I proposed, it's
>> possible that the "oganization deployment manager" (a.k.a Data Platform
>> Team) prepares and manages Airflow as a "whole". - i.e. Running
>> airflow scheduler, webserver, connection to the organisation identity
>> management system. But then - those people are struggling when different
>> teams have different expectations for what dependencies, OS, queues, GPU
>> access etc. and Data Platform "Organisation Deployment Manager" wants to
>> delegate that to each team's deployment manager - such "team DM" should be
>> able - for example to manage their own K8S cluster where their team's job
>> will be running - with appropriate dependencies/hardware resources etc.
>> And important aspect is that each team can manage it without "bothering"
>> the Platform team (but still will be trusted enough to allow installing
>> arbitrary packages),- and the platform team will mostly worry about airflow
>> itself.
>>
>>
>>> 2. DAG File Processing
>>> I am quite perplexed by this and wondering about the overlap between the
>>> "execution bundles" concept which Jed had defined as part of AIP-66 and
>>> since deleted from there.
>>>
>>
>> Actually that was also the result of discussion we had in the doc with
>> Ash and Jed. And I see the potential of joining it. The AIP-66
>> defined bundle environment that could be specified - but the problem with
>> that was that a) it assumed it is based on "pip install --target" venvs
>> (which has a host of potential problems I explained) and also b) it skipped
>> over on the management part of those - i.e. who prepares those
>> environments and manages them - is this a DAG author ? if so, this opens a
>> host of security issues - because those environments are essentially
>> "binary" and "not reviewable" - so there must be some role responsible for
>> it in our security model (i.e... "bundle deployment manager"? ) - which
>> sounds suspiciously similar "team deployment manager". One of the important
>> differences of AIP-67 here is that it explicitly explains and allows to
>> separate all three "vulnerable" components per "team" - > DAG file
>> Processor,. Triggerer,. Worker. AIP-66 does not mention that. Implicitly
>> assuming that running the code from different bundles can be done on shared
>> machines (DAG file processor/Schediuler) or even in the same process
>> (Triggerer). AIP-67 adds an explicit, strong isolation of code execution
>> between the teams - so that the code from different teams are not executed
>> on the same machines, containers, but they can be easily separated (by
>> deployment options). Without AIP-67 it is impossible - or very difficult
>> and error-prone - to make sure that the code of one "bundle" cannot leak to
>> the other "bundle". We seem to silently skip over the fact that both DAG
>> File Processor and Triggerrer can execute the code in the same machine or
>> process and that isolation in such a case is next to impossible. And this
>> is a very important aspect of "enterprise security isolation". In some ways
>> even more important than isolating access to the same DB.
>>
>> And I actually am quite open to joining it - and have either "bundle =
>> team" or "bundle belongs to team".
>>
>>
>>
>>> I will read this doc again for sure.
>>> It has grown and evolved so much that at least for me it was quite
>>> challenging to grasp. Thanks for working through this.
>>>
>>> Vikram
>>>
>>>
>>> On Thu, Jul 18, 2024 at 9:39 PM Amogh Desai <am...@gmail.com>
>>> wrote:
>>>
>>> > Nice, thanks for clarifying all this!
>>> >
>>> > Now that I read the new proposal, it is adding up to me why certain
>>> > decisions were made.
>>> > The decision to separate the "common" part from the "per team" part
>>> adds up
>>> > now. It
>>> > is a traditional paradigm of separating "control plane" from "compute".
>>> >
>>> > Thanks & Regards,
>>> > Amogh Desai
>>> >
>>> >
>>> > On Mon, Jul 15, 2024 at 8:53 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >
>>> > > I got the transcript and chat from the last call (thanks Kaxil!) and
>>> it
>>> > > allowed me to answer a few questions that were asked during my
>>> > presentation
>>> > > about AIP-67. I updated the AIP document but here is summary:
>>> > >
>>> > > 1) What about Pools (asked by Elad and Jed, Jorrick): I thought
>>> about it
>>> > > and I propose that pools could have (optional) team_id added. This
>>> will
>>> > > allow users to keep common pools (no team_id assigned) and have
>>> > > team-specific ones. DAG file processor specific for each team will
>>> fail
>>> > DAG
>>> > > if it tries using a pool that is not common and belonging to "other
>>> > team".
>>> > > Also each team will be able to have their own "default_pool"
>>> configured.
>>> > > This will give enough flexibility on 'common vs. team exclusive" use
>>> of
>>> > > pools.
>>> > >
>>> > > 2) Isolation for connections (John, Filip, Elad, Kaxil, Amogh, Ash):
>>> yes.
>>> > > That is part of the design. The connections and variables can be
>>> accessed
>>> > > per team - AIP-72 will only provide the tasks with connections that
>>> > belong
>>> > > to the team. Ash mentioned OPA (which might be used for that
>>> purpose).
>>> > It's
>>> > > not defined how exactly it will be implemented in AIP-72, it's not
>>> > detailed
>>> > > enough, but it can use the very mechanisms that AIP-72 - by only
>>> allowing
>>> > > "global" connections and "my team" connections to be passed by
>>> AIP-72 API
>>> > > to the task and DAG file processor.
>>> > >
>>> > > 3) Whether "Team=deployment" - Igor / Vikram ? -> depends on what you
>>> > > understand by deployment. i'd say "sub-deployment" - each deployment
>>> in a
>>> > > "multi-team" environment will consist of the "common" part and each
>>> team
>>> > > will have their own part (where configuration and management of such
>>> team
>>> > > deployment parts will be delegated to the team deployment manager).
>>> For
>>> > > example such deployment managers will be able to build and publish
>>> the
>>> > > environment (for example container images) used by team A to run
>>> Airflow.
>>> > > Or change "team" specific configuration.
>>> > >
>>> > > 4) "This seems like quite a lot of work to share a scheduler and a
>>> web
>>> > > server. What’s the net benefit of this complexity?" -> Ash, John,
>>> Amogh,
>>> > > Maciej: Yes. I absolutely see it as a valuable option. It reflects
>>> > > organizational structure and needs of many of our users, where they
>>> want
>>> > to
>>> > > manage part of the environment, monitoring of what's going in all of
>>> > their
>>> > > teams centrally (and manage things like upgrades of Airflow, security
>>> > > centrally), while they want to delegate control of environments and
>>> > > resources down to their teams. This is the need that I've heard from
>>> many
>>> > > users who have a "data platform team" that makes Airflow available to
>>> > their
>>> > > several teams. I think the proposal I have is a nice middle ground
>>> that
>>> > > follows Conway's law - that architecture of your system should
>>> reflect
>>> > your
>>> > > organizational structure - and what I separated out as "common"
>>> parts is
>>> > > precisely what "data platform team" would like to manage, where "team
>>> > > environment" is something that data platform should (and want to)
>>> > delegate
>>> > > to their teams.
>>> > >
>>> > > 5) "I am a little surprised by a shared dataset" - Vikram/Elad : The
>>> > > datasets are defined by their URLs and as such - they don't have
>>> > > "ownership". As I see it - It's really important who can trigger a
>>> DAG
>>> > and
>>> > > the controls I proposed allow the DAG author to specify "In this DAG
>>> it's
>>> > > also ok when a different team (specified) triggered the dataset
>>> event".
>>> > But
>>> > > I left a note that it is AIP-73-dependent "Expanded Data Awareness"
>>> and
>>> > > once we get that explained/clarified I am happy to coordinate with
>>> > > Constance and see if we need to do more. Happy to hear more comments
>>> on
>>> > > that one.
>>> > >
>>> > > I reflected the 2 points and 5) in the AIP. Looking forward to more
>>> > > comments on the proposal - in the AIP or here.
>>> > >
>>> > > J.
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Tue, Jul 9, 2024 at 4:48 PM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> > >
>>> > > > Hello Everyone,
>>> > > >
>>> > > > I would like to resume discussion on AIP-67. After going through a
>>> > > > number of discussions and clarifications about the scope of
>>> Airflow 3,
>>> > > > I rewrote the proposal for AIP-67 with the assumption that we will
>>> do
>>> > > > it for Airflow 3 only - and that it will be based on the new
>>> proposed
>>> > > > AIP-72 (Task Execution Interface) rather than Airflow 2-only AIP-44
>>> > > > Internal API.
>>> > > >
>>> > > > The updated proposal is here
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>>> > > >
>>> > > > Feel free to comment there in-line or raise your "big" comments"
>>> here,
>>> > > > but here is the impact of changing the target to Airflow 3:
>>> > > >
>>> > > > 1) I proposed to change configuration of Airflow to use more
>>> > > > structured TOML than plain "ini" - toml is a successor of "ini"
>>> and is
>>> > > > largely compatible, but it has arrays, tables and nesting, has good
>>> > > > support in Python and is "de-facto" standard for configuration now
>>> > > > (pyproject.toml and the like). This was far too big of a change for
>>> > > > Airflow 2 but with Airflow 3 it seems very appropriate.
>>> > > >
>>> > > > 2) On a popular request I added "team_id" as a database field -
>>> this
>>> > > > has quite a few far-reaching implications and it's ripple-effect on
>>> > > > Airflow 2 would be far too big for the "limited" multi-team setup -
>>> > > > but since we are going to do full versioning including DB changes
>>> in
>>> > > > Airflow 3, this is an opportunity to do it well. The implementation
>>> > > > detail of it will however depend on our choice of supported
>>> databases
>>> > > > so there is a little dependency on other decisions here. If we
>>> stick
>>> > > > with both Postgres and MySQL we will likely have to restructure
>>> the DB
>>> > > > to have synthetic UUID identifiers in order to add both versioning
>>> and
>>> > > > multi-team (because of MySQL index limitations).
>>> > > >
>>> > > > 3) The "proper" team identifier also allows to expand the scope of
>>> > > > multi-team to also allow "per-team" connections and variables.
>>> Again
>>> > > > for Airflow 2 case we could limit it to only the case where
>>> > > > connections and variables comes only from "per-team" secrets - but
>>> > > > since we are going to have DB identifiers and we are going to -
>>> anyhow
>>> > > > - reimplement Connections and Variables UI to get rid of FAB models
>>> > > > and implement them in reactive technology, it's only a bit more
>>> > > > complex to add "per-team" access there.
>>> > > >
>>> > > > 4) AIP-72 due to its "task" isolation, allows dropping the idea
>>> about
>>> > > > the "--team" flag from the components. With AIP-72 routing tasks to
>>> > > > particular "team" executors" is enough and there is no need to pass
>>> > > > the team information via "--team" flag that was originally
>>> supposed to
>>> > > > limit access of the components to only a single team. For Airflow 2
>>> > > > and AIP-44 that was a nice "hack" so that we do not have to carry
>>> the
>>> > > > "authorization" information together with the task. But since part
>>> of
>>> > > > AIP-72 is to carry the verifiable meta-data that will allow us to
>>> > > > cryptographically verify task provenance, we can drop this hack and
>>> > > > rely on AIP-72 implementation.
>>> > > >
>>> > > > 5) since DB isolation is "given" by AIP-72, we do not have to split
>>> > > > the delivery of AIP-67 into two phases (with and without DB
>>> isolation)
>>> > > > - it will be delivered as a single "with DB isolation" stage.
>>> > > >
>>> > > > Those are the major differences vs. the proposal from May ( and as
>>> you
>>> > > > might see it is quite a different scope - and this is really why I
>>> > > > insisted on having Airflow 2/ Airflow 3 discussion before we
>>> conclude
>>> > > > the vote on it.
>>> > > >
>>> > > > I will go through the proposal on Thursday during our call as
>>> planned
>>> > > > - but feel free to start discussions and comments before.
>>> > > >
>>> > > > J.
>>> > > >
>>> > >
>>> >
>>>
>>
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Jarek Potiuk <ja...@potiuk.com>.
I responded to some comments and had a long discussion on Slack with Ash
and I would love to hear even more comments and see if my responses are
satisfactory (so I would love to get confirmation/further comments on those
threads opened as discussion points - before I put it to a vote). There are
18 inline comments now that wait for "yeah looks good" or "no I have still
doubts":
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP
-67+Multi-team+deployment+of+Airflow+components
Things that I wanted to address as general "big points" and explain them
here - so that others can make their comments on it here as well.
1) (Mostly Ash)* Is it worth it? Should it be part of the community effort
?* a) On one hand AIP-72 gives a lot of isolation, b) AIP-66 Bundles and
Parsing **might** introduce separate execution environments. c) On the
other hand users can wrap around multiple Airflow instances if the only
"real" benefit is a single "UI/management" endpoint.
My answer - I think it still is.
a) AIP-72 does give isolation (same as AIP-44 only more optimized and
designed for scratch), but it does not allow separation between teams for
DAG file processors. AIP-67 effectively introduces it by allowing each team
to have separate DAG file processors - this is possible even today - but
then it also allows them to effectively use a per-team environment -
because via team_id specific executors all the tasks of all the DAGs of the
team might be executing in their dedicated environment (so for example
whole team might use the same container image for processors and all the
workers they have in Celery. This is currently not part of AIP-72
b) While AIP-66 had initially the concept of a per-bundle environment, I
thought having it defined by the DAG author and the way how it could be
managed is problematic. Especially if we allow to have "any" binary
environment (whether it's container image or zip containing venv) it has
security issue (unreviewable dependencies) and performance issues for
zipped venv (installing such a bundle takes time, and environmental issues
involving shared libraries cases and architectural differences are
basically unsurmountable - for example that would make it very difficult to
test same DAGs across different architectures and even different versions
of the same OS. Also it was not clear who and how would manage those
environments (i.e. roles of actors involved). While this is a great idea, I
think it's too granular to be effectively managed "per team" - while it is
great for some exceptional cases where we can say "this DAG is so special
that it needs a separate environment". I strongly believe the "per team"
environment is a much more common case (and it coincides with "working as a
team" - where the whole team should be able to easily run and develop
various DAGs. Conway's law in full swing. AIP-67 is addressing all that -
various environments per team, Team Deployment Manager is introduced as
Actor who manages those environments and team configuration.
c) while yes, it's possible to wrap around multiple Airflows with an extra
UI layer and have them managed in common - but it either has to provide a
version of Airflow UI "clone" where you can see several DAGs - possibly
interacting with each other, or you have to have some weird ways of easy
switching or showing multiple Airflow UIs at the same time. First is going
to be super-costly to manage over time as Airflow will evolve, second is a
band-aid really IMHO. And when I think about it - I think mostly about
on-premise users, not managed-services users. Yes managed services could
afford such extra layers and it could be thought of as a nice "paid"
feature of Airflow services. So maybe indeed we should think about it as
"not in community" ? But I know a number of users who have on-premise
Airflow (and for good reasons cannot move to managed Airflow) and we cut
all of them off from being able to have a number of their teams having
pipelines isolated from environment and security point of view, while
centrally monitored and managed.
2) (Vikram, Constance and TP ): *Permissions for Data Assets (related to
AIP-73 + sub-aips)* - I proposed that each team's Data Assets are separated
(same as for DAGs automatically team_id is assigned when parsing the DAGs
by DAG File Processor) - and that they are effectively "separate
namespaces". We have no proposal for permissions of Data Assets (yet) - it
might come later - that's why I do not want to interfere with it and
proposed a simple solution where Data Assets can be matched by URIs (which
become optional in AIP-73) and that the DAG using Data Asset with same URI
might get triggered by a different Data Set from another team but the same
URI only if the "triggered" DAG explicitly hase `allowed_triggering_by = [
"team_2", "team 3"]` (or similar).
3) (Niko, Amogh, TP) - *Scope of "access" to team DAGs/folders/managing
access* - I assume from DAG file processing POV "folder(s)" is currently
the right abstraction (and already handled by standalone DAGFileProcessor).
The nice thing about it is that control over the folder "access" can (and
should) be done outside of Airflow - we delegate it out. Similarly like
Auth Manager access - we do not want to record team members who have access
- but we want to delegate it out. We do not want to keep information about
who is in which team nor handle team changes - this should all be handled
in the corporate identity system. This way both sides are delegated out -
DAG authoring and DAG management. Airflow only knows "team_id", nothing
else once the DAG is parsed (DAG and DAG Assets created by parsing the DAG
have team_id assigned automatically). And In the future we might expand it
- when we have - for example - declarative DAGs submitted via API, the API
can have "team_id" added or when it is created via some kind of factory -
that factory might assign their ids. The "folder" that we have now is
simply to make use of DAG file processor --subdir flag - but actually it
does not matter - once the DAG gets serialized - the DAG and it's assets
will just have "team_id" assigned, and they do not have to come from the
same subdir.
Looking forward to closing those discussions and putting it finally up to a
vote.
J.
On Fri, Jul 19, 2024 at 10:49 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> 1. The roles and responsibilities of what you have called "Organization
>> Deployment Managers" vs. "Team Deployment Managers". This is somewhat
>> different than the model I have seen in practice, so trying to reconcile
>> this in my head. If this is a terminology difference or a role difference
>> or where exactly the line is drawn.
>>
>>
> This is what is not possible in current practice, because we have no such
> distinction. Currently there is only one deployment manager role, but at
> least with my talks to some users and with the approach I proposed, it's
> possible that the "oganization deployment manager" (a.k.a Data Platform
> Team) prepares and manages Airflow as a "whole". - i.e. Running
> airflow scheduler, webserver, connection to the organisation identity
> management system. But then - those people are struggling when different
> teams have different expectations for what dependencies, OS, queues, GPU
> access etc. and Data Platform "Organisation Deployment Manager" wants to
> delegate that to each team's deployment manager - such "team DM" should be
> able - for example to manage their own K8S cluster where their team's job
> will be running - with appropriate dependencies/hardware resources etc.
> And important aspect is that each team can manage it without "bothering"
> the Platform team (but still will be trusted enough to allow installing
> arbitrary packages),- and the platform team will mostly worry about airflow
> itself.
>
>
>> 2. DAG File Processing
>> I am quite perplexed by this and wondering about the overlap between the
>> "execution bundles" concept which Jed had defined as part of AIP-66 and
>> since deleted from there.
>>
>
> Actually that was also the result of discussion we had in the doc with Ash
> and Jed. And I see the potential of joining it. The AIP-66 defined bundle
> environment that could be specified - but the problem with that was that a)
> it assumed it is based on "pip install --target" venvs (which has a host of
> potential problems I explained) and also b) it skipped over on the
> management part of those - i.e. who prepares those environments and manages
> them - is this a DAG author ? if so, this opens a host of security issues
> - because those environments are essentially "binary" and "not reviewable"
> - so there must be some role responsible for it in our security model
> (i.e... "bundle deployment manager"? ) - which sounds suspiciously similar
> "team deployment manager". One of the important differences of AIP-67 here
> is that it explicitly explains and allows to separate all three
> "vulnerable" components per "team" - > DAG file Processor,. Triggerer,.
> Worker. AIP-66 does not mention that. Implicitly assuming that running the
> code from different bundles can be done on shared machines (DAG file
> processor/Schediuler) or even in the same process (Triggerer). AIP-67 adds
> an explicit, strong isolation of code execution between the teams - so that
> the code from different teams are not executed on the same machines,
> containers, but they can be easily separated (by deployment options).
> Without AIP-67 it is impossible - or very difficult and error-prone - to
> make sure that the code of one "bundle" cannot leak to the other "bundle".
> We seem to silently skip over the fact that both DAG File Processor and
> Triggerrer can execute the code in the same machine or process and that
> isolation in such a case is next to impossible. And this is a very
> important aspect of "enterprise security isolation". In some ways even more
> important than isolating access to the same DB.
>
> And I actually am quite open to joining it - and have either "bundle =
> team" or "bundle belongs to team".
>
>
>
>> I will read this doc again for sure.
>> It has grown and evolved so much that at least for me it was quite
>> challenging to grasp. Thanks for working through this.
>>
>> Vikram
>>
>>
>> On Thu, Jul 18, 2024 at 9:39 PM Amogh Desai <am...@gmail.com>
>> wrote:
>>
>> > Nice, thanks for clarifying all this!
>> >
>> > Now that I read the new proposal, it is adding up to me why certain
>> > decisions were made.
>> > The decision to separate the "common" part from the "per team" part
>> adds up
>> > now. It
>> > is a traditional paradigm of separating "control plane" from "compute".
>> >
>> > Thanks & Regards,
>> > Amogh Desai
>> >
>> >
>> > On Mon, Jul 15, 2024 at 8:53 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > > I got the transcript and chat from the last call (thanks Kaxil!) and
>> it
>> > > allowed me to answer a few questions that were asked during my
>> > presentation
>> > > about AIP-67. I updated the AIP document but here is summary:
>> > >
>> > > 1) What about Pools (asked by Elad and Jed, Jorrick): I thought about
>> it
>> > > and I propose that pools could have (optional) team_id added. This
>> will
>> > > allow users to keep common pools (no team_id assigned) and have
>> > > team-specific ones. DAG file processor specific for each team will
>> fail
>> > DAG
>> > > if it tries using a pool that is not common and belonging to "other
>> > team".
>> > > Also each team will be able to have their own "default_pool"
>> configured.
>> > > This will give enough flexibility on 'common vs. team exclusive" use
>> of
>> > > pools.
>> > >
>> > > 2) Isolation for connections (John, Filip, Elad, Kaxil, Amogh, Ash):
>> yes.
>> > > That is part of the design. The connections and variables can be
>> accessed
>> > > per team - AIP-72 will only provide the tasks with connections that
>> > belong
>> > > to the team. Ash mentioned OPA (which might be used for that purpose).
>> > It's
>> > > not defined how exactly it will be implemented in AIP-72, it's not
>> > detailed
>> > > enough, but it can use the very mechanisms that AIP-72 - by only
>> allowing
>> > > "global" connections and "my team" connections to be passed by AIP-72
>> API
>> > > to the task and DAG file processor.
>> > >
>> > > 3) Whether "Team=deployment" - Igor / Vikram ? -> depends on what you
>> > > understand by deployment. i'd say "sub-deployment" - each deployment
>> in a
>> > > "multi-team" environment will consist of the "common" part and each
>> team
>> > > will have their own part (where configuration and management of such
>> team
>> > > deployment parts will be delegated to the team deployment manager).
>> For
>> > > example such deployment managers will be able to build and publish the
>> > > environment (for example container images) used by team A to run
>> Airflow.
>> > > Or change "team" specific configuration.
>> > >
>> > > 4) "This seems like quite a lot of work to share a scheduler and a web
>> > > server. What’s the net benefit of this complexity?" -> Ash, John,
>> Amogh,
>> > > Maciej: Yes. I absolutely see it as a valuable option. It reflects
>> > > organizational structure and needs of many of our users, where they
>> want
>> > to
>> > > manage part of the environment, monitoring of what's going in all of
>> > their
>> > > teams centrally (and manage things like upgrades of Airflow, security
>> > > centrally), while they want to delegate control of environments and
>> > > resources down to their teams. This is the need that I've heard from
>> many
>> > > users who have a "data platform team" that makes Airflow available to
>> > their
>> > > several teams. I think the proposal I have is a nice middle ground
>> that
>> > > follows Conway's law - that architecture of your system should reflect
>> > your
>> > > organizational structure - and what I separated out as "common" parts
>> is
>> > > precisely what "data platform team" would like to manage, where "team
>> > > environment" is something that data platform should (and want to)
>> > delegate
>> > > to their teams.
>> > >
>> > > 5) "I am a little surprised by a shared dataset" - Vikram/Elad : The
>> > > datasets are defined by their URLs and as such - they don't have
>> > > "ownership". As I see it - It's really important who can trigger a DAG
>> > and
>> > > the controls I proposed allow the DAG author to specify "In this DAG
>> it's
>> > > also ok when a different team (specified) triggered the dataset
>> event".
>> > But
>> > > I left a note that it is AIP-73-dependent "Expanded Data Awareness"
>> and
>> > > once we get that explained/clarified I am happy to coordinate with
>> > > Constance and see if we need to do more. Happy to hear more comments
>> on
>> > > that one.
>> > >
>> > > I reflected the 2 points and 5) in the AIP. Looking forward to more
>> > > comments on the proposal - in the AIP or here.
>> > >
>> > > J.
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Jul 9, 2024 at 4:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> > >
>> > > > Hello Everyone,
>> > > >
>> > > > I would like to resume discussion on AIP-67. After going through a
>> > > > number of discussions and clarifications about the scope of Airflow
>> 3,
>> > > > I rewrote the proposal for AIP-67 with the assumption that we will
>> do
>> > > > it for Airflow 3 only - and that it will be based on the new
>> proposed
>> > > > AIP-72 (Task Execution Interface) rather than Airflow 2-only AIP-44
>> > > > Internal API.
>> > > >
>> > > > The updated proposal is here
>> > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>> > > >
>> > > > Feel free to comment there in-line or raise your "big" comments"
>> here,
>> > > > but here is the impact of changing the target to Airflow 3:
>> > > >
>> > > > 1) I proposed to change configuration of Airflow to use more
>> > > > structured TOML than plain "ini" - toml is a successor of "ini" and
>> is
>> > > > largely compatible, but it has arrays, tables and nesting, has good
>> > > > support in Python and is "de-facto" standard for configuration now
>> > > > (pyproject.toml and the like). This was far too big of a change for
>> > > > Airflow 2 but with Airflow 3 it seems very appropriate.
>> > > >
>> > > > 2) On a popular request I added "team_id" as a database field - this
>> > > > has quite a few far-reaching implications and it's ripple-effect on
>> > > > Airflow 2 would be far too big for the "limited" multi-team setup -
>> > > > but since we are going to do full versioning including DB changes in
>> > > > Airflow 3, this is an opportunity to do it well. The implementation
>> > > > detail of it will however depend on our choice of supported
>> databases
>> > > > so there is a little dependency on other decisions here. If we stick
>> > > > with both Postgres and MySQL we will likely have to restructure the
>> DB
>> > > > to have synthetic UUID identifiers in order to add both versioning
>> and
>> > > > multi-team (because of MySQL index limitations).
>> > > >
>> > > > 3) The "proper" team identifier also allows to expand the scope of
>> > > > multi-team to also allow "per-team" connections and variables. Again
>> > > > for Airflow 2 case we could limit it to only the case where
>> > > > connections and variables comes only from "per-team" secrets - but
>> > > > since we are going to have DB identifiers and we are going to -
>> anyhow
>> > > > - reimplement Connections and Variables UI to get rid of FAB models
>> > > > and implement them in reactive technology, it's only a bit more
>> > > > complex to add "per-team" access there.
>> > > >
>> > > > 4) AIP-72 due to its "task" isolation, allows dropping the idea
>> about
>> > > > the "--team" flag from the components. With AIP-72 routing tasks to
>> > > > particular "team" executors" is enough and there is no need to pass
>> > > > the team information via "--team" flag that was originally supposed
>> to
>> > > > limit access of the components to only a single team. For Airflow 2
>> > > > and AIP-44 that was a nice "hack" so that we do not have to carry
>> the
>> > > > "authorization" information together with the task. But since part
>> of
>> > > > AIP-72 is to carry the verifiable meta-data that will allow us to
>> > > > cryptographically verify task provenance, we can drop this hack and
>> > > > rely on AIP-72 implementation.
>> > > >
>> > > > 5) since DB isolation is "given" by AIP-72, we do not have to split
>> > > > the delivery of AIP-67 into two phases (with and without DB
>> isolation)
>> > > > - it will be delivered as a single "with DB isolation" stage.
>> > > >
>> > > > Those are the major differences vs. the proposal from May ( and as
>> you
>> > > > might see it is quite a different scope - and this is really why I
>> > > > insisted on having Airflow 2/ Airflow 3 discussion before we
>> conclude
>> > > > the vote on it.
>> > > >
>> > > > I will go through the proposal on Thursday during our call as
>> planned
>> > > > - but feel free to start discussions and comments before.
>> > > >
>> > > > J.
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Jarek Potiuk <ja...@potiuk.com>.
> 1. The roles and responsibilities of what you have called "Organization
> Deployment Managers" vs. "Team Deployment Managers". This is somewhat
> different than the model I have seen in practice, so trying to reconcile
> this in my head. If this is a terminology difference or a role difference
> or where exactly the line is drawn.
>
>
This is what is not possible in current practice, because we have no such
distinction. Currently there is only one deployment manager role, but at
least with my talks to some users and with the approach I proposed, it's
possible that the "oganization deployment manager" (a.k.a Data Platform
Team) prepares and manages Airflow as a "whole". - i.e. Running
airflow scheduler, webserver, connection to the organisation identity
management system. But then - those people are struggling when different
teams have different expectations for what dependencies, OS, queues, GPU
access etc. and Data Platform "Organisation Deployment Manager" wants to
delegate that to each team's deployment manager - such "team DM" should be
able - for example to manage their own K8S cluster where their team's job
will be running - with appropriate dependencies/hardware resources etc.
And important aspect is that each team can manage it without "bothering"
the Platform team (but still will be trusted enough to allow installing
arbitrary packages),- and the platform team will mostly worry about airflow
itself.
> 2. DAG File Processing
> I am quite perplexed by this and wondering about the overlap between the
> "execution bundles" concept which Jed had defined as part of AIP-66 and
> since deleted from there.
>
Actually that was also the result of discussion we had in the doc with Ash
and Jed. And I see the potential of joining it. The AIP-66 defined bundle
environment that could be specified - but the problem with that was that a)
it assumed it is based on "pip install --target" venvs (which has a host of
potential problems I explained) and also b) it skipped over on the
management part of those - i.e. who prepares those environments and manages
them - is this a DAG author ? if so, this opens a host of security issues
- because those environments are essentially "binary" and "not reviewable"
- so there must be some role responsible for it in our security model
(i.e... "bundle deployment manager"? ) - which sounds suspiciously similar
"team deployment manager". One of the important differences of AIP-67 here
is that it explicitly explains and allows to separate all three
"vulnerable" components per "team" - > DAG file Processor,. Triggerer,.
Worker. AIP-66 does not mention that. Implicitly assuming that running the
code from different bundles can be done on shared machines (DAG file
processor/Schediuler) or even in the same process (Triggerer). AIP-67 adds
an explicit, strong isolation of code execution between the teams - so that
the code from different teams are not executed on the same machines,
containers, but they can be easily separated (by deployment options).
Without AIP-67 it is impossible - or very difficult and error-prone - to
make sure that the code of one "bundle" cannot leak to the other "bundle".
We seem to silently skip over the fact that both DAG File Processor and
Triggerrer can execute the code in the same machine or process and that
isolation in such a case is next to impossible. And this is a very
important aspect of "enterprise security isolation". In some ways even more
important than isolating access to the same DB.
And I actually am quite open to joining it - and have either "bundle =
team" or "bundle belongs to team".
> I will read this doc again for sure.
> It has grown and evolved so much that at least for me it was quite
> challenging to grasp. Thanks for working through this.
>
> Vikram
>
>
> On Thu, Jul 18, 2024 at 9:39 PM Amogh Desai <am...@gmail.com>
> wrote:
>
> > Nice, thanks for clarifying all this!
> >
> > Now that I read the new proposal, it is adding up to me why certain
> > decisions were made.
> > The decision to separate the "common" part from the "per team" part adds
> up
> > now. It
> > is a traditional paradigm of separating "control plane" from "compute".
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Mon, Jul 15, 2024 at 8:53 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I got the transcript and chat from the last call (thanks Kaxil!) and it
> > > allowed me to answer a few questions that were asked during my
> > presentation
> > > about AIP-67. I updated the AIP document but here is summary:
> > >
> > > 1) What about Pools (asked by Elad and Jed, Jorrick): I thought about
> it
> > > and I propose that pools could have (optional) team_id added. This will
> > > allow users to keep common pools (no team_id assigned) and have
> > > team-specific ones. DAG file processor specific for each team will fail
> > DAG
> > > if it tries using a pool that is not common and belonging to "other
> > team".
> > > Also each team will be able to have their own "default_pool"
> configured.
> > > This will give enough flexibility on 'common vs. team exclusive" use of
> > > pools.
> > >
> > > 2) Isolation for connections (John, Filip, Elad, Kaxil, Amogh, Ash):
> yes.
> > > That is part of the design. The connections and variables can be
> accessed
> > > per team - AIP-72 will only provide the tasks with connections that
> > belong
> > > to the team. Ash mentioned OPA (which might be used for that purpose).
> > It's
> > > not defined how exactly it will be implemented in AIP-72, it's not
> > detailed
> > > enough, but it can use the very mechanisms that AIP-72 - by only
> allowing
> > > "global" connections and "my team" connections to be passed by AIP-72
> API
> > > to the task and DAG file processor.
> > >
> > > 3) Whether "Team=deployment" - Igor / Vikram ? -> depends on what you
> > > understand by deployment. i'd say "sub-deployment" - each deployment
> in a
> > > "multi-team" environment will consist of the "common" part and each
> team
> > > will have their own part (where configuration and management of such
> team
> > > deployment parts will be delegated to the team deployment manager). For
> > > example such deployment managers will be able to build and publish the
> > > environment (for example container images) used by team A to run
> Airflow.
> > > Or change "team" specific configuration.
> > >
> > > 4) "This seems like quite a lot of work to share a scheduler and a web
> > > server. What’s the net benefit of this complexity?" -> Ash, John,
> Amogh,
> > > Maciej: Yes. I absolutely see it as a valuable option. It reflects
> > > organizational structure and needs of many of our users, where they
> want
> > to
> > > manage part of the environment, monitoring of what's going in all of
> > their
> > > teams centrally (and manage things like upgrades of Airflow, security
> > > centrally), while they want to delegate control of environments and
> > > resources down to their teams. This is the need that I've heard from
> many
> > > users who have a "data platform team" that makes Airflow available to
> > their
> > > several teams. I think the proposal I have is a nice middle ground that
> > > follows Conway's law - that architecture of your system should reflect
> > your
> > > organizational structure - and what I separated out as "common" parts
> is
> > > precisely what "data platform team" would like to manage, where "team
> > > environment" is something that data platform should (and want to)
> > delegate
> > > to their teams.
> > >
> > > 5) "I am a little surprised by a shared dataset" - Vikram/Elad : The
> > > datasets are defined by their URLs and as such - they don't have
> > > "ownership". As I see it - It's really important who can trigger a DAG
> > and
> > > the controls I proposed allow the DAG author to specify "In this DAG
> it's
> > > also ok when a different team (specified) triggered the dataset event".
> > But
> > > I left a note that it is AIP-73-dependent "Expanded Data Awareness" and
> > > once we get that explained/clarified I am happy to coordinate with
> > > Constance and see if we need to do more. Happy to hear more comments on
> > > that one.
> > >
> > > I reflected the 2 points and 5) in the AIP. Looking forward to more
> > > comments on the proposal - in the AIP or here.
> > >
> > > J.
> > >
> > >
> > >
> > >
> > > On Tue, Jul 9, 2024 at 4:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Hello Everyone,
> > > >
> > > > I would like to resume discussion on AIP-67. After going through a
> > > > number of discussions and clarifications about the scope of Airflow
> 3,
> > > > I rewrote the proposal for AIP-67 with the assumption that we will do
> > > > it for Airflow 3 only - and that it will be based on the new proposed
> > > > AIP-72 (Task Execution Interface) rather than Airflow 2-only AIP-44
> > > > Internal API.
> > > >
> > > > The updated proposal is here
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > > >
> > > > Feel free to comment there in-line or raise your "big" comments"
> here,
> > > > but here is the impact of changing the target to Airflow 3:
> > > >
> > > > 1) I proposed to change configuration of Airflow to use more
> > > > structured TOML than plain "ini" - toml is a successor of "ini" and
> is
> > > > largely compatible, but it has arrays, tables and nesting, has good
> > > > support in Python and is "de-facto" standard for configuration now
> > > > (pyproject.toml and the like). This was far too big of a change for
> > > > Airflow 2 but with Airflow 3 it seems very appropriate.
> > > >
> > > > 2) On a popular request I added "team_id" as a database field - this
> > > > has quite a few far-reaching implications and it's ripple-effect on
> > > > Airflow 2 would be far too big for the "limited" multi-team setup -
> > > > but since we are going to do full versioning including DB changes in
> > > > Airflow 3, this is an opportunity to do it well. The implementation
> > > > detail of it will however depend on our choice of supported databases
> > > > so there is a little dependency on other decisions here. If we stick
> > > > with both Postgres and MySQL we will likely have to restructure the
> DB
> > > > to have synthetic UUID identifiers in order to add both versioning
> and
> > > > multi-team (because of MySQL index limitations).
> > > >
> > > > 3) The "proper" team identifier also allows to expand the scope of
> > > > multi-team to also allow "per-team" connections and variables. Again
> > > > for Airflow 2 case we could limit it to only the case where
> > > > connections and variables comes only from "per-team" secrets - but
> > > > since we are going to have DB identifiers and we are going to -
> anyhow
> > > > - reimplement Connections and Variables UI to get rid of FAB models
> > > > and implement them in reactive technology, it's only a bit more
> > > > complex to add "per-team" access there.
> > > >
> > > > 4) AIP-72 due to its "task" isolation, allows dropping the idea about
> > > > the "--team" flag from the components. With AIP-72 routing tasks to
> > > > particular "team" executors" is enough and there is no need to pass
> > > > the team information via "--team" flag that was originally supposed
> to
> > > > limit access of the components to only a single team. For Airflow 2
> > > > and AIP-44 that was a nice "hack" so that we do not have to carry the
> > > > "authorization" information together with the task. But since part of
> > > > AIP-72 is to carry the verifiable meta-data that will allow us to
> > > > cryptographically verify task provenance, we can drop this hack and
> > > > rely on AIP-72 implementation.
> > > >
> > > > 5) since DB isolation is "given" by AIP-72, we do not have to split
> > > > the delivery of AIP-67 into two phases (with and without DB
> isolation)
> > > > - it will be delivered as a single "with DB isolation" stage.
> > > >
> > > > Those are the major differences vs. the proposal from May ( and as
> you
> > > > might see it is quite a different scope - and this is really why I
> > > > insisted on having Airflow 2/ Airflow 3 discussion before we conclude
> > > > the vote on it.
> > > >
> > > > I will go through the proposal on Thursday during our call as planned
> > > > - but feel free to start discussions and comments before.
> > > >
> > > > J.
> > > >
> > >
> >
>
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
Thanks for the updated write up, Jarek.
I read the Confluence doc a couple of times and left a couple of small
comments there as well.
The biggest items I am trying to wrap my head around now are:
1. The roles and responsibilities of what you have called "Organization
Deployment Managers" vs. "Team Deployment Managers". This is somewhat
different than the model I have seen in practice, so trying to reconcile
this in my head. If this is a terminology difference or a role difference
or where exactly the line is drawn.
2. DAG File Processing
I am quite perplexed by this and wondering about the overlap between the
"execution bundles" concept which Jed had defined as part of AIP-66 and
since deleted from there.
I will read this doc again for sure.
It has grown and evolved so much that at least for me it was quite
challenging to grasp. Thanks for working through this.
Vikram
On Thu, Jul 18, 2024 at 9:39 PM Amogh Desai <am...@gmail.com>
wrote:
> Nice, thanks for clarifying all this!
>
> Now that I read the new proposal, it is adding up to me why certain
> decisions were made.
> The decision to separate the "common" part from the "per team" part adds up
> now. It
> is a traditional paradigm of separating "control plane" from "compute".
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Mon, Jul 15, 2024 at 8:53 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I got the transcript and chat from the last call (thanks Kaxil!) and it
> > allowed me to answer a few questions that were asked during my
> presentation
> > about AIP-67. I updated the AIP document but here is summary:
> >
> > 1) What about Pools (asked by Elad and Jed, Jorrick): I thought about it
> > and I propose that pools could have (optional) team_id added. This will
> > allow users to keep common pools (no team_id assigned) and have
> > team-specific ones. DAG file processor specific for each team will fail
> DAG
> > if it tries using a pool that is not common and belonging to "other
> team".
> > Also each team will be able to have their own "default_pool" configured.
> > This will give enough flexibility on 'common vs. team exclusive" use of
> > pools.
> >
> > 2) Isolation for connections (John, Filip, Elad, Kaxil, Amogh, Ash): yes.
> > That is part of the design. The connections and variables can be accessed
> > per team - AIP-72 will only provide the tasks with connections that
> belong
> > to the team. Ash mentioned OPA (which might be used for that purpose).
> It's
> > not defined how exactly it will be implemented in AIP-72, it's not
> detailed
> > enough, but it can use the very mechanisms that AIP-72 - by only allowing
> > "global" connections and "my team" connections to be passed by AIP-72 API
> > to the task and DAG file processor.
> >
> > 3) Whether "Team=deployment" - Igor / Vikram ? -> depends on what you
> > understand by deployment. i'd say "sub-deployment" - each deployment in a
> > "multi-team" environment will consist of the "common" part and each team
> > will have their own part (where configuration and management of such team
> > deployment parts will be delegated to the team deployment manager). For
> > example such deployment managers will be able to build and publish the
> > environment (for example container images) used by team A to run Airflow.
> > Or change "team" specific configuration.
> >
> > 4) "This seems like quite a lot of work to share a scheduler and a web
> > server. What’s the net benefit of this complexity?" -> Ash, John, Amogh,
> > Maciej: Yes. I absolutely see it as a valuable option. It reflects
> > organizational structure and needs of many of our users, where they want
> to
> > manage part of the environment, monitoring of what's going in all of
> their
> > teams centrally (and manage things like upgrades of Airflow, security
> > centrally), while they want to delegate control of environments and
> > resources down to their teams. This is the need that I've heard from many
> > users who have a "data platform team" that makes Airflow available to
> their
> > several teams. I think the proposal I have is a nice middle ground that
> > follows Conway's law - that architecture of your system should reflect
> your
> > organizational structure - and what I separated out as "common" parts is
> > precisely what "data platform team" would like to manage, where "team
> > environment" is something that data platform should (and want to)
> delegate
> > to their teams.
> >
> > 5) "I am a little surprised by a shared dataset" - Vikram/Elad : The
> > datasets are defined by their URLs and as such - they don't have
> > "ownership". As I see it - It's really important who can trigger a DAG
> and
> > the controls I proposed allow the DAG author to specify "In this DAG it's
> > also ok when a different team (specified) triggered the dataset event".
> But
> > I left a note that it is AIP-73-dependent "Expanded Data Awareness" and
> > once we get that explained/clarified I am happy to coordinate with
> > Constance and see if we need to do more. Happy to hear more comments on
> > that one.
> >
> > I reflected the 2 points and 5) in the AIP. Looking forward to more
> > comments on the proposal - in the AIP or here.
> >
> > J.
> >
> >
> >
> >
> > On Tue, Jul 9, 2024 at 4:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Hello Everyone,
> > >
> > > I would like to resume discussion on AIP-67. After going through a
> > > number of discussions and clarifications about the scope of Airflow 3,
> > > I rewrote the proposal for AIP-67 with the assumption that we will do
> > > it for Airflow 3 only - and that it will be based on the new proposed
> > > AIP-72 (Task Execution Interface) rather than Airflow 2-only AIP-44
> > > Internal API.
> > >
> > > The updated proposal is here
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> > >
> > > Feel free to comment there in-line or raise your "big" comments" here,
> > > but here is the impact of changing the target to Airflow 3:
> > >
> > > 1) I proposed to change configuration of Airflow to use more
> > > structured TOML than plain "ini" - toml is a successor of "ini" and is
> > > largely compatible, but it has arrays, tables and nesting, has good
> > > support in Python and is "de-facto" standard for configuration now
> > > (pyproject.toml and the like). This was far too big of a change for
> > > Airflow 2 but with Airflow 3 it seems very appropriate.
> > >
> > > 2) On a popular request I added "team_id" as a database field - this
> > > has quite a few far-reaching implications and it's ripple-effect on
> > > Airflow 2 would be far too big for the "limited" multi-team setup -
> > > but since we are going to do full versioning including DB changes in
> > > Airflow 3, this is an opportunity to do it well. The implementation
> > > detail of it will however depend on our choice of supported databases
> > > so there is a little dependency on other decisions here. If we stick
> > > with both Postgres and MySQL we will likely have to restructure the DB
> > > to have synthetic UUID identifiers in order to add both versioning and
> > > multi-team (because of MySQL index limitations).
> > >
> > > 3) The "proper" team identifier also allows to expand the scope of
> > > multi-team to also allow "per-team" connections and variables. Again
> > > for Airflow 2 case we could limit it to only the case where
> > > connections and variables comes only from "per-team" secrets - but
> > > since we are going to have DB identifiers and we are going to - anyhow
> > > - reimplement Connections and Variables UI to get rid of FAB models
> > > and implement them in reactive technology, it's only a bit more
> > > complex to add "per-team" access there.
> > >
> > > 4) AIP-72 due to its "task" isolation, allows dropping the idea about
> > > the "--team" flag from the components. With AIP-72 routing tasks to
> > > particular "team" executors" is enough and there is no need to pass
> > > the team information via "--team" flag that was originally supposed to
> > > limit access of the components to only a single team. For Airflow 2
> > > and AIP-44 that was a nice "hack" so that we do not have to carry the
> > > "authorization" information together with the task. But since part of
> > > AIP-72 is to carry the verifiable meta-data that will allow us to
> > > cryptographically verify task provenance, we can drop this hack and
> > > rely on AIP-72 implementation.
> > >
> > > 5) since DB isolation is "given" by AIP-72, we do not have to split
> > > the delivery of AIP-67 into two phases (with and without DB isolation)
> > > - it will be delivered as a single "with DB isolation" stage.
> > >
> > > Those are the major differences vs. the proposal from May ( and as you
> > > might see it is quite a different scope - and this is really why I
> > > insisted on having Airflow 2/ Airflow 3 discussion before we conclude
> > > the vote on it.
> > >
> > > I will go through the proposal on Thursday during our call as planned
> > > - but feel free to start discussions and comments before.
> > >
> > > J.
> > >
> >
>
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Amogh Desai <am...@gmail.com>.
Nice, thanks for clarifying all this!
Now that I read the new proposal, it is adding up to me why certain
decisions were made.
The decision to separate the "common" part from the "per team" part adds up
now. It
is a traditional paradigm of separating "control plane" from "compute".
Thanks & Regards,
Amogh Desai
On Mon, Jul 15, 2024 at 8:53 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> I got the transcript and chat from the last call (thanks Kaxil!) and it
> allowed me to answer a few questions that were asked during my presentation
> about AIP-67. I updated the AIP document but here is summary:
>
> 1) What about Pools (asked by Elad and Jed, Jorrick): I thought about it
> and I propose that pools could have (optional) team_id added. This will
> allow users to keep common pools (no team_id assigned) and have
> team-specific ones. DAG file processor specific for each team will fail DAG
> if it tries using a pool that is not common and belonging to "other team".
> Also each team will be able to have their own "default_pool" configured.
> This will give enough flexibility on 'common vs. team exclusive" use of
> pools.
>
> 2) Isolation for connections (John, Filip, Elad, Kaxil, Amogh, Ash): yes.
> That is part of the design. The connections and variables can be accessed
> per team - AIP-72 will only provide the tasks with connections that belong
> to the team. Ash mentioned OPA (which might be used for that purpose). It's
> not defined how exactly it will be implemented in AIP-72, it's not detailed
> enough, but it can use the very mechanisms that AIP-72 - by only allowing
> "global" connections and "my team" connections to be passed by AIP-72 API
> to the task and DAG file processor.
>
> 3) Whether "Team=deployment" - Igor / Vikram ? -> depends on what you
> understand by deployment. i'd say "sub-deployment" - each deployment in a
> "multi-team" environment will consist of the "common" part and each team
> will have their own part (where configuration and management of such team
> deployment parts will be delegated to the team deployment manager). For
> example such deployment managers will be able to build and publish the
> environment (for example container images) used by team A to run Airflow.
> Or change "team" specific configuration.
>
> 4) "This seems like quite a lot of work to share a scheduler and a web
> server. What’s the net benefit of this complexity?" -> Ash, John, Amogh,
> Maciej: Yes. I absolutely see it as a valuable option. It reflects
> organizational structure and needs of many of our users, where they want to
> manage part of the environment, monitoring of what's going in all of their
> teams centrally (and manage things like upgrades of Airflow, security
> centrally), while they want to delegate control of environments and
> resources down to their teams. This is the need that I've heard from many
> users who have a "data platform team" that makes Airflow available to their
> several teams. I think the proposal I have is a nice middle ground that
> follows Conway's law - that architecture of your system should reflect your
> organizational structure - and what I separated out as "common" parts is
> precisely what "data platform team" would like to manage, where "team
> environment" is something that data platform should (and want to) delegate
> to their teams.
>
> 5) "I am a little surprised by a shared dataset" - Vikram/Elad : The
> datasets are defined by their URLs and as such - they don't have
> "ownership". As I see it - It's really important who can trigger a DAG and
> the controls I proposed allow the DAG author to specify "In this DAG it's
> also ok when a different team (specified) triggered the dataset event". But
> I left a note that it is AIP-73-dependent "Expanded Data Awareness" and
> once we get that explained/clarified I am happy to coordinate with
> Constance and see if we need to do more. Happy to hear more comments on
> that one.
>
> I reflected the 2 points and 5) in the AIP. Looking forward to more
> comments on the proposal - in the AIP or here.
>
> J.
>
>
>
>
> On Tue, Jul 9, 2024 at 4:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hello Everyone,
> >
> > I would like to resume discussion on AIP-67. After going through a
> > number of discussions and clarifications about the scope of Airflow 3,
> > I rewrote the proposal for AIP-67 with the assumption that we will do
> > it for Airflow 3 only - and that it will be based on the new proposed
> > AIP-72 (Task Execution Interface) rather than Airflow 2-only AIP-44
> > Internal API.
> >
> > The updated proposal is here
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
> >
> > Feel free to comment there in-line or raise your "big" comments" here,
> > but here is the impact of changing the target to Airflow 3:
> >
> > 1) I proposed to change configuration of Airflow to use more
> > structured TOML than plain "ini" - toml is a successor of "ini" and is
> > largely compatible, but it has arrays, tables and nesting, has good
> > support in Python and is "de-facto" standard for configuration now
> > (pyproject.toml and the like). This was far too big of a change for
> > Airflow 2 but with Airflow 3 it seems very appropriate.
> >
> > 2) On a popular request I added "team_id" as a database field - this
> > has quite a few far-reaching implications and it's ripple-effect on
> > Airflow 2 would be far too big for the "limited" multi-team setup -
> > but since we are going to do full versioning including DB changes in
> > Airflow 3, this is an opportunity to do it well. The implementation
> > detail of it will however depend on our choice of supported databases
> > so there is a little dependency on other decisions here. If we stick
> > with both Postgres and MySQL we will likely have to restructure the DB
> > to have synthetic UUID identifiers in order to add both versioning and
> > multi-team (because of MySQL index limitations).
> >
> > 3) The "proper" team identifier also allows to expand the scope of
> > multi-team to also allow "per-team" connections and variables. Again
> > for Airflow 2 case we could limit it to only the case where
> > connections and variables comes only from "per-team" secrets - but
> > since we are going to have DB identifiers and we are going to - anyhow
> > - reimplement Connections and Variables UI to get rid of FAB models
> > and implement them in reactive technology, it's only a bit more
> > complex to add "per-team" access there.
> >
> > 4) AIP-72 due to its "task" isolation, allows dropping the idea about
> > the "--team" flag from the components. With AIP-72 routing tasks to
> > particular "team" executors" is enough and there is no need to pass
> > the team information via "--team" flag that was originally supposed to
> > limit access of the components to only a single team. For Airflow 2
> > and AIP-44 that was a nice "hack" so that we do not have to carry the
> > "authorization" information together with the task. But since part of
> > AIP-72 is to carry the verifiable meta-data that will allow us to
> > cryptographically verify task provenance, we can drop this hack and
> > rely on AIP-72 implementation.
> >
> > 5) since DB isolation is "given" by AIP-72, we do not have to split
> > the delivery of AIP-67 into two phases (with and without DB isolation)
> > - it will be delivered as a single "with DB isolation" stage.
> >
> > Those are the major differences vs. the proposal from May ( and as you
> > might see it is quite a different scope - and this is really why I
> > insisted on having Airflow 2/ Airflow 3 discussion before we conclude
> > the vote on it.
> >
> > I will go through the proposal on Thursday during our call as planned
> > - but feel free to start discussions and comments before.
> >
> > J.
> >
>
Re: [DISCUSS] AIP-67 Multi-team deployment of Airflow components (reloaded)
Posted by Jarek Potiuk <ja...@potiuk.com>.
I got the transcript and chat from the last call (thanks Kaxil!) and it
allowed me to answer a few questions that were asked during my presentation
about AIP-67. I updated the AIP document but here is summary:
1) What about Pools (asked by Elad and Jed, Jorrick): I thought about it
and I propose that pools could have (optional) team_id added. This will
allow users to keep common pools (no team_id assigned) and have
team-specific ones. DAG file processor specific for each team will fail DAG
if it tries using a pool that is not common and belonging to "other team".
Also each team will be able to have their own "default_pool" configured.
This will give enough flexibility on 'common vs. team exclusive" use of
pools.
2) Isolation for connections (John, Filip, Elad, Kaxil, Amogh, Ash): yes.
That is part of the design. The connections and variables can be accessed
per team - AIP-72 will only provide the tasks with connections that belong
to the team. Ash mentioned OPA (which might be used for that purpose). It's
not defined how exactly it will be implemented in AIP-72, it's not detailed
enough, but it can use the very mechanisms that AIP-72 - by only allowing
"global" connections and "my team" connections to be passed by AIP-72 API
to the task and DAG file processor.
3) Whether "Team=deployment" - Igor / Vikram ? -> depends on what you
understand by deployment. i'd say "sub-deployment" - each deployment in a
"multi-team" environment will consist of the "common" part and each team
will have their own part (where configuration and management of such team
deployment parts will be delegated to the team deployment manager). For
example such deployment managers will be able to build and publish the
environment (for example container images) used by team A to run Airflow.
Or change "team" specific configuration.
4) "This seems like quite a lot of work to share a scheduler and a web
server. What’s the net benefit of this complexity?" -> Ash, John, Amogh,
Maciej: Yes. I absolutely see it as a valuable option. It reflects
organizational structure and needs of many of our users, where they want to
manage part of the environment, monitoring of what's going in all of their
teams centrally (and manage things like upgrades of Airflow, security
centrally), while they want to delegate control of environments and
resources down to their teams. This is the need that I've heard from many
users who have a "data platform team" that makes Airflow available to their
several teams. I think the proposal I have is a nice middle ground that
follows Conway's law - that architecture of your system should reflect your
organizational structure - and what I separated out as "common" parts is
precisely what "data platform team" would like to manage, where "team
environment" is something that data platform should (and want to) delegate
to their teams.
5) "I am a little surprised by a shared dataset" - Vikram/Elad : The
datasets are defined by their URLs and as such - they don't have
"ownership". As I see it - It's really important who can trigger a DAG and
the controls I proposed allow the DAG author to specify "In this DAG it's
also ok when a different team (specified) triggered the dataset event". But
I left a note that it is AIP-73-dependent "Expanded Data Awareness" and
once we get that explained/clarified I am happy to coordinate with
Constance and see if we need to do more. Happy to hear more comments on
that one.
I reflected the 2 points and 5) in the AIP. Looking forward to more
comments on the proposal - in the AIP or here.
J.
On Tue, Jul 9, 2024 at 4:48 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello Everyone,
>
> I would like to resume discussion on AIP-67. After going through a
> number of discussions and clarifications about the scope of Airflow 3,
> I rewrote the proposal for AIP-67 with the assumption that we will do
> it for Airflow 3 only - and that it will be based on the new proposed
> AIP-72 (Task Execution Interface) rather than Airflow 2-only AIP-44
> Internal API.
>
> The updated proposal is here
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components
>
> Feel free to comment there in-line or raise your "big" comments" here,
> but here is the impact of changing the target to Airflow 3:
>
> 1) I proposed to change configuration of Airflow to use more
> structured TOML than plain "ini" - toml is a successor of "ini" and is
> largely compatible, but it has arrays, tables and nesting, has good
> support in Python and is "de-facto" standard for configuration now
> (pyproject.toml and the like). This was far too big of a change for
> Airflow 2 but with Airflow 3 it seems very appropriate.
>
> 2) On a popular request I added "team_id" as a database field - this
> has quite a few far-reaching implications and it's ripple-effect on
> Airflow 2 would be far too big for the "limited" multi-team setup -
> but since we are going to do full versioning including DB changes in
> Airflow 3, this is an opportunity to do it well. The implementation
> detail of it will however depend on our choice of supported databases
> so there is a little dependency on other decisions here. If we stick
> with both Postgres and MySQL we will likely have to restructure the DB
> to have synthetic UUID identifiers in order to add both versioning and
> multi-team (because of MySQL index limitations).
>
> 3) The "proper" team identifier also allows to expand the scope of
> multi-team to also allow "per-team" connections and variables. Again
> for Airflow 2 case we could limit it to only the case where
> connections and variables comes only from "per-team" secrets - but
> since we are going to have DB identifiers and we are going to - anyhow
> - reimplement Connections and Variables UI to get rid of FAB models
> and implement them in reactive technology, it's only a bit more
> complex to add "per-team" access there.
>
> 4) AIP-72 due to its "task" isolation, allows dropping the idea about
> the "--team" flag from the components. With AIP-72 routing tasks to
> particular "team" executors" is enough and there is no need to pass
> the team information via "--team" flag that was originally supposed to
> limit access of the components to only a single team. For Airflow 2
> and AIP-44 that was a nice "hack" so that we do not have to carry the
> "authorization" information together with the task. But since part of
> AIP-72 is to carry the verifiable meta-data that will allow us to
> cryptographically verify task provenance, we can drop this hack and
> rely on AIP-72 implementation.
>
> 5) since DB isolation is "given" by AIP-72, we do not have to split
> the delivery of AIP-67 into two phases (with and without DB isolation)
> - it will be delivered as a single "with DB isolation" stage.
>
> Those are the major differences vs. the proposal from May ( and as you
> might see it is quite a different scope - and this is really why I
> insisted on having Airflow 2/ Airflow 3 discussion before we conclude
> the vote on it.
>
> I will go through the proposal on Thursday during our call as planned
> - but feel free to start discussions and comments before.
>
> J.
>