You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2021/11/05 17:38:02 UTC

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Hello Ian, Everyone,

I wonder if there are any notes from the meeting in April? Has there
been any more work on that one from Cloudera to formalize and plan
work on it?

I was not able to participate, but I think it's about the time to
seriously start work on that and I am super happy to take more lead on
this project and involve all the interested parties. The ideas
described in the email and discussed after are I think super
reasonable and definitely necessary to get to the multi-tenancy and I
believe that there are already ideas that can be turned into reality
rather soon. I had a talk today also with the Google Composer team and
they are also fully on board with dedicating a lot of effort on this
one (and their ideas are I think super-aligned with Cloudera's), so I
think we have a critical mass and engineering power to make it happen
:)

I plan to put quite a lot of focus on that one over the coming months
and I am happy to lead or co-lead the AIP and take a big part in
implementation.

Possibly we should create a special interest group around that and
start drafting the AIP proposals in a smaller group of people who are
interested and start planning the work. I already have some ideas
where we could start gradually implementing it (of course after we
prepare the AIP and get it through the community's approval process).

How does it sound?

J.

On Wed, Apr 21, 2021 at 8:56 AM Ian Buss <ia...@gmail.com> wrote:
>
> Yes, no invite required. See you tomorrow!
> On 21 Apr 2021, 07:46 +0100, Sumit Maheshwari <ms...@apache.org>, wrote:
>
> I'll join as well (I believe the zoom link will work without an invite)
>
> On Wed, Apr 21, 2021 at 10:48 AM Dimitris Stafylarakis <xa...@gmail.com> wrote:
>>
>> hi all,
>>
>> great to read about this, I'd like to join in! Can I just join using the zoom link tomorrow or do I need an invitation? (If I do need one, please invite me :))
>>
>> cheers
>>
>>
>> On Wed, Apr 14, 2021 at 8:15 PM Daniel Imberman <da...@gmail.com> wrote:
>>>
>>> Thank you Ian,
>>>
>>> I’ve invited everyone on this thread to the meeting with that zoom link. Anyone else who wants to join can add the calendar event here calendar.google.com/event?action=TEMPLATE&tmeid=Mm4zN2Q3MnFwNnBqbW9hMmNocXMyNzJpdHYgZGFuaWVsQGFzdHJvbm9tZXIuaW8&tmsrc=daniel@astronomer.io
>>>
>>> On Wed, Apr 14, 2021 at 11:05 AM, Ian Buss <ia...@gmail.com> wrote:
>>>
>>> If this works for everyone, here's a zoom link for Thursday 8AM PST: https://cloudera.zoom.us/j/99928254235?pwd=VTFlQk4vQjQ5Z2JzUDM3ZWZKKy9MQT09
>>>
>>> Happy to move or use an alternate method as needed.
>>>
>>> On Wed, Apr 14, 2021 at 6:58 PM Daniel Imberman <da...@gmail.com> wrote:
>>>>
>>>> Thursday works for me!
>>>>
>>>> On Wed, Apr 14, 2021 at 10:05 AM, Ian Buss <ia...@gmail.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I actually can’t do Wednesday next week as I’m moving house :) Any chance we could do Thursday or Friday at the same time?
>>>>
>>>> Cheers
>>>>
>>>> Ian
>>>> On 14 Apr 2021, 17:49 +0100, Kaxil Naik <ka...@gmail.com>, wrote:
>>>>
>>>> Just few comments here:
>>>>
>>>> Currently -- atleast for the foreseeable future Airflow workers will need access to the DAG Files, so workers can not run using the Serialized DAGs.
>>>>
>>>> Also serialized DAGs do not even have all the info needed for it to run it. Currently the serialization happens in the parsing process in the scheduler which can be in future separated as a separator "parsining" component, but that won't solve the "isolation" problem you are trying to solve. The only current way it can be solved is pickling -- and we have strictly decided against using pickling for DAGs.
>>>>
>>>> The idea in Statement (2) & (3) would help solve the isolation problem in (1) and can be done with some work now.
>>>>
>>>> Happy to talk about it in more detail here or on call, the time Daniel suggested works for me.
>>>>
>>>> Regards,
>>>> Kaxil
>>>>
>>>> On Wed, Apr 14, 2021 at 5:35 PM Daniel Imberman <da...@gmail.com> wrote:
>>>>>
>>>>> How about Wednesday, April 21 at 8:00AM PST?
>>>>>
>>>>> On Wed, Apr 14, 2021 at 9:33 AM, Xinbin Huang <bi...@gmail.com> wrote:
>>>>>
>>>>> I am available any days.
>>>>>
>>>>> On Wed, Apr 14, 2021, 9:32 AM Daniel Imberman <da...@gmail.com> wrote:
>>>>>>
>>>>>> Hi everyone!
>>>>>>
>>>>>> Would people be available around 8AM/9AM PST some point next week? I’m in PST and Ian is UTC+1 so would be great to find a timezone that accomodates everyone.
>>>>>>
>>>>>> Daniel
>>>>>> On Wed, Apr 14, 2021 at 6:26 AM, Ryan Hatter <ry...@gmail.com> wrote:
>>>>>>
>>>>>> I’d also like to be added please :)
>>>>>>
>>>>>> On Apr 13, 2021, at 21:27, Xinbin Huang <bi...@gmail.com> wrote:
>>>>>>
>>>>>> 
>>>>>> Hi Daniel & Ian,
>>>>>>
>>>>>> I am also interested in the idea of a serialization representation that can be executed by workers directly. Can you also add me to the call?
>>>>>>
>>>>>> Thanks
>>>>>> Bin
>>>>>>
>>>>>> On Tue, Apr 13, 2021 at 2:49 PM Ian Buss <ia...@gmail.com> wrote:
>>>>>>>
>>>>>>> Daniel,
>>>>>>>
>>>>>>> Thanks for your warm welcome and quick response and the advice on providers! Will certainly check out the examples you sent.
>>>>>>>
>>>>>>> 1. An "airflow register" command definitely sounds promising, would love to collaborate on an AIP there so let's set something up.
>>>>>>> 2. We use KubernetesExecutor exclusively as well. We've noticed significant additional load on the metadata DB as we scale up task pods so I've also thought about an API-based approach. Such an API could also open up the possibility of per-task security tokens which are injected by the scheduler, which should improve the security of such a system. Food for thought at least. I will start putting some of these thoughts down on paper in a sharable format.
>>>>>>>
>>>>>>> Ian
>>>>>>>
>>>>>>> On Tue, Apr 13, 2021 at 7:46 PM Daniel Imberman <da...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Ian,
>>>>>>>>
>>>>>>>>
>>>>>>>> Firstly, welcome to the Airflow community :). I'm glad to hear you've had a positive experience so far. It's great to hear that you want to contribute back, and I think that multi-tenancy/DAG isolation is a pretty fantastic project for the community as a whole (a lot of things are are things we want but are limited by hours in a day).
>>>>>>>>
>>>>>>>>
>>>>>>>> 1. I've personally been kicking around some ideas lately about an "airflow register" command that would write the DAG into the metadata DB in a way that could be "gettable" by the workers via the API. This work is very early. I'd love to get some help on it. Perhaps we can set up a zoom chat to discuss drafting an AIP?
>>>>>>>>
>>>>>>>>
>>>>>>>> 2. Limiting worker access to the DB is not only good security practice; it also opens up the door to a lot of valuable features. This feature would be especially close to my heart as it would make the KubernetesExecutor significantly more efficient. It should be possible to set up a system where the workers only ever speak to an API server and never need to touch the DB.
>>>>>>>>
>>>>>>>>
>>>>>>>> 3. This is not something I personally have insight into, but I think it sounds like a good idea.
>>>>>>>>
>>>>>>>>
>>>>>>>> Finally, addressing your question about a Cloudera provider. If anything, it would probably give the provider _more_ legitimacy if you hosted it under the Cloudera GitHub org (we very purposely created the provider packages with this workflow in mind). There are multiple places where we can work to surface this provider so it is easy to find and use.
>>>>>>>>
>>>>>>>>
>>>>>>>> Astronomer has a pretty good sample provider here. One example of it running in the wild is the Great Expectations provider here. I'd also be glad to get you in contact with people who have built providers in the past to help you with that process.
>>>>>>>>
>>>>>>>>
>>>>>>>> Looking forward to seeing some of these things come to fruition!
>>>>>>>>
>>>>>>>>
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 13, 2021 at 9:43 AM, Ian Buss <ia...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> First a quick introduction: I'm an engineer with Cloudera working on our Data Engineering product (CDE). Airflow is working great for us so far. We've been looking into how we can enhance the multi-tenancy story of Apache Airflow as we currently deploy it. We have the following areas which we'd like (with community consensus) to work on and contribute back to Apache Airflow to enhance the isolation between tenants in a single Airflow deployment.
>>>>>>>>
>>>>>>>> 1. Isolating code execution and parsing of DAG files. At the moment, DAG files are parsed in a few locations in Airflow, including the scheduler and in tasks. There is already the concept of DAG serialization (and we're using that for the web component) but we'd be interested to see if we can sandbox the execution of arbitrary user code to a locked down process/container without full access to the metadata DB and connection secrets etc. The idea would be to parse and serialize the DAG in this isolated container and pass back a serialized representation for persistence in the DB. Has anyone explored this idea?
>>>>>>>>
>>>>>>>> 2. Limiting task access to the metadata DB. It would be great if we could remove the requirement for tasks to have full access to the metadata DB and to report task status in a different (but still scalable) way. We'd need to tackle access or injection of connection, variable and xcom data as well for each task naturally.
>>>>>>>>
>>>>>>>> 3. Finer-grained access controls on connection secrets. Right now, although there are nice at-rest encryption options with Fernet or Vault, IIUC any DAG can access any connection (and thus any secret). Since the "run as" user is largely defined within the DAG and its tasks, this is challenging for a multi-tenant environment (see caveat below)
>>>>>>>>
>>>>>>>> Caveat: It's definitely noted that to some extent we should assume that an Airflow deployment is a "trusted" environment and that best practices such as git+PR workflows are the gold standard and that any malicious code and dependencies should be identified through this process. Also that there is a clear admin role for connection management etc.
>>>>>>>>
>>>>>>>> We have some ideas informally sketched out as to how to address the above but would be keen to hear the community opinion on this and to see if anyone is keen to collaborate on designs and implementation, or to hear if anything is already in the works. In particular I noticed that the very first improvement proposal (AIP-1) addresses much of the above :). However, it seems fairly dormant at the moment.
>>>>>>>>
>>>>>>>> One other question: we have a provider (operators and hooks) for interacting with Cloudera components that we'd like to contribute to the project. The provider FAQs indicate that new provider contributions are still welcome in the project in 2.x, is that accurate?
>>>>>>>>
>>>>>>>> Thanks in advance!
>>>>>>>>
>>>>>>>> Ian

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
Hey everyone,

Great to see others chiming-in :). It's really great that we have
various stakeholders and users taking part in the discussion - and I
hope what we come up with will be something that will be driven and
supported by the whole community.
Also John from Amazon promised to bring his team in and I really see
that we are getting in the right direction and what we come up with
will be well thought/discussed.

TL;DR; I am genuinely excited. However I also know we have to hold
some horses and cannot get mine (and others :) excitement to go too
wild. So I propose that we complete discussions and approvals of
AIP-43 and AIP-44 before we go further.


I think there are multiple "layers" of isolation we can talk about.
AIP-43 and AIP-44 build on the foundations laid in Airflow 2 but I am
well aware this is a "long haul" and there will be multiple AIPs that
will follow and make it really robust. We have discussed so far the
"fine-grained" access  level, and also those areas you talk about Ping
Zhang are important  and are definitely important to be looked at.

I think the "docker" level isolation (as an option) is quite possible
and really interesting direction. The separation of Dag processing to
a different component is slightly "higher" level of isolation, because
it allows for example (this is part of the
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+DAG+Processor+separation
) to run DAG processors for separate tenants not only in separate
process or docker container, but also on a separate virtual machine
and even in separate "security zone". This is the basic assumption
behind AIP-43, to be able (if there is a need) to physically separate
dag processing for different teams. And in fact Docker separation as
optional (future) isolation layer for DagProcessor has been mentioned
in AIP-44: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API
see "Securing access to authorization token in DagProcessor and
Triggerer" chapter. Similarly "DAG Serialization" discussion you
started is cool and I think we should closely take a look at that.

Of course we will have to think about how it fits in Airflow 2 and
current features that are not there (or are in early state)  in
Airflow 1.10.

However, I have an important request here.

We can extremely easily get lost and distracted in the number of
different threads that are opened in this direction. If we open too
many threads and start discussing all of them at once, I think we can
simply spread ourselves too thin.

While there should be a "grand vision" worked out I think we should
not actively discuss more than 2 AIPs, and only move on to "next
steps" once we discussed and agreed (and approved) the work on the
"foundations" so that we keep focus.

I think specifically AIP-43 and AIP-44 are designed in the way that
they are "opening" more of further discussions than closing them. The
changes proposed in AIP-43 and 44 are more of reshuffling some of the
existing logic and separating it - without impacting the internals of
Airflow (such as running tasks) and they are laying foundation to
further steps.
My proposal is to discuss those two first (also Ping Zhang -
specifically on how AIP-43 and AIP-44 would interact with the future
proposals of yours), see if we can get consensus, approval and then we
can move on with discussing the follow up steps. Especially if we all
agree that AIP-43 and AIP-44 are more of the "opener" and they are not
"clashing

I am really looking forward to discussing and taking the next
steps/discussing more AIPs, I am just afraid that if we want to
discuss and clarify too many things too quickly in the area of
multi-tenancy, we will quickly get lost in details.

What I propose is that some time mid January (where we hopefully have
consensus and approval for AIP-43/44) and I will ask you for a public
demo and some deeper explanation on how you've implemented the
features you explained, so that (especially people involved in Airflow
2 can comment on how it maps into current features - Airflow 2.2 is
really far from the 1.10.* version that your implementation is based
on and there are some interesting new features (like @task.docker
decorator) that might influence the way we think about docker
isolation.

In the meantime - I would really appreciate it if you could get
involved in AIP-43 and AIP-44 discussions - I think that might be
super-useful that we are all on the same page here and we have some
foundation to discuss further steps.

J.


On Fri, Dec 17, 2021 at 7:18 AM Ping Zhang <pi...@umich.edu> wrote:
>
> Ah, I guess I am a little bit late now.
>
> In terms of the
>>
>> Isolating code execution and parsing of DAG files.
>
>
> In Airbnb, we use `Docker runtime isolation for airflow tasks` plus `Parsing Service` to totally isolate the dag parsing, task execution from the airflow infra runtime.
>
> `Docker runtime isolation for airflow tasks` (see email thread: [DISCUSS] Docker runtime isolation for airflow tasks) introduces a docker layer which wraps the dag parsing and task execution so each dag file can have its own docker runtime.
> `Parsing Service` totally removes the dag file parsing from the scheduler.
>
> These two features have been running in Airbnb's production for close to 1 year. I am working on open source them.
>
> Ping
>
>
> Best wishes
>
> Ping Zhang
>
>
> On Wed, Dec 1, 2021 at 11:43 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Very good/important thoughts.
>>
>> From the discussions and looking at the (upcoming) proposals from
>> Mateusz  we are going to have this all optional:
>>
>> We plan to have two config options:
>>
>> * DB Isolation mode for separating out DB access
>> * Standalone DAG processor
>>
>> I totally agree that standalone/quick/dirty access mode for Airflow
>> should be the default (so business as usual). Moreover - that will
>> allow the introduction of the multi-tenant mode as "optional" in
>> otherwise backwards-compatible Airflow - i.e. it could start to be
>> available in 2.x line.
>>
>> Actually (and this is something up for discussion in the AIP) we could
>> introduce "soft" multi-tenancy mode, where DB access will be still
>> possible but flagged as a warning.
>> This could give the user an option to switch gradually their DAGs to
>> the multi-tenancy mode, if they are already using some direct DB
>> access (for example in their callbacks or custom operator).
>>
>> Also I think part of the AIP and proof of concept while discussing it
>> should be initially rough, and later more comprehensive performance
>> testing of some "real-life" scenarios.
>>
>> J.
>>
>> On Wed, Dec 1, 2021 at 6:12 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>> >
>> > I look forward to seeing these propsals etc.
>> >
>> > One thought I've just had is that we should be careful about two things when taking on this work:
>> >
>> > 1. That performance is not impacted (specifically of the scheduler "throughput") -- at least when only a single "tenant" is in use if not for all.
>> > 2. That we don't make the deployment story more complex for the small deployments, nor for the "getting started on a laptop" initial user experience.
>> >
>> > -ash
>> >
>> > On Fri, Nov 26 2021 at 18:23:32 +0100, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > Recording available here: https://drive.google.com/file/d/1Irw7qxxeTOHZTfdvT5lAbGowIfm9DHzi/view On Fri, Nov 26, 2021 at 6:17 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > Thanks for the meeting this morning/afternoon :) ! It was very productive, I believe: The notes are available here: https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit The most important take is that it looks like if the use cases are slightly different, we are all aligned of what needs to be done and how Action points: * Composer team (Mateusz) will soon submit AIP's (they are close to be ready for proposing) for * DB access isolation * Separating out DAG processor * Cloudera team (Ian) will work on follow-up Fine-grained resource access AIP - it can be implemented as next steps. The two AIPs above will implement "coarse" access level but in the way that the "fine-grained" access will be possible to be plugged-in I recorded the meeting and I am waiting for the video to be processed - I will send/add it to notes when I get it. J. J. On Fri, Nov 26, 2021 at 2:29 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > Reminder: the SIG meeting is today in ~2.5 hrs. > > Calendar link here: > https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com > Notes/material links will be added here > https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit?usp=sharing > > I will record the meeting and post the link together with the notes. > > On Thu, Nov 25, 2021 at 3:31 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > Just a reminder -> multi-tenancy meeting tomorrow. Few people worked > > on what will be presented tomorrow, and I am super excited we will be > > able to kick that one off - it has been a long time on my waiting list > > :) > > > > J. > > > > On Sat, Nov 20, 2021 at 10:14 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > The meeting is set for Friday 26th Nov 5 PM CET (4 PM UTC) > > > > > > This is the calendar link (google meet link there): > > > https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com > > > > > > The initial agenda: > > > > > > 1) The goal of the group, intro about the "isolation" and various "scopes" > > > of the multi-tenancy - Jarek Potiuk > > > > > > 2) The review of the example architecture that > > > needs the "multitenancy" - this is from the Google Composer team - > > > Mateusz Henc > > > > > > 3) Maybe others would like to get their case explain similarly > > > > > > 4) Discus proposals on the scope of the AIP(s) we want to write > > > and rough approach we can take for implementation and who will do > > > whatGoogle Meet call: meet.google.com/rxu-tvdz-vpv (edited) > > > > > > We will send more info/slides then. Anyone who would like to show/add > > > something, please respond here :). > > > > > > J.

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Ping Zhang <pi...@umich.edu>.
Ah, I guess I am a little bit late now.

In terms of the

> Isolating code execution and parsing of DAG files.
>

In Airbnb, we use `Docker runtime isolation for airflow tasks` plus
`Parsing Service` to totally isolate the dag parsing, task execution from
the airflow infra runtime.

`Docker runtime isolation for airflow tasks` (see email thread: [DISCUSS]
Docker runtime isolation for airflow tasks) introduces a docker layer which
wraps the dag parsing and task execution so each dag file can have its own
docker runtime.
`Parsing Service` totally removes the dag file parsing from the scheduler.

These two features have been running in Airbnb's production for close to 1
year. I am working on open source them.

Ping


Best wishes

Ping Zhang


On Wed, Dec 1, 2021 at 11:43 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Very good/important thoughts.
>
> From the discussions and looking at the (upcoming) proposals from
> Mateusz  we are going to have this all optional:
>
> We plan to have two config options:
>
> * DB Isolation mode for separating out DB access
> * Standalone DAG processor
>
> I totally agree that standalone/quick/dirty access mode for Airflow
> should be the default (so business as usual). Moreover - that will
> allow the introduction of the multi-tenant mode as "optional" in
> otherwise backwards-compatible Airflow - i.e. it could start to be
> available in 2.x line.
>
> Actually (and this is something up for discussion in the AIP) we could
> introduce "soft" multi-tenancy mode, where DB access will be still
> possible but flagged as a warning.
> This could give the user an option to switch gradually their DAGs to
> the multi-tenancy mode, if they are already using some direct DB
> access (for example in their callbacks or custom operator).
>
> Also I think part of the AIP and proof of concept while discussing it
> should be initially rough, and later more comprehensive performance
> testing of some "real-life" scenarios.
>
> J.
>
> On Wed, Dec 1, 2021 at 6:12 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > I look forward to seeing these propsals etc.
> >
> > One thought I've just had is that we should be careful about two things
> when taking on this work:
> >
> > 1. That performance is not impacted (specifically of the scheduler
> "throughput") -- at least when only a single "tenant" is in use if not for
> all.
> > 2. That we don't make the deployment story more complex for the small
> deployments, nor for the "getting started on a laptop" initial user
> experience.
> >
> > -ash
> >
> > On Fri, Nov 26 2021 at 18:23:32 +0100, Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >
> > Recording available here:
> https://drive.google.com/file/d/1Irw7qxxeTOHZTfdvT5lAbGowIfm9DHzi/view On
> Fri, Nov 26, 2021 at 6:17 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Thanks for the meeting this morning/afternoon :) ! It was very
> productive, I believe: The notes are available here:
> https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit
> The most important take is that it looks like if the use cases are slightly
> different, we are all aligned of what needs to be done and how Action
> points: * Composer team (Mateusz) will soon submit AIP's (they are close to
> be ready for proposing) for * DB access isolation * Separating out DAG
> processor * Cloudera team (Ian) will work on follow-up Fine-grained
> resource access AIP - it can be implemented as next steps. The two AIPs
> above will implement "coarse" access level but in the way that the
> "fine-grained" access will be possible to be plugged-in I recorded the
> meeting and I am waiting for the video to be processed - I will send/add it
> to notes when I get it. J. J. On Fri, Nov 26, 2021 at 2:29 PM Jarek Potiuk <
> jarek@potiuk.com> wrote: > > Reminder: the SIG meeting is today in ~2.5
> hrs. > > Calendar link here: >
> https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com
> > Notes/material links will be added here >
> https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit?usp=sharing
> > > I will record the meeting and post the link together with the notes. >
> > On Thu, Nov 25, 2021 at 3:31 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > > > > Just a reminder -> multi-tenancy meeting tomorrow. Few people
> worked > > on what will be presented tomorrow, and I am super excited we
> will be > > able to kick that one off - it has been a long time on my
> waiting list > > :) > > > > J. > > > > On Sat, Nov 20, 2021 at 10:14 AM
> Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > The meeting is set for
> Friday 26th Nov 5 PM CET (4 PM UTC) > > > > > > This is the calendar link
> (google meet link there): > > >
> https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com
> > > > > > > The initial agenda: > > > > > > 1) The goal of the group, intro
> about the "isolation" and various "scopes" > > > of the multi-tenancy -
> Jarek Potiuk > > > > > > 2) The review of the example architecture that > >
> > needs the "multitenancy" - this is from the Google Composer team - > > >
> Mateusz Henc > > > > > > 3) Maybe others would like to get their case
> explain similarly > > > > > > 4) Discus proposals on the scope of the
> AIP(s) we want to write > > > and rough approach we can take for
> implementation and who will do > > > whatGoogle Meet call:
> meet.google.com/rxu-tvdz-vpv (edited) > > > > > > We will send more
> info/slides then. Anyone who would like to show/add > > > something, please
> respond here :). > > > > > > J.
>

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
Very good/important thoughts.

From the discussions and looking at the (upcoming) proposals from
Mateusz  we are going to have this all optional:

We plan to have two config options:

* DB Isolation mode for separating out DB access
* Standalone DAG processor

I totally agree that standalone/quick/dirty access mode for Airflow
should be the default (so business as usual). Moreover - that will
allow the introduction of the multi-tenant mode as "optional" in
otherwise backwards-compatible Airflow - i.e. it could start to be
available in 2.x line.

Actually (and this is something up for discussion in the AIP) we could
introduce "soft" multi-tenancy mode, where DB access will be still
possible but flagged as a warning.
This could give the user an option to switch gradually their DAGs to
the multi-tenancy mode, if they are already using some direct DB
access (for example in their callbacks or custom operator).

Also I think part of the AIP and proof of concept while discussing it
should be initially rough, and later more comprehensive performance
testing of some "real-life" scenarios.

J.

On Wed, Dec 1, 2021 at 6:12 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> I look forward to seeing these propsals etc.
>
> One thought I've just had is that we should be careful about two things when taking on this work:
>
> 1. That performance is not impacted (specifically of the scheduler "throughput") -- at least when only a single "tenant" is in use if not for all.
> 2. That we don't make the deployment story more complex for the small deployments, nor for the "getting started on a laptop" initial user experience.
>
> -ash
>
> On Fri, Nov 26 2021 at 18:23:32 +0100, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Recording available here: https://drive.google.com/file/d/1Irw7qxxeTOHZTfdvT5lAbGowIfm9DHzi/view On Fri, Nov 26, 2021 at 6:17 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Thanks for the meeting this morning/afternoon :) ! It was very productive, I believe: The notes are available here: https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit The most important take is that it looks like if the use cases are slightly different, we are all aligned of what needs to be done and how Action points: * Composer team (Mateusz) will soon submit AIP's (they are close to be ready for proposing) for * DB access isolation * Separating out DAG processor * Cloudera team (Ian) will work on follow-up Fine-grained resource access AIP - it can be implemented as next steps. The two AIPs above will implement "coarse" access level but in the way that the "fine-grained" access will be possible to be plugged-in I recorded the meeting and I am waiting for the video to be processed - I will send/add it to notes when I get it. J. J. On Fri, Nov 26, 2021 at 2:29 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > Reminder: the SIG meeting is today in ~2.5 hrs. > > Calendar link here: > https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com > Notes/material links will be added here > https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit?usp=sharing > > I will record the meeting and post the link together with the notes. > > On Thu, Nov 25, 2021 at 3:31 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > Just a reminder -> multi-tenancy meeting tomorrow. Few people worked > > on what will be presented tomorrow, and I am super excited we will be > > able to kick that one off - it has been a long time on my waiting list > > :) > > > > J. > > > > On Sat, Nov 20, 2021 at 10:14 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > The meeting is set for Friday 26th Nov 5 PM CET (4 PM UTC) > > > > > > This is the calendar link (google meet link there): > > > https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com > > > > > > The initial agenda: > > > > > > 1) The goal of the group, intro about the "isolation" and various "scopes" > > > of the multi-tenancy - Jarek Potiuk > > > > > > 2) The review of the example architecture that > > > needs the "multitenancy" - this is from the Google Composer team - > > > Mateusz Henc > > > > > > 3) Maybe others would like to get their case explain similarly > > > > > > 4) Discus proposals on the scope of the AIP(s) we want to write > > > and rough approach we can take for implementation and who will do > > > whatGoogle Meet call: meet.google.com/rxu-tvdz-vpv (edited) > > > > > > We will send more info/slides then. Anyone who would like to show/add > > > something, please respond here :). > > > > > > J.

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Ash Berlin-Taylor <as...@apache.org>.
I look forward to seeing these propsals etc.

One thought I've just had is that we should be careful about two things 
when taking on this work:

1. That performance is not impacted (specifically of the scheduler 
"throughput") -- at least when only a single "tenant" is in use if not 
for all.
2. That we don't make the deployment story more complex for the small 
deployments, nor for the "getting started on a laptop" initial user 
experience.

-ash

On Fri, Nov 26 2021 at 18:23:32 +0100, Jarek Potiuk <ja...@potiuk.com> 
wrote:
> Recording available here:
> <https://drive.google.com/file/d/1Irw7qxxeTOHZTfdvT5lAbGowIfm9DHzi/view>
> 
> On Fri, Nov 26, 2021 at 6:17 PM Jarek Potiuk <jarek@potiuk.com 
> <ma...@potiuk.com>> wrote:
>> 
>>  Thanks for the meeting this morning/afternoon :) ! It was very
>>  productive, I believe:
>> 
>>  The notes are available here:
>>  
>> <https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit>
>> 
>>  The most important take is that it looks like if the use cases are
>>  slightly different, we are all aligned of what needs to be done and
>>  how
>> 
>>  Action points:
>> 
>>  * Composer team (Mateusz) will soon submit AIP's (they are close to 
>> be
>>  ready for proposing) for
>>     * DB access isolation
>>     * Separating out DAG processor
>>  * Cloudera team (Ian) will work on follow-up Fine-grained resource
>>  access AIP - it can be implemented as next steps. The two AIPs above
>>  will implement "coarse" access level but in the way that the
>>  "fine-grained" access will be possible to be plugged-in
>> 
>>  I recorded the meeting and I am waiting for the video to be 
>> processed
>>  - I will send/add it to notes when I get it.
>> 
>>  J.
>> 
>> 
>> 
>> 
>> 
>>  J.
>> 
>>  On Fri, Nov 26, 2021 at 2:29 PM Jarek Potiuk <jarek@potiuk.com 
>> <ma...@potiuk.com>> wrote:
>>  >
>>  > Reminder: the SIG meeting is today in ~2.5 hrs.
>>  >
>>  > Calendar link here:
>>  > 
>> <https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com>
>>  > Notes/material links will be added here
>>  > 
>> <https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit?usp=sharing>
>>  >
>>  > I will record the meeting and post the link together with the 
>> notes.
>>  >
>>  > On Thu, Nov 25, 2021 at 3:31 PM Jarek Potiuk <jarek@potiuk.com 
>> <ma...@potiuk.com>> wrote:
>>  > >
>>  > > Just a reminder -> multi-tenancy meeting tomorrow. Few people 
>> worked
>>  > > on what will be presented tomorrow, and I am super excited we 
>> will be
>>  > > able to kick that one off - it has been a long time on my 
>> waiting list
>>  > > :)
>>  > >
>>  > > J.
>>  > >
>>  > > On Sat, Nov 20, 2021 at 10:14 AM Jarek Potiuk <jarek@potiuk.com 
>> <ma...@potiuk.com>> wrote:
>>  > > >
>>  > > > The meeting is set for Friday 26th Nov 5 PM CET (4 PM UTC)
>>  > > >
>>  > > > This is the calendar link (google meet link there):
>>  > > > 
>> <https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com>
>>  > > >
>>  > > > The initial agenda:
>>  > > >
>>  > > > 1) The goal of the group, intro about the "isolation" and 
>> various "scopes"
>>  > > > of the multi-tenancy - Jarek Potiuk
>>  > > >
>>  > > > 2) The review of the example architecture that
>>  > > > needs the "multitenancy" - this is from the Google Composer 
>> team -
>>  > > > Mateusz Henc
>>  > > >
>>  > > > 3) Maybe others would like to get their case explain similarly
>>  > > >
>>  > > > 4) Discus proposals on the scope of the AIP(s) we want to 
>> write
>>  > > > and rough approach we can take for implementation and who 
>> will do
>>  > > > whatGoogle Meet call: meet.google.com/rxu-tvdz-vpv (edited)
>>  > > >
>>  > > > We will send more info/slides then. Anyone who would like to 
>> show/add
>>  > > > something, please respond here :).
>>  > > >
>>  > > > J.

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
Recording available here:
https://drive.google.com/file/d/1Irw7qxxeTOHZTfdvT5lAbGowIfm9DHzi/view

On Fri, Nov 26, 2021 at 6:17 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Thanks for the meeting this morning/afternoon :) ! It was very
> productive, I believe:
>
> The notes are available here:
> https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit
>
> The most important take is that it looks like if the use cases are
> slightly different, we are all aligned of what needs to be done and
> how
>
> Action points:
>
> * Composer team (Mateusz) will soon submit AIP's (they are close to be
> ready for proposing) for
>    * DB access isolation
>    * Separating out DAG processor
> * Cloudera team (Ian) will work on follow-up Fine-grained resource
> access AIP - it can be implemented as next steps. The two AIPs above
> will implement "coarse" access level but in the way that the
> "fine-grained" access will be possible to be plugged-in
>
> I recorded the meeting and I am waiting for the video to be processed
> - I will send/add it to notes when I get it.
>
> J.
>
>
>
>
>
> J.
>
> On Fri, Nov 26, 2021 at 2:29 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Reminder: the SIG meeting is today in ~2.5 hrs.
> >
> > Calendar link here:
> > https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com
> > Notes/material links will be added here
> > https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit?usp=sharing
> >
> > I will record the meeting and post the link together with the notes.
> >
> > On Thu, Nov 25, 2021 at 3:31 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > Just a reminder -> multi-tenancy meeting tomorrow. Few people worked
> > > on what will be presented tomorrow, and I am super excited we will be
> > > able to kick that one off - it has been a long time on my waiting list
> > > :)
> > >
> > > J.
> > >
> > > On Sat, Nov 20, 2021 at 10:14 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > > The meeting is set for Friday 26th Nov 5 PM CET (4 PM UTC)
> > > >
> > > > This is the calendar link (google meet link there):
> > > > https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com
> > > >
> > > > The initial agenda:
> > > >
> > > > 1) The goal of the group, intro about the "isolation" and various "scopes"
> > > > of the multi-tenancy - Jarek Potiuk
> > > >
> > > > 2) The review of the example architecture that
> > > > needs the "multitenancy" - this is from the Google Composer team -
> > > > Mateusz Henc
> > > >
> > > > 3) Maybe others would like to get their case explain similarly
> > > >
> > > > 4) Discus proposals on the scope of the AIP(s) we want to write
> > > > and rough approach we can take for implementation and who will do
> > > > whatGoogle Meet call: meet.google.com/rxu-tvdz-vpv (edited)
> > > >
> > > > We will send more info/slides then. Anyone who would like to show/add
> > > > something, please respond here :).
> > > >
> > > > J.

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
Thanks for the meeting this morning/afternoon :) ! It was very
productive, I believe:

The notes are available here:
https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit

The most important take is that it looks like if the use cases are
slightly different, we are all aligned of what needs to be done and
how

Action points:

* Composer team (Mateusz) will soon submit AIP's (they are close to be
ready for proposing) for
   * DB access isolation
   * Separating out DAG processor
* Cloudera team (Ian) will work on follow-up Fine-grained resource
access AIP - it can be implemented as next steps. The two AIPs above
will implement "coarse" access level but in the way that the
"fine-grained" access will be possible to be plugged-in

I recorded the meeting and I am waiting for the video to be processed
- I will send/add it to notes when I get it.

J.





J.

On Fri, Nov 26, 2021 at 2:29 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Reminder: the SIG meeting is today in ~2.5 hrs.
>
> Calendar link here:
> https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com
> Notes/material links will be added here
> https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit?usp=sharing
>
> I will record the meeting and post the link together with the notes.
>
> On Thu, Nov 25, 2021 at 3:31 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Just a reminder -> multi-tenancy meeting tomorrow. Few people worked
> > on what will be presented tomorrow, and I am super excited we will be
> > able to kick that one off - it has been a long time on my waiting list
> > :)
> >
> > J.
> >
> > On Sat, Nov 20, 2021 at 10:14 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > The meeting is set for Friday 26th Nov 5 PM CET (4 PM UTC)
> > >
> > > This is the calendar link (google meet link there):
> > > https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com
> > >
> > > The initial agenda:
> > >
> > > 1) The goal of the group, intro about the "isolation" and various "scopes"
> > > of the multi-tenancy - Jarek Potiuk
> > >
> > > 2) The review of the example architecture that
> > > needs the "multitenancy" - this is from the Google Composer team -
> > > Mateusz Henc
> > >
> > > 3) Maybe others would like to get their case explain similarly
> > >
> > > 4) Discus proposals on the scope of the AIP(s) we want to write
> > > and rough approach we can take for implementation and who will do
> > > whatGoogle Meet call: meet.google.com/rxu-tvdz-vpv (edited)
> > >
> > > We will send more info/slides then. Anyone who would like to show/add
> > > something, please respond here :).
> > >
> > > J.

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
Reminder: the SIG meeting is today in ~2.5 hrs.

Calendar link here:
https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com
Notes/material links will be added here
https://docs.google.com/document/d/19d0jQeARnWm8VTVc_qSIEDldQ81acRk0KuhVwAd96wo/edit?usp=sharing

I will record the meeting and post the link together with the notes.

On Thu, Nov 25, 2021 at 3:31 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Just a reminder -> multi-tenancy meeting tomorrow. Few people worked
> on what will be presented tomorrow, and I am super excited we will be
> able to kick that one off - it has been a long time on my waiting list
> :)
>
> J.
>
> On Sat, Nov 20, 2021 at 10:14 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > The meeting is set for Friday 26th Nov 5 PM CET (4 PM UTC)
> >
> > This is the calendar link (google meet link there):
> > https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com
> >
> > The initial agenda:
> >
> > 1) The goal of the group, intro about the "isolation" and various "scopes"
> > of the multi-tenancy - Jarek Potiuk
> >
> > 2) The review of the example architecture that
> > needs the "multitenancy" - this is from the Google Composer team -
> > Mateusz Henc
> >
> > 3) Maybe others would like to get their case explain similarly
> >
> > 4) Discus proposals on the scope of the AIP(s) we want to write
> > and rough approach we can take for implementation and who will do
> > whatGoogle Meet call: meet.google.com/rxu-tvdz-vpv (edited)
> >
> > We will send more info/slides then. Anyone who would like to show/add
> > something, please respond here :).
> >
> > J.

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
Just a reminder -> multi-tenancy meeting tomorrow. Few people worked
on what will be presented tomorrow, and I am super excited we will be
able to kick that one off - it has been a long time on my waiting list
:)

J.

On Sat, Nov 20, 2021 at 10:14 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> The meeting is set for Friday 26th Nov 5 PM CET (4 PM UTC)
>
> This is the calendar link (google meet link there):
> https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com
>
> The initial agenda:
>
> 1) The goal of the group, intro about the "isolation" and various "scopes"
> of the multi-tenancy - Jarek Potiuk
>
> 2) The review of the example architecture that
> needs the "multitenancy" - this is from the Google Composer team -
> Mateusz Henc
>
> 3) Maybe others would like to get their case explain similarly
>
> 4) Discus proposals on the scope of the AIP(s) we want to write
> and rough approach we can take for implementation and who will do
> whatGoogle Meet call: meet.google.com/rxu-tvdz-vpv (edited)
>
> We will send more info/slides then. Anyone who would like to show/add
> something, please respond here :).
>
> J.

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
The meeting is set for Friday 26th Nov 5 PM CET (4 PM UTC)

This is the calendar link (google meet link there):
https://calendar.google.com/event?action=TEMPLATE&tmeid=N3ZmbGFxNGF1OXBtajc2ODU3bWduMWVvc2YgcG90aXVrLmFwYWNoZS5vcmdAbQ&tmsrc=potiuk.apache.org%40gmail.com

The initial agenda:

1) The goal of the group, intro about the "isolation" and various "scopes"
of the multi-tenancy - Jarek Potiuk

2) The review of the example architecture that
needs the "multitenancy" - this is from the Google Composer team -
Mateusz Henc

3) Maybe others would like to get their case explain similarly

4) Discus proposals on the scope of the AIP(s) we want to write
and rough approach we can take for implementation and who will do
whatGoogle Meet call: meet.google.com/rxu-tvdz-vpv (edited)

We will send more info/slides then. Anyone who would like to show/add
something, please respond here :).

J.

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello Everyone,

I would like to call for a meeting where we can talk about the
multi-tenancy AIP (AIPs?)

I prepared a doodle to find a good time next week
https://doodle.com/poll/nrfaqbmcwfqqhbsw?utm_source=poll&utm_medium=link

The initial agenda I plan for now:

1) I would like to start the kinds of "isolation" and various "scopes"
of multi-tenancy
2) We will have a "draft" review of the example architecture that
needs the  "multitenancy" - this is from the Google Composer team -
Mateusz Henc will walk through the composer team goals. That will give
us a well thought and discussed "use" of the multitenancy features.
3) Possibly others (Ian? anyone else?) would like to get their case
laid out similarly (maybe just focusing on what's different - because
after studying earlier discussions I think we all have similar ideas
on what needs to be done).
4) Discussing proposals on the scope of the AIP(s) we want to write
and rough approach we can take for implementation and initial
commitment on who will do what

Please mark your good time in the doodle, or if the time is not great
and you would like to propose other times - let us know here.

J.


On Mon, Nov 15, 2021 at 11:11 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Cool. I will propose next steps tomorrow most likely.
>
> On Sat, Nov 13, 2021 at 12:37 PM Ian Buss <ia...@gmail.com> wrote:
> >
> > Thanks Jarek! Will join the channel. This is still something close to my heart and an area where I think we could make great progress. I agree that we should get together and start a draft AIP around this to gather the various strands of this together.
> > On 7 Nov 2021, 13:19 +0000, Jarek Potiuk <ja...@potiuk.com>, wrote:
> >
> > For now I created a channel for the SIG:
> > https://apache-airflow.slack.com/archives/C02M551UDA4 - feel free to
> > join anyone and in the next weeks once all the people involved so far
> > expressed their interest, we should set some plan on getting the
> > AIP(s)? drafted/discussed and start implementing it.
> >
> > On Sat, Nov 6, 2021 at 11:22 PM Xinbin Huang <bi...@gmail.com> wrote:
> >
> >
> > Hi Jarek,
> >
> > The plan sounds great! And +1 to a special interest group. Please add me to the group if you do create one.
> >
> > Here is the doc ( Airflow Multi-tenancy discussion ) we used to discuss back in April. It's not a note per-se, but I think it can shed some light on what we talked about. Other folks may have an actual note or even a draft proposal on this topic.
> >
> > I'm excited for us to move forward with this.
> >
> > Bin
> >
> > On Fri, Nov 5, 2021 at 10:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >
> > Hello Ian, Everyone,
> >
> > I wonder if there are any notes from the meeting in April? Has there
> > been any more work on that one from Cloudera to formalize and plan
> > work on it?
> >
> > I was not able to participate, but I think it's about the time to
> > seriously start work on that and I am super happy to take more lead on
> > this project and involve all the interested parties. The ideas
> > described in the email and discussed after are I think super
> > reasonable and definitely necessary to get to the multi-tenancy and I
> > believe that there are already ideas that can be turned into reality
> > rather soon. I had a talk today also with the Google Composer team and
> > they are also fully on board with dedicating a lot of effort on this
> > one (and their ideas are I think super-aligned with Cloudera's), so I
> > think we have a critical mass and engineering power to make it happen
> > :)
> >
> > I plan to put quite a lot of focus on that one over the coming months
> > and I am happy to lead or co-lead the AIP and take a big part in
> > implementation.
> >
> > Possibly we should create a special interest group around that and
> > start drafting the AIP proposals in a smaller group of people who are
> > interested and start planning the work. I already have some ideas
> > where we could start gradually implementing it (of course after we
> > prepare the AIP and get it through the community's approval process).
> >
> > How does it sound?
> >
> > J.
> >
> > On Wed, Apr 21, 2021 at 8:56 AM Ian Buss <ia...@gmail.com> wrote:
> >
> >
> > Yes, no invite required. See you tomorrow!
> > On 21 Apr 2021, 07:46 +0100, Sumit Maheshwari <ms...@apache.org>, wrote:
> >
> > I'll join as well (I believe the zoom link will work without an invite)
> >
> > On Wed, Apr 21, 2021 at 10:48 AM Dimitris Stafylarakis <xa...@gmail.com> wrote:
> >
> >
> > hi all,
> >
> > great to read about this, I'd like to join in! Can I just join using the zoom link tomorrow or do I need an invitation? (If I do need one, please invite me :))
> >
> > cheers
> >
> >
> > On Wed, Apr 14, 2021 at 8:15 PM Daniel Imberman <da...@gmail.com> wrote:
> >
> >
> > Thank you Ian,
> >
> > I’ve invited everyone on this thread to the meeting with that zoom link. Anyone else who wants to join can add the calendar event here calendar.google.com/event?action=TEMPLATE&tmeid=Mm4zN2Q3MnFwNnBqbW9hMmNocXMyNzJpdHYgZGFuaWVsQGFzdHJvbm9tZXIuaW8&tmsrc=daniel@astronomer.io
> >
> > On Wed, Apr 14, 2021 at 11:05 AM, Ian Buss <ia...@gmail.com> wrote:
> >
> > If this works for everyone, here's a zoom link for Thursday 8AM PST: https://cloudera.zoom.us/j/99928254235?pwd=VTFlQk4vQjQ5Z2JzUDM3ZWZKKy9MQT09
> >
> > Happy to move or use an alternate method as needed.
> >
> > On Wed, Apr 14, 2021 at 6:58 PM Daniel Imberman <da...@gmail.com> wrote:
> >
> >
> > Thursday works for me!
> >
> > On Wed, Apr 14, 2021 at 10:05 AM, Ian Buss <ia...@gmail.com> wrote:
> >
> > Hi all,
> >
> > I actually can’t do Wednesday next week as I’m moving house :) Any chance we could do Thursday or Friday at the same time?
> >
> > Cheers
> >
> > Ian
> > On 14 Apr 2021, 17:49 +0100, Kaxil Naik <ka...@gmail.com>, wrote:
> >
> > Just few comments here:
> >
> > Currently -- atleast for the foreseeable future Airflow workers will need access to the DAG Files, so workers can not run using the Serialized DAGs.
> >
> > Also serialized DAGs do not even have all the info needed for it to run it. Currently the serialization happens in the parsing process in the scheduler which can be in future separated as a separator "parsining" component, but that won't solve the "isolation" problem you are trying to solve. The only current way it can be solved is pickling -- and we have strictly decided against using pickling for DAGs.
> >
> > The idea in Statement (2) & (3) would help solve the isolation problem in (1) and can be done with some work now.
> >
> > Happy to talk about it in more detail here or on call, the time Daniel suggested works for me.
> >
> > Regards,
> > Kaxil
> >
> > On Wed, Apr 14, 2021 at 5:35 PM Daniel Imberman <da...@gmail.com> wrote:
> >
> >
> > How about Wednesday, April 21 at 8:00AM PST?
> >
> > On Wed, Apr 14, 2021 at 9:33 AM, Xinbin Huang <bi...@gmail.com> wrote:
> >
> > I am available any days.
> >
> > On Wed, Apr 14, 2021, 9:32 AM Daniel Imberman <da...@gmail.com> wrote:
> >
> >
> > Hi everyone!
> >
> > Would people be available around 8AM/9AM PST some point next week? I’m in PST and Ian is UTC+1 so would be great to find a timezone that accomodates everyone.
> >
> > Daniel
> > On Wed, Apr 14, 2021 at 6:26 AM, Ryan Hatter <ry...@gmail.com> wrote:
> >
> > I’d also like to be added please :)
> >
> > On Apr 13, 2021, at 21:27, Xinbin Huang <bi...@gmail.com> wrote:
> >
> > 
> > Hi Daniel & Ian,
> >
> > I am also interested in the idea of a serialization representation that can be executed by workers directly. Can you also add me to the call?
> >
> > Thanks
> > Bin
> >
> > On Tue, Apr 13, 2021 at 2:49 PM Ian Buss <ia...@gmail.com> wrote:
> >
> >
> > Daniel,
> >
> > Thanks for your warm welcome and quick response and the advice on providers! Will certainly check out the examples you sent.
> >
> > 1. An "airflow register" command definitely sounds promising, would love to collaborate on an AIP there so let's set something up.
> > 2. We use KubernetesExecutor exclusively as well. We've noticed significant additional load on the metadata DB as we scale up task pods so I've also thought about an API-based approach. Such an API could also open up the possibility of per-task security tokens which are injected by the scheduler, which should improve the security of such a system. Food for thought at least. I will start putting some of these thoughts down on paper in a sharable format.
> >
> > Ian
> >
> > On Tue, Apr 13, 2021 at 7:46 PM Daniel Imberman <da...@gmail.com> wrote:
> >
> >
> > Hi Ian,
> >
> >
> > Firstly, welcome to the Airflow community :). I'm glad to hear you've had a positive experience so far. It's great to hear that you want to contribute back, and I think that multi-tenancy/DAG isolation is a pretty fantastic project for the community as a whole (a lot of things are are things we want but are limited by hours in a day).
> >
> >
> > 1. I've personally been kicking around some ideas lately about an "airflow register" command that would write the DAG into the metadata DB in a way that could be "gettable" by the workers via the API. This work is very early. I'd love to get some help on it. Perhaps we can set up a zoom chat to discuss drafting an AIP?
> >
> >
> > 2. Limiting worker access to the DB is not only good security practice; it also opens up the door to a lot of valuable features. This feature would be especially close to my heart as it would make the KubernetesExecutor significantly more efficient. It should be possible to set up a system where the workers only ever speak to an API server and never need to touch the DB.
> >
> >
> > 3. This is not something I personally have insight into, but I think it sounds like a good idea.
> >
> >
> > Finally, addressing your question about a Cloudera provider. If anything, it would probably give the provider _more_ legitimacy if you hosted it under the Cloudera GitHub org (we very purposely created the provider packages with this workflow in mind). There are multiple places where we can work to surface this provider so it is easy to find and use.
> >
> >
> > Astronomer has a pretty good sample provider here. One example of it running in the wild is the Great Expectations provider here. I'd also be glad to get you in contact with people who have built providers in the past to help you with that process.
> >
> >
> > Looking forward to seeing some of these things come to fruition!
> >
> >
> > Daniel
> >
> >
> > On Tue, Apr 13, 2021 at 9:43 AM, Ian Buss <ia...@gmail.com> wrote:
> >
> > Hi all,
> >
> > First a quick introduction: I'm an engineer with Cloudera working on our Data Engineering product (CDE). Airflow is working great for us so far. We've been looking into how we can enhance the multi-tenancy story of Apache Airflow as we currently deploy it. We have the following areas which we'd like (with community consensus) to work on and contribute back to Apache Airflow to enhance the isolation between tenants in a single Airflow deployment.
> >
> > 1. Isolating code execution and parsing of DAG files. At the moment, DAG files are parsed in a few locations in Airflow, including the scheduler and in tasks. There is already the concept of DAG serialization (and we're using that for the web component) but we'd be interested to see if we can sandbox the execution of arbitrary user code to a locked down process/container without full access to the metadata DB and connection secrets etc. The idea would be to parse and serialize the DAG in this isolated container and pass back a serialized representation for persistence in the DB. Has anyone explored this idea?
> >
> > 2. Limiting task access to the metadata DB. It would be great if we could remove the requirement for tasks to have full access to the metadata DB and to report task status in a different (but still scalable) way. We'd need to tackle access or injection of connection, variable and xcom data as well for each task naturally.
> >
> > 3. Finer-grained access controls on connection secrets. Right now, although there are nice at-rest encryption options with Fernet or Vault, IIUC any DAG can access any connection (and thus any secret). Since the "run as" user is largely defined within the DAG and its tasks, this is challenging for a multi-tenant environment (see caveat below)
> >
> > Caveat: It's definitely noted that to some extent we should assume that an Airflow deployment is a "trusted" environment and that best practices such as git+PR workflows are the gold standard and that any malicious code and dependencies should be identified through this process. Also that there is a clear admin role for connection management etc.
> >
> > We have some ideas informally sketched out as to how to address the above but would be keen to hear the community opinion on this and to see if anyone is keen to collaborate on designs and implementation, or to hear if anything is already in the works. In particular I noticed that the very first improvement proposal (AIP-1) addresses much of the above :). However, it seems fairly dormant at the moment.
> >
> > One other question: we have a provider (operators and hooks) for interacting with Cloudera components that we'd like to contribute to the project. The provider FAQs indicate that new provider contributions are still welcome in the project in 2.x, is that accurate?
> >
> > Thanks in advance!
> >
> > Ian

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
Cool. I will propose next steps tomorrow most likely.

On Sat, Nov 13, 2021 at 12:37 PM Ian Buss <ia...@gmail.com> wrote:
>
> Thanks Jarek! Will join the channel. This is still something close to my heart and an area where I think we could make great progress. I agree that we should get together and start a draft AIP around this to gather the various strands of this together.
> On 7 Nov 2021, 13:19 +0000, Jarek Potiuk <ja...@potiuk.com>, wrote:
>
> For now I created a channel for the SIG:
> https://apache-airflow.slack.com/archives/C02M551UDA4 - feel free to
> join anyone and in the next weeks once all the people involved so far
> expressed their interest, we should set some plan on getting the
> AIP(s)? drafted/discussed and start implementing it.
>
> On Sat, Nov 6, 2021 at 11:22 PM Xinbin Huang <bi...@gmail.com> wrote:
>
>
> Hi Jarek,
>
> The plan sounds great! And +1 to a special interest group. Please add me to the group if you do create one.
>
> Here is the doc ( Airflow Multi-tenancy discussion ) we used to discuss back in April. It's not a note per-se, but I think it can shed some light on what we talked about. Other folks may have an actual note or even a draft proposal on this topic.
>
> I'm excited for us to move forward with this.
>
> Bin
>
> On Fri, Nov 5, 2021 at 10:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>
> Hello Ian, Everyone,
>
> I wonder if there are any notes from the meeting in April? Has there
> been any more work on that one from Cloudera to formalize and plan
> work on it?
>
> I was not able to participate, but I think it's about the time to
> seriously start work on that and I am super happy to take more lead on
> this project and involve all the interested parties. The ideas
> described in the email and discussed after are I think super
> reasonable and definitely necessary to get to the multi-tenancy and I
> believe that there are already ideas that can be turned into reality
> rather soon. I had a talk today also with the Google Composer team and
> they are also fully on board with dedicating a lot of effort on this
> one (and their ideas are I think super-aligned with Cloudera's), so I
> think we have a critical mass and engineering power to make it happen
> :)
>
> I plan to put quite a lot of focus on that one over the coming months
> and I am happy to lead or co-lead the AIP and take a big part in
> implementation.
>
> Possibly we should create a special interest group around that and
> start drafting the AIP proposals in a smaller group of people who are
> interested and start planning the work. I already have some ideas
> where we could start gradually implementing it (of course after we
> prepare the AIP and get it through the community's approval process).
>
> How does it sound?
>
> J.
>
> On Wed, Apr 21, 2021 at 8:56 AM Ian Buss <ia...@gmail.com> wrote:
>
>
> Yes, no invite required. See you tomorrow!
> On 21 Apr 2021, 07:46 +0100, Sumit Maheshwari <ms...@apache.org>, wrote:
>
> I'll join as well (I believe the zoom link will work without an invite)
>
> On Wed, Apr 21, 2021 at 10:48 AM Dimitris Stafylarakis <xa...@gmail.com> wrote:
>
>
> hi all,
>
> great to read about this, I'd like to join in! Can I just join using the zoom link tomorrow or do I need an invitation? (If I do need one, please invite me :))
>
> cheers
>
>
> On Wed, Apr 14, 2021 at 8:15 PM Daniel Imberman <da...@gmail.com> wrote:
>
>
> Thank you Ian,
>
> I’ve invited everyone on this thread to the meeting with that zoom link. Anyone else who wants to join can add the calendar event here calendar.google.com/event?action=TEMPLATE&tmeid=Mm4zN2Q3MnFwNnBqbW9hMmNocXMyNzJpdHYgZGFuaWVsQGFzdHJvbm9tZXIuaW8&tmsrc=daniel@astronomer.io
>
> On Wed, Apr 14, 2021 at 11:05 AM, Ian Buss <ia...@gmail.com> wrote:
>
> If this works for everyone, here's a zoom link for Thursday 8AM PST: https://cloudera.zoom.us/j/99928254235?pwd=VTFlQk4vQjQ5Z2JzUDM3ZWZKKy9MQT09
>
> Happy to move or use an alternate method as needed.
>
> On Wed, Apr 14, 2021 at 6:58 PM Daniel Imberman <da...@gmail.com> wrote:
>
>
> Thursday works for me!
>
> On Wed, Apr 14, 2021 at 10:05 AM, Ian Buss <ia...@gmail.com> wrote:
>
> Hi all,
>
> I actually can’t do Wednesday next week as I’m moving house :) Any chance we could do Thursday or Friday at the same time?
>
> Cheers
>
> Ian
> On 14 Apr 2021, 17:49 +0100, Kaxil Naik <ka...@gmail.com>, wrote:
>
> Just few comments here:
>
> Currently -- atleast for the foreseeable future Airflow workers will need access to the DAG Files, so workers can not run using the Serialized DAGs.
>
> Also serialized DAGs do not even have all the info needed for it to run it. Currently the serialization happens in the parsing process in the scheduler which can be in future separated as a separator "parsining" component, but that won't solve the "isolation" problem you are trying to solve. The only current way it can be solved is pickling -- and we have strictly decided against using pickling for DAGs.
>
> The idea in Statement (2) & (3) would help solve the isolation problem in (1) and can be done with some work now.
>
> Happy to talk about it in more detail here or on call, the time Daniel suggested works for me.
>
> Regards,
> Kaxil
>
> On Wed, Apr 14, 2021 at 5:35 PM Daniel Imberman <da...@gmail.com> wrote:
>
>
> How about Wednesday, April 21 at 8:00AM PST?
>
> On Wed, Apr 14, 2021 at 9:33 AM, Xinbin Huang <bi...@gmail.com> wrote:
>
> I am available any days.
>
> On Wed, Apr 14, 2021, 9:32 AM Daniel Imberman <da...@gmail.com> wrote:
>
>
> Hi everyone!
>
> Would people be available around 8AM/9AM PST some point next week? I’m in PST and Ian is UTC+1 so would be great to find a timezone that accomodates everyone.
>
> Daniel
> On Wed, Apr 14, 2021 at 6:26 AM, Ryan Hatter <ry...@gmail.com> wrote:
>
> I’d also like to be added please :)
>
> On Apr 13, 2021, at 21:27, Xinbin Huang <bi...@gmail.com> wrote:
>
> 
> Hi Daniel & Ian,
>
> I am also interested in the idea of a serialization representation that can be executed by workers directly. Can you also add me to the call?
>
> Thanks
> Bin
>
> On Tue, Apr 13, 2021 at 2:49 PM Ian Buss <ia...@gmail.com> wrote:
>
>
> Daniel,
>
> Thanks for your warm welcome and quick response and the advice on providers! Will certainly check out the examples you sent.
>
> 1. An "airflow register" command definitely sounds promising, would love to collaborate on an AIP there so let's set something up.
> 2. We use KubernetesExecutor exclusively as well. We've noticed significant additional load on the metadata DB as we scale up task pods so I've also thought about an API-based approach. Such an API could also open up the possibility of per-task security tokens which are injected by the scheduler, which should improve the security of such a system. Food for thought at least. I will start putting some of these thoughts down on paper in a sharable format.
>
> Ian
>
> On Tue, Apr 13, 2021 at 7:46 PM Daniel Imberman <da...@gmail.com> wrote:
>
>
> Hi Ian,
>
>
> Firstly, welcome to the Airflow community :). I'm glad to hear you've had a positive experience so far. It's great to hear that you want to contribute back, and I think that multi-tenancy/DAG isolation is a pretty fantastic project for the community as a whole (a lot of things are are things we want but are limited by hours in a day).
>
>
> 1. I've personally been kicking around some ideas lately about an "airflow register" command that would write the DAG into the metadata DB in a way that could be "gettable" by the workers via the API. This work is very early. I'd love to get some help on it. Perhaps we can set up a zoom chat to discuss drafting an AIP?
>
>
> 2. Limiting worker access to the DB is not only good security practice; it also opens up the door to a lot of valuable features. This feature would be especially close to my heart as it would make the KubernetesExecutor significantly more efficient. It should be possible to set up a system where the workers only ever speak to an API server and never need to touch the DB.
>
>
> 3. This is not something I personally have insight into, but I think it sounds like a good idea.
>
>
> Finally, addressing your question about a Cloudera provider. If anything, it would probably give the provider _more_ legitimacy if you hosted it under the Cloudera GitHub org (we very purposely created the provider packages with this workflow in mind). There are multiple places where we can work to surface this provider so it is easy to find and use.
>
>
> Astronomer has a pretty good sample provider here. One example of it running in the wild is the Great Expectations provider here. I'd also be glad to get you in contact with people who have built providers in the past to help you with that process.
>
>
> Looking forward to seeing some of these things come to fruition!
>
>
> Daniel
>
>
> On Tue, Apr 13, 2021 at 9:43 AM, Ian Buss <ia...@gmail.com> wrote:
>
> Hi all,
>
> First a quick introduction: I'm an engineer with Cloudera working on our Data Engineering product (CDE). Airflow is working great for us so far. We've been looking into how we can enhance the multi-tenancy story of Apache Airflow as we currently deploy it. We have the following areas which we'd like (with community consensus) to work on and contribute back to Apache Airflow to enhance the isolation between tenants in a single Airflow deployment.
>
> 1. Isolating code execution and parsing of DAG files. At the moment, DAG files are parsed in a few locations in Airflow, including the scheduler and in tasks. There is already the concept of DAG serialization (and we're using that for the web component) but we'd be interested to see if we can sandbox the execution of arbitrary user code to a locked down process/container without full access to the metadata DB and connection secrets etc. The idea would be to parse and serialize the DAG in this isolated container and pass back a serialized representation for persistence in the DB. Has anyone explored this idea?
>
> 2. Limiting task access to the metadata DB. It would be great if we could remove the requirement for tasks to have full access to the metadata DB and to report task status in a different (but still scalable) way. We'd need to tackle access or injection of connection, variable and xcom data as well for each task naturally.
>
> 3. Finer-grained access controls on connection secrets. Right now, although there are nice at-rest encryption options with Fernet or Vault, IIUC any DAG can access any connection (and thus any secret). Since the "run as" user is largely defined within the DAG and its tasks, this is challenging for a multi-tenant environment (see caveat below)
>
> Caveat: It's definitely noted that to some extent we should assume that an Airflow deployment is a "trusted" environment and that best practices such as git+PR workflows are the gold standard and that any malicious code and dependencies should be identified through this process. Also that there is a clear admin role for connection management etc.
>
> We have some ideas informally sketched out as to how to address the above but would be keen to hear the community opinion on this and to see if anyone is keen to collaborate on designs and implementation, or to hear if anything is already in the works. In particular I noticed that the very first improvement proposal (AIP-1) addresses much of the above :). However, it seems fairly dormant at the moment.
>
> One other question: we have a provider (operators and hooks) for interacting with Cloudera components that we'd like to contribute to the project. The provider FAQs indicate that new provider contributions are still welcome in the project in 2.x, is that accurate?
>
> Thanks in advance!
>
> Ian

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Ian Buss <ia...@gmail.com>.
Thanks Jarek! Will join the channel. This is still something close to my heart and an area where I think we could make great progress. I agree that we should get together and start a draft AIP around this to gather the various strands of this together.
On 7 Nov 2021, 13:19 +0000, Jarek Potiuk <ja...@potiuk.com>, wrote:
> For now I created a channel for the SIG:
> https://apache-airflow.slack.com/archives/C02M551UDA4 - feel free to
> join anyone and in the next weeks once all the people involved so far
> expressed their interest, we should set some plan on getting the
> AIP(s)? drafted/discussed and start implementing it.
>
> On Sat, Nov 6, 2021 at 11:22 PM Xinbin Huang <bi...@gmail.com> wrote:
> >
> > Hi Jarek,
> >
> > The plan sounds great! And +1 to a special interest group. Please add me to the group if you do create one.
> >
> > Here is the doc ( Airflow Multi-tenancy discussion ) we used to discuss back in April. It's not a note per-se, but I think it can shed some light on what we talked about. Other folks may have an actual note or even a draft proposal on this topic.
> >
> > I'm excited for us to move forward with this.
> >
> > Bin
> >
> > On Fri, Nov 5, 2021 at 10:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > Hello Ian, Everyone,
> > >
> > > I wonder if there are any notes from the meeting in April? Has there
> > > been any more work on that one from Cloudera to formalize and plan
> > > work on it?
> > >
> > > I was not able to participate, but I think it's about the time to
> > > seriously start work on that and I am super happy to take more lead on
> > > this project and involve all the interested parties. The ideas
> > > described in the email and discussed after are I think super
> > > reasonable and definitely necessary to get to the multi-tenancy and I
> > > believe that there are already ideas that can be turned into reality
> > > rather soon. I had a talk today also with the Google Composer team and
> > > they are also fully on board with dedicating a lot of effort on this
> > > one (and their ideas are I think super-aligned with Cloudera's), so I
> > > think we have a critical mass and engineering power to make it happen
> > > :)
> > >
> > > I plan to put quite a lot of focus on that one over the coming months
> > > and I am happy to lead or co-lead the AIP and take a big part in
> > > implementation.
> > >
> > > Possibly we should create a special interest group around that and
> > > start drafting the AIP proposals in a smaller group of people who are
> > > interested and start planning the work. I already have some ideas
> > > where we could start gradually implementing it (of course after we
> > > prepare the AIP and get it through the community's approval process).
> > >
> > > How does it sound?
> > >
> > > J.
> > >
> > > On Wed, Apr 21, 2021 at 8:56 AM Ian Buss <ia...@gmail.com> wrote:
> > > >
> > > > Yes, no invite required. See you tomorrow!
> > > > On 21 Apr 2021, 07:46 +0100, Sumit Maheshwari <ms...@apache.org>, wrote:
> > > >
> > > > I'll join as well (I believe the zoom link will work without an invite)
> > > >
> > > > On Wed, Apr 21, 2021 at 10:48 AM Dimitris Stafylarakis <xa...@gmail.com> wrote:
> > > > >
> > > > > hi all,
> > > > >
> > > > > great to read about this, I'd like to join in! Can I just join using the zoom link tomorrow or do I need an invitation? (If I do need one, please invite me :))
> > > > >
> > > > > cheers
> > > > >
> > > > >
> > > > > On Wed, Apr 14, 2021 at 8:15 PM Daniel Imberman <da...@gmail.com> wrote:
> > > > > >
> > > > > > Thank you Ian,
> > > > > >
> > > > > > I’ve invited everyone on this thread to the meeting with that zoom link. Anyone else who wants to join can add the calendar event here calendar.google.com/event?action=TEMPLATE&tmeid=Mm4zN2Q3MnFwNnBqbW9hMmNocXMyNzJpdHYgZGFuaWVsQGFzdHJvbm9tZXIuaW8&tmsrc=daniel@astronomer.io
> > > > > >
> > > > > > On Wed, Apr 14, 2021 at 11:05 AM, Ian Buss <ia...@gmail.com> wrote:
> > > > > >
> > > > > > If this works for everyone, here's a zoom link for Thursday 8AM PST: https://cloudera.zoom.us/j/99928254235?pwd=VTFlQk4vQjQ5Z2JzUDM3ZWZKKy9MQT09
> > > > > >
> > > > > > Happy to move or use an alternate method as needed.
> > > > > >
> > > > > > On Wed, Apr 14, 2021 at 6:58 PM Daniel Imberman <da...@gmail.com> wrote:
> > > > > > >
> > > > > > > Thursday works for me!
> > > > > > >
> > > > > > > On Wed, Apr 14, 2021 at 10:05 AM, Ian Buss <ia...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I actually can’t do Wednesday next week as I’m moving house :) Any chance we could do Thursday or Friday at the same time?
> > > > > > >
> > > > > > > Cheers
> > > > > > >
> > > > > > > Ian
> > > > > > > On 14 Apr 2021, 17:49 +0100, Kaxil Naik <ka...@gmail.com>, wrote:
> > > > > > >
> > > > > > > Just few comments here:
> > > > > > >
> > > > > > > Currently -- atleast for the foreseeable future Airflow workers will need access to the DAG Files, so workers can not run using the Serialized DAGs.
> > > > > > >
> > > > > > > Also serialized DAGs do not even have all the info needed for it to run it. Currently the serialization happens in the parsing process in the scheduler which can be in future separated as a separator "parsining" component, but that won't solve the "isolation" problem you are trying to solve. The only current way it can be solved is pickling -- and we have strictly decided against using pickling for DAGs.
> > > > > > >
> > > > > > > The idea in Statement (2) & (3) would help solve the isolation problem in (1) and can be done with some work now.
> > > > > > >
> > > > > > > Happy to talk about it in more detail here or on call, the time Daniel suggested works for me.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Kaxil
> > > > > > >
> > > > > > > On Wed, Apr 14, 2021 at 5:35 PM Daniel Imberman <da...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > How about Wednesday, April 21 at 8:00AM PST?
> > > > > > > >
> > > > > > > > On Wed, Apr 14, 2021 at 9:33 AM, Xinbin Huang <bi...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > I am available any days.
> > > > > > > >
> > > > > > > > On Wed, Apr 14, 2021, 9:32 AM Daniel Imberman <da...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi everyone!
> > > > > > > > >
> > > > > > > > > Would people be available around 8AM/9AM PST some point next week? I’m in PST and Ian is UTC+1 so would be great to find a timezone that accomodates everyone.
> > > > > > > > >
> > > > > > > > > Daniel
> > > > > > > > > On Wed, Apr 14, 2021 at 6:26 AM, Ryan Hatter <ry...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > I’d also like to be added please :)
> > > > > > > > >
> > > > > > > > > On Apr 13, 2021, at 21:27, Xinbin Huang <bi...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Daniel & Ian,
> > > > > > > > >
> > > > > > > > > I am also interested in the idea of a serialization representation that can be executed by workers directly. Can you also add me to the call?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Bin
> > > > > > > > >
> > > > > > > > > On Tue, Apr 13, 2021 at 2:49 PM Ian Buss <ia...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Daniel,
> > > > > > > > > >
> > > > > > > > > > Thanks for your warm welcome and quick response and the advice on providers! Will certainly check out the examples you sent.
> > > > > > > > > >
> > > > > > > > > > 1. An "airflow register" command definitely sounds promising, would love to collaborate on an AIP there so let's set something up.
> > > > > > > > > > 2. We use KubernetesExecutor exclusively as well. We've noticed significant additional load on the metadata DB as we scale up task pods so I've also thought about an API-based approach. Such an API could also open up the possibility of per-task security tokens which are injected by the scheduler, which should improve the security of such a system. Food for thought at least. I will start putting some of these thoughts down on paper in a sharable format.
> > > > > > > > > >
> > > > > > > > > > Ian
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 13, 2021 at 7:46 PM Daniel Imberman <da...@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Ian,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Firstly, welcome to the Airflow community :). I'm glad to hear you've had a positive experience so far. It's great to hear that you want to contribute back, and I think that multi-tenancy/DAG isolation is a pretty fantastic project for the community as a whole (a lot of things are are things we want but are limited by hours in a day).
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1. I've personally been kicking around some ideas lately about an "airflow register" command that would write the DAG into the metadata DB in a way that could be "gettable" by the workers via the API. This work is very early. I'd love to get some help on it. Perhaps we can set up a zoom chat to discuss drafting an AIP?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 2. Limiting worker access to the DB is not only good security practice; it also opens up the door to a lot of valuable features. This feature would be especially close to my heart as it would make the KubernetesExecutor significantly more efficient. It should be possible to set up a system where the workers only ever speak to an API server and never need to touch the DB.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 3. This is not something I personally have insight into, but I think it sounds like a good idea.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Finally, addressing your question about a Cloudera provider. If anything, it would probably give the provider _more_ legitimacy if you hosted it under the Cloudera GitHub org (we very purposely created the provider packages with this workflow in mind). There are multiple places where we can work to surface this provider so it is easy to find and use.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Astronomer has a pretty good sample provider here. One example of it running in the wild is the Great Expectations provider here. I'd also be glad to get you in contact with people who have built providers in the past to help you with that process.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Looking forward to seeing some of these things come to fruition!
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Daniel
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 13, 2021 at 9:43 AM, Ian Buss <ia...@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > First a quick introduction: I'm an engineer with Cloudera working on our Data Engineering product (CDE). Airflow is working great for us so far. We've been looking into how we can enhance the multi-tenancy story of Apache Airflow as we currently deploy it. We have the following areas which we'd like (with community consensus) to work on and contribute back to Apache Airflow to enhance the isolation between tenants in a single Airflow deployment.
> > > > > > > > > > >
> > > > > > > > > > > 1. Isolating code execution and parsing of DAG files. At the moment, DAG files are parsed in a few locations in Airflow, including the scheduler and in tasks. There is already the concept of DAG serialization (and we're using that for the web component) but we'd be interested to see if we can sandbox the execution of arbitrary user code to a locked down process/container without full access to the metadata DB and connection secrets etc. The idea would be to parse and serialize the DAG in this isolated container and pass back a serialized representation for persistence in the DB. Has anyone explored this idea?
> > > > > > > > > > >
> > > > > > > > > > > 2. Limiting task access to the metadata DB. It would be great if we could remove the requirement for tasks to have full access to the metadata DB and to report task status in a different (but still scalable) way. We'd need to tackle access or injection of connection, variable and xcom data as well for each task naturally.
> > > > > > > > > > >
> > > > > > > > > > > 3. Finer-grained access controls on connection secrets. Right now, although there are nice at-rest encryption options with Fernet or Vault, IIUC any DAG can access any connection (and thus any secret). Since the "run as" user is largely defined within the DAG and its tasks, this is challenging for a multi-tenant environment (see caveat below)
> > > > > > > > > > >
> > > > > > > > > > > Caveat: It's definitely noted that to some extent we should assume that an Airflow deployment is a "trusted" environment and that best practices such as git+PR workflows are the gold standard and that any malicious code and dependencies should be identified through this process. Also that there is a clear admin role for connection management etc.
> > > > > > > > > > >
> > > > > > > > > > > We have some ideas informally sketched out as to how to address the above but would be keen to hear the community opinion on this and to see if anyone is keen to collaborate on designs and implementation, or to hear if anything is already in the works. In particular I noticed that the very first improvement proposal (AIP-1) addresses much of the above :). However, it seems fairly dormant at the moment.
> > > > > > > > > > >
> > > > > > > > > > > One other question: we have a provider (operators and hooks) for interacting with Cloudera components that we'd like to contribute to the project. The provider FAQs indicate that new provider contributions are still welcome in the project in 2.x, is that accurate?
> > > > > > > > > > >
> > > > > > > > > > > Thanks in advance!
> > > > > > > > > > >
> > > > > > > > > > > Ian

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Jarek Potiuk <ja...@potiuk.com>.
For now I created a channel for the SIG:
https://apache-airflow.slack.com/archives/C02M551UDA4 - feel free to
join anyone and in the next weeks once all the people involved so far
expressed their interest, we should set some plan on getting the
AIP(s)?  drafted/discussed and start implementing it.

On Sat, Nov 6, 2021 at 11:22 PM Xinbin Huang <bi...@gmail.com> wrote:
>
> Hi Jarek,
>
> The plan sounds great! And +1 to a special interest group. Please add me to the group if you do create one.
>
> Here is the doc ( Airflow Multi-tenancy discussion ) we used to discuss back in April. It's not a note per-se, but I think it can shed some light on what we talked about. Other folks may have an actual note or even a draft proposal on this topic.
>
>   I'm excited for us to move forward with this.
>
> Bin
>
> On Fri, Nov 5, 2021 at 10:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Hello Ian, Everyone,
>>
>> I wonder if there are any notes from the meeting in April? Has there
>> been any more work on that one from Cloudera to formalize and plan
>> work on it?
>>
>> I was not able to participate, but I think it's about the time to
>> seriously start work on that and I am super happy to take more lead on
>> this project and involve all the interested parties. The ideas
>> described in the email and discussed after are I think super
>> reasonable and definitely necessary to get to the multi-tenancy and I
>> believe that there are already ideas that can be turned into reality
>> rather soon. I had a talk today also with the Google Composer team and
>> they are also fully on board with dedicating a lot of effort on this
>> one (and their ideas are I think super-aligned with Cloudera's), so I
>> think we have a critical mass and engineering power to make it happen
>> :)
>>
>> I plan to put quite a lot of focus on that one over the coming months
>> and I am happy to lead or co-lead the AIP and take a big part in
>> implementation.
>>
>> Possibly we should create a special interest group around that and
>> start drafting the AIP proposals in a smaller group of people who are
>> interested and start planning the work. I already have some ideas
>> where we could start gradually implementing it (of course after we
>> prepare the AIP and get it through the community's approval process).
>>
>> How does it sound?
>>
>> J.
>>
>> On Wed, Apr 21, 2021 at 8:56 AM Ian Buss <ia...@gmail.com> wrote:
>> >
>> > Yes, no invite required. See you tomorrow!
>> > On 21 Apr 2021, 07:46 +0100, Sumit Maheshwari <ms...@apache.org>, wrote:
>> >
>> > I'll join as well (I believe the zoom link will work without an invite)
>> >
>> > On Wed, Apr 21, 2021 at 10:48 AM Dimitris Stafylarakis <xa...@gmail.com> wrote:
>> >>
>> >> hi all,
>> >>
>> >> great to read about this, I'd like to join in! Can I just join using the zoom link tomorrow or do I need an invitation? (If I do need one, please invite me :))
>> >>
>> >> cheers
>> >>
>> >>
>> >> On Wed, Apr 14, 2021 at 8:15 PM Daniel Imberman <da...@gmail.com> wrote:
>> >>>
>> >>> Thank you Ian,
>> >>>
>> >>> I’ve invited everyone on this thread to the meeting with that zoom link. Anyone else who wants to join can add the calendar event here calendar.google.com/event?action=TEMPLATE&tmeid=Mm4zN2Q3MnFwNnBqbW9hMmNocXMyNzJpdHYgZGFuaWVsQGFzdHJvbm9tZXIuaW8&tmsrc=daniel@astronomer.io
>> >>>
>> >>> On Wed, Apr 14, 2021 at 11:05 AM, Ian Buss <ia...@gmail.com> wrote:
>> >>>
>> >>> If this works for everyone, here's a zoom link for Thursday 8AM PST: https://cloudera.zoom.us/j/99928254235?pwd=VTFlQk4vQjQ5Z2JzUDM3ZWZKKy9MQT09
>> >>>
>> >>> Happy to move or use an alternate method as needed.
>> >>>
>> >>> On Wed, Apr 14, 2021 at 6:58 PM Daniel Imberman <da...@gmail.com> wrote:
>> >>>>
>> >>>> Thursday works for me!
>> >>>>
>> >>>> On Wed, Apr 14, 2021 at 10:05 AM, Ian Buss <ia...@gmail.com> wrote:
>> >>>>
>> >>>> Hi all,
>> >>>>
>> >>>> I actually can’t do Wednesday next week as I’m moving house :) Any chance we could do Thursday or Friday at the same time?
>> >>>>
>> >>>> Cheers
>> >>>>
>> >>>> Ian
>> >>>> On 14 Apr 2021, 17:49 +0100, Kaxil Naik <ka...@gmail.com>, wrote:
>> >>>>
>> >>>> Just few comments here:
>> >>>>
>> >>>> Currently -- atleast for the foreseeable future Airflow workers will need access to the DAG Files, so workers can not run using the Serialized DAGs.
>> >>>>
>> >>>> Also serialized DAGs do not even have all the info needed for it to run it. Currently the serialization happens in the parsing process in the scheduler which can be in future separated as a separator "parsining" component, but that won't solve the "isolation" problem you are trying to solve. The only current way it can be solved is pickling -- and we have strictly decided against using pickling for DAGs.
>> >>>>
>> >>>> The idea in Statement (2) & (3) would help solve the isolation problem in (1) and can be done with some work now.
>> >>>>
>> >>>> Happy to talk about it in more detail here or on call, the time Daniel suggested works for me.
>> >>>>
>> >>>> Regards,
>> >>>> Kaxil
>> >>>>
>> >>>> On Wed, Apr 14, 2021 at 5:35 PM Daniel Imberman <da...@gmail.com> wrote:
>> >>>>>
>> >>>>> How about Wednesday, April 21 at 8:00AM PST?
>> >>>>>
>> >>>>> On Wed, Apr 14, 2021 at 9:33 AM, Xinbin Huang <bi...@gmail.com> wrote:
>> >>>>>
>> >>>>> I am available any days.
>> >>>>>
>> >>>>> On Wed, Apr 14, 2021, 9:32 AM Daniel Imberman <da...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> Hi everyone!
>> >>>>>>
>> >>>>>> Would people be available around 8AM/9AM PST some point next week? I’m in PST and Ian is UTC+1 so would be great to find a timezone that accomodates everyone.
>> >>>>>>
>> >>>>>> Daniel
>> >>>>>> On Wed, Apr 14, 2021 at 6:26 AM, Ryan Hatter <ry...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> I’d also like to be added please :)
>> >>>>>>
>> >>>>>> On Apr 13, 2021, at 21:27, Xinbin Huang <bi...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> 
>> >>>>>> Hi Daniel & Ian,
>> >>>>>>
>> >>>>>> I am also interested in the idea of a serialization representation that can be executed by workers directly. Can you also add me to the call?
>> >>>>>>
>> >>>>>> Thanks
>> >>>>>> Bin
>> >>>>>>
>> >>>>>> On Tue, Apr 13, 2021 at 2:49 PM Ian Buss <ia...@gmail.com> wrote:
>> >>>>>>>
>> >>>>>>> Daniel,
>> >>>>>>>
>> >>>>>>> Thanks for your warm welcome and quick response and the advice on providers! Will certainly check out the examples you sent.
>> >>>>>>>
>> >>>>>>> 1. An "airflow register" command definitely sounds promising, would love to collaborate on an AIP there so let's set something up.
>> >>>>>>> 2. We use KubernetesExecutor exclusively as well. We've noticed significant additional load on the metadata DB as we scale up task pods so I've also thought about an API-based approach. Such an API could also open up the possibility of per-task security tokens which are injected by the scheduler, which should improve the security of such a system. Food for thought at least. I will start putting some of these thoughts down on paper in a sharable format.
>> >>>>>>>
>> >>>>>>> Ian
>> >>>>>>>
>> >>>>>>> On Tue, Apr 13, 2021 at 7:46 PM Daniel Imberman <da...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>> Hi Ian,
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Firstly, welcome to the Airflow community :). I'm glad to hear you've had a positive experience so far. It's great to hear that you want to contribute back, and I think that multi-tenancy/DAG isolation is a pretty fantastic project for the community as a whole (a lot of things are are things we want but are limited by hours in a day).
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> 1. I've personally been kicking around some ideas lately about an "airflow register" command that would write the DAG into the metadata DB in a way that could be "gettable" by the workers via the API. This work is very early. I'd love to get some help on it. Perhaps we can set up a zoom chat to discuss drafting an AIP?
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> 2. Limiting worker access to the DB is not only good security practice; it also opens up the door to a lot of valuable features. This feature would be especially close to my heart as it would make the KubernetesExecutor significantly more efficient. It should be possible to set up a system where the workers only ever speak to an API server and never need to touch the DB.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> 3. This is not something I personally have insight into, but I think it sounds like a good idea.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Finally, addressing your question about a Cloudera provider. If anything, it would probably give the provider _more_ legitimacy if you hosted it under the Cloudera GitHub org (we very purposely created the provider packages with this workflow in mind). There are multiple places where we can work to surface this provider so it is easy to find and use.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Astronomer has a pretty good sample provider here. One example of it running in the wild is the Great Expectations provider here. I'd also be glad to get you in contact with people who have built providers in the past to help you with that process.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Looking forward to seeing some of these things come to fruition!
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Daniel
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Tue, Apr 13, 2021 at 9:43 AM, Ian Buss <ia...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>> Hi all,
>> >>>>>>>>
>> >>>>>>>> First a quick introduction: I'm an engineer with Cloudera working on our Data Engineering product (CDE). Airflow is working great for us so far. We've been looking into how we can enhance the multi-tenancy story of Apache Airflow as we currently deploy it. We have the following areas which we'd like (with community consensus) to work on and contribute back to Apache Airflow to enhance the isolation between tenants in a single Airflow deployment.
>> >>>>>>>>
>> >>>>>>>> 1. Isolating code execution and parsing of DAG files. At the moment, DAG files are parsed in a few locations in Airflow, including the scheduler and in tasks. There is already the concept of DAG serialization (and we're using that for the web component) but we'd be interested to see if we can sandbox the execution of arbitrary user code to a locked down process/container without full access to the metadata DB and connection secrets etc. The idea would be to parse and serialize the DAG in this isolated container and pass back a serialized representation for persistence in the DB. Has anyone explored this idea?
>> >>>>>>>>
>> >>>>>>>> 2. Limiting task access to the metadata DB. It would be great if we could remove the requirement for tasks to have full access to the metadata DB and to report task status in a different (but still scalable) way. We'd need to tackle access or injection of connection, variable and xcom data as well for each task naturally.
>> >>>>>>>>
>> >>>>>>>> 3. Finer-grained access controls on connection secrets. Right now, although there are nice at-rest encryption options with Fernet or Vault, IIUC any DAG can access any connection (and thus any secret). Since the "run as" user is largely defined within the DAG and its tasks, this is challenging for a multi-tenant environment (see caveat below)
>> >>>>>>>>
>> >>>>>>>> Caveat: It's definitely noted that to some extent we should assume that an Airflow deployment is a "trusted" environment and that best practices such as git+PR workflows are the gold standard and that any malicious code and dependencies should be identified through this process. Also that there is a clear admin role for connection management etc.
>> >>>>>>>>
>> >>>>>>>> We have some ideas informally sketched out as to how to address the above but would be keen to hear the community opinion on this and to see if anyone is keen to collaborate on designs and implementation, or to hear if anything is already in the works. In particular I noticed that the very first improvement proposal (AIP-1) addresses much of the above :). However, it seems fairly dormant at the moment.
>> >>>>>>>>
>> >>>>>>>> One other question: we have a provider (operators and hooks) for interacting with Cloudera components that we'd like to contribute to the project. The provider FAQs indicate that new provider contributions are still welcome in the project in 2.x, is that accurate?
>> >>>>>>>>
>> >>>>>>>> Thanks in advance!
>> >>>>>>>>
>> >>>>>>>> Ian

Re: [DISCUSS] AIP-1 and Airflow multi-tenancy

Posted by Xinbin Huang <bi...@gmail.com>.
Hi Jarek,

The plan sounds great! And +1 to a special interest group. Please add me to
the group if you do create one.

Here is the doc ( Airflow Multi-tenancy discussion
<https://docs.google.com/document/d/17kgfLO2fpNC62YuCxo1l0yRipt48YE6g1d1SGUok2I8/edit#heading=h.d14mqct3autb>
)
we used to discuss back in April. It's not a note per-se, but I think it
can shed some light on what we talked about. Other folks may have an actual
note or even a draft proposal on this topic.

  I'm excited for us to move forward with this.

Bin

On Fri, Nov 5, 2021 at 10:38 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hello Ian, Everyone,
>
> I wonder if there are any notes from the meeting in April? Has there
> been any more work on that one from Cloudera to formalize and plan
> work on it?
>
> I was not able to participate, but I think it's about the time to
> seriously start work on that and I am super happy to take more lead on
> this project and involve all the interested parties. The ideas
> described in the email and discussed after are I think super
> reasonable and definitely necessary to get to the multi-tenancy and I
> believe that there are already ideas that can be turned into reality
> rather soon. I had a talk today also with the Google Composer team and
> they are also fully on board with dedicating a lot of effort on this
> one (and their ideas are I think super-aligned with Cloudera's), so I
> think we have a critical mass and engineering power to make it happen
> :)
>
> I plan to put quite a lot of focus on that one over the coming months
> and I am happy to lead or co-lead the AIP and take a big part in
> implementation.
>
> Possibly we should create a special interest group around that and
> start drafting the AIP proposals in a smaller group of people who are
> interested and start planning the work. I already have some ideas
> where we could start gradually implementing it (of course after we
> prepare the AIP and get it through the community's approval process).
>
> How does it sound?
>
> J.
>
> On Wed, Apr 21, 2021 at 8:56 AM Ian Buss <ia...@gmail.com> wrote:
> >
> > Yes, no invite required. See you tomorrow!
> > On 21 Apr 2021, 07:46 +0100, Sumit Maheshwari <ms...@apache.org>,
> wrote:
> >
> > I'll join as well (I believe the zoom link will work without an invite)
> >
> > On Wed, Apr 21, 2021 at 10:48 AM Dimitris Stafylarakis <xa...@gmail.com>
> wrote:
> >>
> >> hi all,
> >>
> >> great to read about this, I'd like to join in! Can I just join using
> the zoom link tomorrow or do I need an invitation? (If I do need one,
> please invite me :))
> >>
> >> cheers
> >>
> >>
> >> On Wed, Apr 14, 2021 at 8:15 PM Daniel Imberman <
> daniel.imberman@gmail.com> wrote:
> >>>
> >>> Thank you Ian,
> >>>
> >>> I’ve invited everyone on this thread to the meeting with that zoom
> link. Anyone else who wants to join can add the calendar event here
> calendar.google.com/event?action=TEMPLATE&tmeid=Mm4zN2Q3MnFwNnBqbW9hMmNocXMyNzJpdHYgZGFuaWVsQGFzdHJvbm9tZXIuaW8&tmsrc=daniel@astronomer.io
> >>>
> >>> On Wed, Apr 14, 2021 at 11:05 AM, Ian Buss <ia...@gmail.com> wrote:
> >>>
> >>> If this works for everyone, here's a zoom link for Thursday 8AM PST:
> https://cloudera.zoom.us/j/99928254235?pwd=VTFlQk4vQjQ5Z2JzUDM3ZWZKKy9MQT09
> >>>
> >>> Happy to move or use an alternate method as needed.
> >>>
> >>> On Wed, Apr 14, 2021 at 6:58 PM Daniel Imberman <
> daniel.imberman@gmail.com> wrote:
> >>>>
> >>>> Thursday works for me!
> >>>>
> >>>> On Wed, Apr 14, 2021 at 10:05 AM, Ian Buss <ia...@gmail.com>
> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I actually can’t do Wednesday next week as I’m moving house :) Any
> chance we could do Thursday or Friday at the same time?
> >>>>
> >>>> Cheers
> >>>>
> >>>> Ian
> >>>> On 14 Apr 2021, 17:49 +0100, Kaxil Naik <ka...@gmail.com>, wrote:
> >>>>
> >>>> Just few comments here:
> >>>>
> >>>> Currently -- atleast for the foreseeable future Airflow workers will
> need access to the DAG Files, so workers can not run using the Serialized
> DAGs.
> >>>>
> >>>> Also serialized DAGs do not even have all the info needed for it to
> run it. Currently the serialization happens in the parsing process in the
> scheduler which can be in future separated as a separator "parsining"
> component, but that won't solve the "isolation" problem you are trying to
> solve. The only current way it can be solved is pickling -- and we have
> strictly decided against using pickling for DAGs.
> >>>>
> >>>> The idea in Statement (2) & (3) would help solve the isolation
> problem in (1) and can be done with some work now.
> >>>>
> >>>> Happy to talk about it in more detail here or on call, the time
> Daniel suggested works for me.
> >>>>
> >>>> Regards,
> >>>> Kaxil
> >>>>
> >>>> On Wed, Apr 14, 2021 at 5:35 PM Daniel Imberman <
> daniel.imberman@gmail.com> wrote:
> >>>>>
> >>>>> How about Wednesday, April 21 at 8:00AM PST?
> >>>>>
> >>>>> On Wed, Apr 14, 2021 at 9:33 AM, Xinbin Huang <bi...@gmail.com>
> wrote:
> >>>>>
> >>>>> I am available any days.
> >>>>>
> >>>>> On Wed, Apr 14, 2021, 9:32 AM Daniel Imberman <
> daniel.imberman@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi everyone!
> >>>>>>
> >>>>>> Would people be available around 8AM/9AM PST some point next week?
> I’m in PST and Ian is UTC+1 so would be great to find a timezone that
> accomodates everyone.
> >>>>>>
> >>>>>> Daniel
> >>>>>> On Wed, Apr 14, 2021 at 6:26 AM, Ryan Hatter <ry...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> I’d also like to be added please :)
> >>>>>>
> >>>>>> On Apr 13, 2021, at 21:27, Xinbin Huang <bi...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> 
> >>>>>> Hi Daniel & Ian,
> >>>>>>
> >>>>>> I am also interested in the idea of a serialization representation
> that can be executed by workers directly. Can you also add me to the call?
> >>>>>>
> >>>>>> Thanks
> >>>>>> Bin
> >>>>>>
> >>>>>> On Tue, Apr 13, 2021 at 2:49 PM Ian Buss <ia...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> Daniel,
> >>>>>>>
> >>>>>>> Thanks for your warm welcome and quick response and the advice on
> providers! Will certainly check out the examples you sent.
> >>>>>>>
> >>>>>>> 1. An "airflow register" command definitely sounds promising,
> would love to collaborate on an AIP there so let's set something up.
> >>>>>>> 2. We use KubernetesExecutor exclusively as well. We've noticed
> significant additional load on the metadata DB as we scale up task pods so
> I've also thought about an API-based approach. Such an API could also open
> up the possibility of per-task security tokens which are injected by the
> scheduler, which should improve the security of such a system. Food for
> thought at least. I will start putting some of these thoughts down on paper
> in a sharable format.
> >>>>>>>
> >>>>>>> Ian
> >>>>>>>
> >>>>>>> On Tue, Apr 13, 2021 at 7:46 PM Daniel Imberman <
> daniel.imberman@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Hi Ian,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Firstly, welcome to the Airflow community :). I'm glad to hear
> you've had a positive experience so far. It's great to hear that you want
> to contribute back, and I think that multi-tenancy/DAG isolation is a
> pretty fantastic project for the community as a whole (a lot of things are
> are things we want but are limited by hours in a day).
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 1. I've personally been kicking around some ideas lately about an
> "airflow register" command that would write the DAG into the metadata DB in
> a way that could be "gettable" by the workers via the API. This work is
> very early. I'd love to get some help on it. Perhaps we can set up a zoom
> chat to discuss drafting an AIP?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 2. Limiting worker access to the DB is not only good security
> practice; it also opens up the door to a lot of valuable features. This
> feature would be especially close to my heart as it would make the
> KubernetesExecutor significantly more efficient. It should be possible to
> set up a system where the workers only ever speak to an API server and
> never need to touch the DB.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 3. This is not something I personally have insight into, but I
> think it sounds like a good idea.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Finally, addressing your question about a Cloudera provider. If
> anything, it would probably give the provider _more_ legitimacy if you
> hosted it under the Cloudera GitHub org (we very purposely created the
> provider packages with this workflow in mind). There are multiple places
> where we can work to surface this provider so it is easy to find and use.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Astronomer has a pretty good sample provider here. One example of
> it running in the wild is the Great Expectations provider here. I'd also be
> glad to get you in contact with people who have built providers in the past
> to help you with that process.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Looking forward to seeing some of these things come to fruition!
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Daniel
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Apr 13, 2021 at 9:43 AM, Ian Buss <ia...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> First a quick introduction: I'm an engineer with Cloudera working
> on our Data Engineering product (CDE). Airflow is working great for us so
> far. We've been looking into how we can enhance the multi-tenancy story of
> Apache Airflow as we currently deploy it. We have the following areas which
> we'd like (with community consensus) to work on and contribute back to
> Apache Airflow to enhance the isolation between tenants in a single Airflow
> deployment.
> >>>>>>>>
> >>>>>>>> 1. Isolating code execution and parsing of DAG files. At the
> moment, DAG files are parsed in a few locations in Airflow, including the
> scheduler and in tasks. There is already the concept of DAG serialization
> (and we're using that for the web component) but we'd be interested to see
> if we can sandbox the execution of arbitrary user code to a locked down
> process/container without full access to the metadata DB and connection
> secrets etc. The idea would be to parse and serialize the DAG in this
> isolated container and pass back a serialized representation for
> persistence in the DB. Has anyone explored this idea?
> >>>>>>>>
> >>>>>>>> 2. Limiting task access to the metadata DB. It would be great if
> we could remove the requirement for tasks to have full access to the
> metadata DB and to report task status in a different (but still scalable)
> way. We'd need to tackle access or injection of connection, variable and
> xcom data as well for each task naturally.
> >>>>>>>>
> >>>>>>>> 3. Finer-grained access controls on connection secrets. Right
> now, although there are nice at-rest encryption options with Fernet or
> Vault, IIUC any DAG can access any connection (and thus any secret). Since
> the "run as" user is largely defined within the DAG and its tasks, this is
> challenging for a multi-tenant environment (see caveat below)
> >>>>>>>>
> >>>>>>>> Caveat: It's definitely noted that to some extent we should
> assume that an Airflow deployment is a "trusted" environment and that best
> practices such as git+PR workflows are the gold standard and that any
> malicious code and dependencies should be identified through this process.
> Also that there is a clear admin role for connection management etc.
> >>>>>>>>
> >>>>>>>> We have some ideas informally sketched out as to how to address
> the above but would be keen to hear the community opinion on this and to
> see if anyone is keen to collaborate on designs and implementation, or to
> hear if anything is already in the works. In particular I noticed that the
> very first improvement proposal (AIP-1) addresses much of the above :).
> However, it seems fairly dormant at the moment.
> >>>>>>>>
> >>>>>>>> One other question: we have a provider (operators and hooks) for
> interacting with Cloudera components that we'd like to contribute to the
> project. The provider FAQs indicate that new provider contributions are
> still welcome in the project in 2.x, is that accurate?
> >>>>>>>>
> >>>>>>>> Thanks in advance!
> >>>>>>>>
> >>>>>>>> Ian
>