You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Beck, Vincent" <vi...@amazon.com.INVALID> on 2023/03/28 21:04:33 UTC
AIP-56 Extensible user management
Hi all,
I would like to get your feedback on an AIP I wrote: "Extensible user management". This AIP is a follow-up on a discussion we had in this email list on the multi tenancy topic. I decided to create a new email thread because I feel the topic had diverged a bit from the original topic (multi tenancy).
As a summary, the purpose of this AIP is to extract the user management from core Airflow by introducing a user manager interface in the core Airflow that can be extended by any provider package who want to support user management natively. As a result, if this AIP gets approved and go through, the multi tenancy feature will be implemented as a second step in a new user manager and not in Airflow directly.
As always, feel very free to give your opinion on this email thread or on the AIP by adding comments.
References:
- AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
- Initial email list discussion: https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61
Vincent
Re: AIP-56 Extensible user management
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
The AIP has been voted and accepted. I created a board on Github to track all the different tasks related to the AIP. The board is here: https://github.com/orgs/apache/projects/276
If you're interested to work on any specific issue/task, feel free to comment, I'll be happy to assign it to you. If you also think some tasks are missing, feel free to create them directly.
On 2023-06-20, 10:54 AM, "Beck, Vincent" <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>LID> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
It is never too late :)
These views won't change or won't move, they are not part of the views we will move into auth managers. I am not sure you meant that but I wanted to clarify that. And to your second question, the goal is to replace this decorator which checks if a user has permissions to access to this specific DAG/view to a call (which can be wrapped into a decorator as well, leave that to implementation details) to the method `is_authorized` of the auth manager. We now want the auth manager to take these authz decisions through this method. We will still be able to configure permissions for these views as we are today, as a matter of fact, the purpose of FAB provider is to provide a backward compatible experience, so it is safe to say that whatever you can do today, you should be able to do it with the FAB auth manager.
On 2023-06-20, 9:14 AM, "Pankaj Koti" <pankaj.koti@astronomer.io.INVA <ma...@astronomer.io.INVA> <mailto:pankaj.koti@astronomer.io.INVA <ma...@astronomer.io.INVA>>LID> wrote:
hi,
I like the AIP a lot.
Sorry for joining the discussion late.
I had a question regarding access to UI views in Airflow.
I checked the list of resources in Security -> Resources. I see there we
have DAGs as resources. However for these DAGs there are views exposed with
the help of flask app
e.g. the /dags/<dag_id>/graph or the /dags/<dag_id>/code are views in the
UI. Currently, we rely on a set of decorators to verify that a user has
perms to access the DAG or not.
Would we also consider such views as resources in the AIP and be able to
configure perms for users for those views as part of the AIP?
Regards,
Pankaj Koti
*Senior Software Engineer, *OSS Engineering Team.
Location: Pune, India
Timezone: Indian Standard Time (IST)
Email: pankaj.koti@astronomer.io <ma...@astronomer.io> <mailto:pankaj.koti@astronomer.io <ma...@astronomer.io>>
Mobile: +91 9730079985
On Fri, Jun 16, 2023 at 10:21 PM Jed Cunningham <jedcunningham@apache.org <ma...@apache.org> <mailto:jedcunningham@apache.org <ma...@apache.org>>>
wrote:
> Sounds good to me. Hopefully we can make it happen, but we give ourselves
> an escape hatch :)
>
> On Fri, Jun 16, 2023 at 9:45 AM Beck, Vincent <vincbeck@amazon.com.inva <ma...@amazon.com.inva> <mailto:vincbeck@amazon.com.inva <ma...@amazon.com.inva>>lid
> >
> wrote:
>
> > Thanks for the feedback Jed, that's something I confess I had not thought
> > about. It is a valid concern. What I can propose as a compromise is to
> > leave the "move FAB auth manager to a separate provider" task to the end
> of
> > the project and mark it as optional. In other words, we can build
> > everything as described in the AIP but keep it in core Airflow as opposed
> > to a separate provider. All the user management logic would still be
> > "packaged" in the auth manager, and then when everything is implemented
> and
> > done we can decide whether moving this FAB auth manager to a separate
> > provider is feasible/a good idea.
> >
> > What do you think?
> >
> > On 2023-06-12, 11:18 AM, "Jed Cunningham" <jed@astronomer.io.INVA <ma...@astronomer.io.INVA> <mailto:jed@astronomer.io.INVA <ma...@astronomer.io.INVA>>
> > <mailto:jed@astronomer.io.INVA <ma...@astronomer.io.INVA> <mailto:jed@astronomer.io.INVA <ma...@astronomer.io.INVA>>>LID> wrote:
> >
> >
> >
> > Overall I'm happy with the proposal. One thing that concerns me though is
> > moving the FAB auth manager into a separate provider. That auth manager
> > will need to be able to hook into the db migration tooling, and we don't
> > expose that to providers or plugins today. So if we do want to move it,
> we
> > have to account for that as well.
> >
> >
> > I feel the vast majority of the benefit here comes from being able to sub
> > in another auth manager, but having the FAB one "not in core" isn't all
> > that consequential at the end of the day. It would keep the strict
> version
> > requirements of FAB in core though - pick your poison I guess :)
> >
> >
> >
> >
>
?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB�?�?[��X��ܚX�K??K[XZ[?�??]�][��X��ܚX�P?Z\��?�˘\?X�?K�ܙ�B��܈?Y??]?[ۘ[??��[X[�?�??K[XZ[?�??]�Z?[???Z\��?�˘\?X�?K�ܙ�B
Re: AIP-56 Extensible user management
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
It is never too late :)
These views won't change or won't move, they are not part of the views we will move into auth managers. I am not sure you meant that but I wanted to clarify that. And to your second question, the goal is to replace this decorator which checks if a user has permissions to access to this specific DAG/view to a call (which can be wrapped into a decorator as well, leave that to implementation details) to the method `is_authorized` of the auth manager. We now want the auth manager to take these authz decisions through this method. We will still be able to configure permissions for these views as we are today, as a matter of fact, the purpose of FAB provider is to provide a backward compatible experience, so it is safe to say that whatever you can do today, you should be able to do it with the FAB auth manager.
On 2023-06-20, 9:14 AM, "Pankaj Koti" <pankaj.koti@astronomer.io.INVA <ma...@astronomer.io.INVA>LID> wrote:
hi,
I like the AIP a lot.
Sorry for joining the discussion late.
I had a question regarding access to UI views in Airflow.
I checked the list of resources in Security -> Resources. I see there we
have DAGs as resources. However for these DAGs there are views exposed with
the help of flask app
e.g. the /dags/<dag_id>/graph or the /dags/<dag_id>/code are views in the
UI. Currently, we rely on a set of decorators to verify that a user has
perms to access the DAG or not.
Would we also consider such views as resources in the AIP and be able to
configure perms for users for those views as part of the AIP?
Regards,
Pankaj Koti
*Senior Software Engineer, *OSS Engineering Team.
Location: Pune, India
Timezone: Indian Standard Time (IST)
Email: pankaj.koti@astronomer.io <ma...@astronomer.io>
Mobile: +91 9730079985
On Fri, Jun 16, 2023 at 10:21 PM Jed Cunningham <jedcunningham@apache.org <ma...@apache.org>>
wrote:
> Sounds good to me. Hopefully we can make it happen, but we give ourselves
> an escape hatch :)
>
> On Fri, Jun 16, 2023 at 9:45 AM Beck, Vincent <vincbeck@amazon.com.inva <ma...@amazon.com.inva>lid
> >
> wrote:
>
> > Thanks for the feedback Jed, that's something I confess I had not thought
> > about. It is a valid concern. What I can propose as a compromise is to
> > leave the "move FAB auth manager to a separate provider" task to the end
> of
> > the project and mark it as optional. In other words, we can build
> > everything as described in the AIP but keep it in core Airflow as opposed
> > to a separate provider. All the user management logic would still be
> > "packaged" in the auth manager, and then when everything is implemented
> and
> > done we can decide whether moving this FAB auth manager to a separate
> > provider is feasible/a good idea.
> >
> > What do you think?
> >
> > On 2023-06-12, 11:18 AM, "Jed Cunningham" <jed@astronomer.io.INVA <ma...@astronomer.io.INVA>
> > <mailto:jed@astronomer.io.INVA <ma...@astronomer.io.INVA>>LID> wrote:
> >
> >
> >
> > Overall I'm happy with the proposal. One thing that concerns me though is
> > moving the FAB auth manager into a separate provider. That auth manager
> > will need to be able to hook into the db migration tooling, and we don't
> > expose that to providers or plugins today. So if we do want to move it,
> we
> > have to account for that as well.
> >
> >
> > I feel the vast majority of the benefit here comes from being able to sub
> > in another auth manager, but having the FAB one "not in core" isn't all
> > that consequential at the end of the day. It would keep the strict
> version
> > requirements of FAB in core though - pick your poison I guess :)
> >
> >
> >
> >
>
Re: AIP-56 Extensible user management
Posted by Pankaj Koti <pa...@astronomer.io.INVALID>.
hi,
I like the AIP a lot.
Sorry for joining the discussion late.
I had a question regarding access to UI views in Airflow.
I checked the list of resources in Security -> Resources. I see there we
have DAGs as resources. However for these DAGs there are views exposed with
the help of flask app
e.g. the /dags/<dag_id>/graph or the /dags/<dag_id>/code are views in the
UI. Currently, we rely on a set of decorators to verify that a user has
perms to access the DAG or not.
Would we also consider such views as resources in the AIP and be able to
configure perms for users for those views as part of the AIP?
Regards,
Pankaj Koti
*Senior Software Engineer, *OSS Engineering Team.
Location: Pune, India
Timezone: Indian Standard Time (IST)
Email: pankaj.koti@astronomer.io
Mobile: +91 9730079985
On Fri, Jun 16, 2023 at 10:21 PM Jed Cunningham <je...@apache.org>
wrote:
> Sounds good to me. Hopefully we can make it happen, but we give ourselves
> an escape hatch :)
>
> On Fri, Jun 16, 2023 at 9:45 AM Beck, Vincent <vincbeck@amazon.com.invalid
> >
> wrote:
>
> > Thanks for the feedback Jed, that's something I confess I had not thought
> > about. It is a valid concern. What I can propose as a compromise is to
> > leave the "move FAB auth manager to a separate provider" task to the end
> of
> > the project and mark it as optional. In other words, we can build
> > everything as described in the AIP but keep it in core Airflow as opposed
> > to a separate provider. All the user management logic would still be
> > "packaged" in the auth manager, and then when everything is implemented
> and
> > done we can decide whether moving this FAB auth manager to a separate
> > provider is feasible/a good idea.
> >
> > What do you think?
> >
> > On 2023-06-12, 11:18 AM, "Jed Cunningham" <jed@astronomer.io.INVA
> > <ma...@astronomer.io.INVA>LID> wrote:
> >
> >
> >
> > Overall I'm happy with the proposal. One thing that concerns me though is
> > moving the FAB auth manager into a separate provider. That auth manager
> > will need to be able to hook into the db migration tooling, and we don't
> > expose that to providers or plugins today. So if we do want to move it,
> we
> > have to account for that as well.
> >
> >
> > I feel the vast majority of the benefit here comes from being able to sub
> > in another auth manager, but having the FAB one "not in core" isn't all
> > that consequential at the end of the day. It would keep the strict
> version
> > requirements of FAB in core though - pick your poison I guess :)
> >
> >
> >
> >
>
Re: AIP-56 Extensible user management
Posted by Jed Cunningham <je...@apache.org>.
Sounds good to me. Hopefully we can make it happen, but we give ourselves
an escape hatch :)
On Fri, Jun 16, 2023 at 9:45 AM Beck, Vincent <vi...@amazon.com.invalid>
wrote:
> Thanks for the feedback Jed, that's something I confess I had not thought
> about. It is a valid concern. What I can propose as a compromise is to
> leave the "move FAB auth manager to a separate provider" task to the end of
> the project and mark it as optional. In other words, we can build
> everything as described in the AIP but keep it in core Airflow as opposed
> to a separate provider. All the user management logic would still be
> "packaged" in the auth manager, and then when everything is implemented and
> done we can decide whether moving this FAB auth manager to a separate
> provider is feasible/a good idea.
>
> What do you think?
>
> On 2023-06-12, 11:18 AM, "Jed Cunningham" <jed@astronomer.io.INVA
> <ma...@astronomer.io.INVA>LID> wrote:
>
>
>
> Overall I'm happy with the proposal. One thing that concerns me though is
> moving the FAB auth manager into a separate provider. That auth manager
> will need to be able to hook into the db migration tooling, and we don't
> expose that to providers or plugins today. So if we do want to move it, we
> have to account for that as well.
>
>
> I feel the vast majority of the benefit here comes from being able to sub
> in another auth manager, but having the FAB one "not in core" isn't all
> that consequential at the end of the day. It would keep the strict version
> requirements of FAB in core though - pick your poison I guess :)
>
>
>
>
Re: AIP-56 Extensible user management
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
Thanks for the feedback Jed, that's something I confess I had not thought about. It is a valid concern. What I can propose as a compromise is to leave the "move FAB auth manager to a separate provider" task to the end of the project and mark it as optional. In other words, we can build everything as described in the AIP but keep it in core Airflow as opposed to a separate provider. All the user management logic would still be "packaged" in the auth manager, and then when everything is implemented and done we can decide whether moving this FAB auth manager to a separate provider is feasible/a good idea.
What do you think?
On 2023-06-12, 11:18 AM, "Jed Cunningham" <jed@astronomer.io.INVA <ma...@astronomer.io.INVA>LID> wrote:
Overall I'm happy with the proposal. One thing that concerns me though is
moving the FAB auth manager into a separate provider. That auth manager
will need to be able to hook into the db migration tooling, and we don't
expose that to providers or plugins today. So if we do want to move it, we
have to account for that as well.
I feel the vast majority of the benefit here comes from being able to sub
in another auth manager, but having the FAB one "not in core" isn't all
that consequential at the end of the day. It would keep the strict version
requirements of FAB in core though - pick your poison I guess :)
Re: AIP-56 Extensible user management
Posted by Jed Cunningham <je...@astronomer.io.INVALID>.
Overall I'm happy with the proposal. One thing that concerns me though is
moving the FAB auth manager into a separate provider. That auth manager
will need to be able to hook into the db migration tooling, and we don't
expose that to providers or plugins today. So if we do want to move it, we
have to account for that as well.
I feel the vast majority of the benefit here comes from being able to sub
in another auth manager, but having the FAB one "not in core" isn't all
that consequential at the end of the day. It would keep the strict version
requirements of FAB in core though - pick your poison I guess :)
Re: AIP-56 Extensible user management
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
Hello,
I updated the AIP (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management) to add a new section about the "is_authorized" API context. I tried to address all the concerns and questions gathered in this thread. You can find this new section under "Authorization flow". As usual, free feel to go through it and give feedbacks.
One clarification on a point brought up in the thread before:
> I think this should be extensible (= flexible). Because, I would add remote_addr, proxy_ips, roles, tag etc.
I definitely agree with you on the extensible part of it. I tried to explained it clearly in the AIP. Although, as far as I understand, "remote_addr", "proxy_ips", "roles" are metadata related to users. Therefore, I would not necessarily include it in the request because the auth manager can retrieve them itself when processing the "is_authorized" request. Any context information related to resources are welcome to be added in the context of the "is_authorized" API, but I don’t think we should add context information related to users.
Vincent
On 2023-05-30, 2:04 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>LID> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Somehow, my last email does not show up on the Apache list web view (https://lists.apache.org/thread/ck8dsj5w82lvr0cpwr4wlptmydqwnsqc <https://lists.apache.org/thread/ck8dsj5w82lvr0cpwr4wlptmydqwnsqc>). Re-sending it to hopefully add it there. Sorry for the noise.
Hello,
I tried to answer to all open questions:
> If I understand correctly, the proposition in AIP-56 is to introduce an interface as generic as possible so that any potential federated identity protocol can be integrated in Airflow through User Manager (or as proposed in this thread, AuthManager)
You're 100% correct. This is the ultimate goal. Note on this one, based on the last feedbacks, it seems Auth manager makes more sense than User manager, which I agree. Thus, I updated the AIP to replace user manager by auth manager.
> What do you think would be the context of "current user" provided by Airflow to the implementation of AuthManager?
When a user is logged in Airflow (through an auth manager), a Flask session will be set as it is today. This will be the context of "current user". In this session, you can find user information such as user ID or OIDC token (if the auth manager used use that kind of token). This is the way I envisioned so that the auth manager can retrieve the OIDC token in order to interact with the auth manager backend. For now we know that the FAB auth manager will need the user ID as context and the KeyCloak auth manager will need a OIDC token. However, this set of information must be extensible so that new auth manager which do not use OIDC token can be added in Airflow in the future.
> Do you think it makes sense to document this in the AIP or leave decisions for later?
I think it makes total sense to add it to the AIP, which I will do, I hope it will make it clearer :)
> is_authorized should include context so that it becomes possible the determine whether a user has access to a resource based on for example the location the user is at. Or in which environment. It is also useful for auditing
I see your points. I agree then, we should come up with a context of the is_authorized API. I have not started to work on that yet but will do in the following days. If you also want to give it a shot and come up with something, that would be very much appreciated :)
> Particularly with security, I'd prefer clarity of how we intend to implement this. If we consider an access_token equivalent to an OIDC access_token that is fine, but let us be clear on what we base our design on instead of seemingly inventing our own.
I agree with you. Every time I refer to access token in the AIP, I meant OIDC access token, I'll add that clarification, thanks for the callout. However, this is very specific to each auth manager backend. In the case of KeyCloak, the access token I referred to is indeed an OIDC access token but it is impossible to assert this will be true for all auth managers implemented in the future. We need to keep in mind we cannot enforce third party auth services to implement the protocol we want but we need to be flexible/adaptable enough so we can integrate with majority of them.
> At this moment, *any* DAG has full access to the database. That means if we store user_id -> access_token any DAG can access any resource on behalf of someone else. This means the AIrflow DB is one of the most insecure places to store this kind of information.
That is correct. Though this will be fixed in the future with AIP-44 (or a following on this one) if I am not mistaken.
> post_login is not required if authentication is entirely handled by the AuthManager. If we consider post_login to be equivalent to a callback method from OIDC (again: on what do we base our security design?) then sure but then let's call it as such.
This is indeed a callback method from OIDC. Again, I did not want to be OIDC specific because we cannot assert all auth managers in the future will use OIDC protocol. That's why I came up with the name "post_login". I wanted something generic. Though, I am happy to change the name if that makes things easier to understand.
As usual, thanks for your feedbacks!
Vincent
On 2023-05-19, 11:35 AM, "Jakub Kośmider" <kosmider@google.com.INVA <ma...@google.com.INVA> <mailto:kosmider@google.com.INVA <ma...@google.com.INVA>>LID> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hello,
If I understand correctly, the proposition in AIP-56 is to introduce an
interface as generic as possible so that any potential federated identity
protocol can be integrated in Airflow through User Manager (or as proposed
in this thread, AuthManager). In light of this, note that the methods in
AuthManager in the AIP refer to the "current user", while the example of
"is_authorized" used above is based only on specific resources and
permissions provided as method parameters.
What do you think would be the context of "current user" provided by
Airflow to the implementation of AuthManager?
Would it be the entire request or maybe just cookies/auth tokens associated
with the request or something else?
Do you think it makes sense to document this in the AIP or leave decisions
for later?
Best regards,
Jakub
On Fri, May 19, 2023 at 11:22 AM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> wrote:
> > Overall: I strongly suggest basing our security design on something
> existing. For AuthN I like the OIDC scheme. Furthermore, I suggest looking
> at https://flask-login.readthedocs.io/en/latest/ <https://flask-login.readthedocs.io/en/latest/> <https://flask-login.readthedocs.io/en/latest/> <https://flask-login.readthedocs.io/en/latest/>> to see how this is
> handled
> in Flask.
>
> Very much agree. Good suggestion Bolke.
>
>
> On Fri, May 19, 2023 at 9:44 AM Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>> wrote:
>
> > Hi,
> >
> > On Wed, 17 May 2023 at 18:48, Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> wrote:
> >
> > > >
> > > >
> > > > I am very open to change the name of "User Manager" but I do not
> think
> > we
> > > > should call it "Security Manager". "Security Manager" is the current
> > name
> > > > and I think this is a very bad name because: it is very generic and
> > does
> > > > not say much about what it is actually doing. By reading "Security
> > > Manager"
> > > > I would not think this is where users and roles are managed. The main
> > > goal
> > > > of AIP-56 is removing the whole user management part from core
> Airflow
> > to
> > > > providers. Keeping the name "Security Manager" would, I think, leave
> a
> > > > legacy for all future user managers (or whatever name we use) on
> > > > Flask-AppBuilder (FAB) and that's exactly what I want to avoid. I
> chose
> > > the
> > > > name “User Manager” but I think it is simple, the main goal of this
> new
> > > > module is to manage users and access for them.
> > > >
> > > >
> > > How about *AuthManager* ? (Auth standing for both Authentication and
> > > Authorisation). I do agree "Security" is much bigger promise thant what
> > we
> > > are implementing here :).
> > >
> > >
> > That works better in my mind. Note we are not just managing Users here:
> it
> > can be non personal accounts etc.
> >
> >
> > >
> > >
> > > > > is_authorized should include context so that it becomes possible
> the
> > > > determine whether a user has access to a resource based on for
> example
> > > the
> > > > location the user is at. Or in which environment. It is also useful
> for
> > > > auditing. Non implementing security managers can ignore it.
> > > >
> > > > I agree, that can be indeed a good improvement. Although I have to
> say
> > I
> > > > am pretty confident the AIPs I defined in the API will change while
> > > > implementing them and over time. Thus, another option would be to add
> > > this
> > > > context when needed. I do not think (I might be wrong) features such
> as
> > > "
> > > > determine whether a user has access to a resource based on for
> example
> > > the
> > > > location the user is at. Or in which environment" currently exist in
> > > > Airflow. The goal of the AIP is not to implement new features but to
> > > have
> > > > the same features as we have today with this separation from core
> > Airflow
> > > > to providers. Again, I am not saying this is a wrong idea (and I
> > actually
> > > > think this is a good one), I just think if this feature does not
> exist
> > in
> > > > Airflow today, we should add that later.
> > > >
> > >
> > > I think Bolke is right: we should add information that we will pass
> > context
> > > and broadly what will be in. We can leave details for the
> implementation
> > > time but I think the context should be there. We should add what will
> be
> > > part of the context:
> > >
> > > * action
> > > * dag_id
> > > * task_id
> > > * labels for the dag
> > > * location in the DAG_FOLDER
> > > * connection_id (when we change connection)
> > > * variable-id (when we change variable)
> > > * ...
> > >
> > >
> > > This should be complete enough to make any kind of decisions. Probably
> it
> > > will also evolve over time in the future versions, but we need to have
> > > something to start with that we think will be comprehensive and
> complete
> > > enough to implement usable AuthManager(s) (providing that the name will
> > > stick).
> > >
> > >
> > I think this should be extensible (= flexible). Because, I would add
> > remote_addr, proxy_ips, roles, tag etc.
> >
> > For context, I would like to be able to setup policies like this:
> >
> >
> https://towardsdatascience.com/integrating-trino-and-apache-ranger-b808f6b96ad8 <https://towardsdatascience.com/integrating-trino-and-apache-ranger-b808f6b96ad8> <https://towardsdatascience.com/integrating-trino-and-apache-ranger-b808f6b96ad8> <https://towardsdatascience.com/integrating-trino-and-apache-ranger-b808f6b96ad8>>
> >
> > Or in open policy agent:
> >
> > # Financial department staff can view any customer dags
> >
> > allow = true {
> > input.method = "GET"
> > input.path = ["payments", customer_id]
> > finance[input.user]}
> >
> > finance = {"john","mary","peter","vivian"}
> >
> >
> >
> > >
> > > > > What is the need of post_login? Why would you want to store the
> > access
> > > > token? The security manager should, imho, do this on its own: it
> > executes
> > > > the login flow.
> > > >
> > > > The access token is needed to call any API from the user manager to
> the
> > > > third party backend (KeyCloak, ...). How would you call
> is_authorized()
> > > > without it?
> > > >
> > >
> > > Yeah. we need to map whatever authentication information comes from
> > outside
> > > (usually a token coming from external authentication information) with
> > > (likely?) flask session in Airflow and I **think** the best is that it
> > > should be kept in AIrflow's DB the session in Airflow. I consider this
> a
> > > bit of an implementation detail, but maybe this needs a bit more
> hashing
> > ou
> > > maybe others can also comment here..
> > >
> > >
> > Particularly with security, I'd prefer clarity of how we intend to
> > implement this. If we consider an access_token equivalent to an OIDC
> > access_token that is fine, but let us be clear on what we base our design
> > on instead of seemingly inventing our own.
> >
> > At this moment, *any* DAG has full access to the database. That means if
> we
> > store user_id -> access_token any DAG can access any resource on behalf
> of
> > someone else. This means the AIrflow DB is one of the most insecure
> places
> > to store this kind of information.
> >
> > post_login is not required if authentication is entirely handled by the
> > AuthManager. If we consider post_login to be equivalent to a callback
> > method from OIDC (again: on what do we base our security design?) then
> sure
> > but then let's call it as such.
> >
> > Overall: I strongly suggest basing our security design on something
> > existing. For AuthN I like the OIDC scheme. Furthermore, I suggest
> looking
> > at https://flask-login.readthedocs.io/en/latest/ <https://flask-login.readthedocs.io/en/latest/> <https://flask-login.readthedocs.io/en/latest/> <https://flask-login.readthedocs.io/en/latest/>> to see how this is
> > handled
> > in Flask.
> >
> >
> > > >
> > > > > Why is get_tab_configuration special? We have this possibility in
> > > > standard “plugins”. I suggest using that framework
> > > >
> > > > I am not familiar with plugins. I'll take a look at it, thanks for
> > > > bringing it up.
> > > >
> > >
> > > Yeah. The plugin mechanism is indeed foreseen for that - and the
> minimal
> > > FAB implementation should indeed implement plugin interface to add the
> > > current Admin menu - that was what I had in mind by the current
> > "Security"
> > > tab - it can **just** be added using the current plugin mechanism.
> > >
> > > >
> > > > > What is the need for “get_user_name”? Are we going to invent our
> own
> > > > Framework? Otherwise Flask might work?
> > > >
> > > > get_user_name is needed for the UI. On the top right corner of
> Airflow
> > > > console, you can see your name. This is why it is needed. The only
> goal
> > > of
> > > > this API is to retrieve your user name from the user manager. That's
> > it.
> > > > Flask will then use this API to get the user name and display it in
> the
> > > UI.
> > > > My experience in Flask is pretty limited so I might miss something
> but
> > > the
> > > > way I see it is, you need a way to retrieve your user name from
> > whatever
> > > > user manager you are using. This is the way.
> > >
> > >
> > Please have a look at flask login (
> > https://flask-login.readthedocs.io/en/latest/ <https://flask-login.readthedocs.io/en/latest/> <https://flask-login.readthedocs.io/en/latest/> <https://flask-login.readthedocs.io/en/latest/>>) or even base your design
> > upon Flask-Login. This ensures that the right information is available
> > across all components in a way that Flask components understand. The
> > question then becomes how does the DAG subsystem (which does not rely on
> > Flask) deal with this. It might need a kind of wrapper possibly.
> >
> >
> >
> > >
> > > >
> > > > I hope that helps,
> > > > Vincent
> > > >
> > > > On 2023-05-15, 5:30 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>
> <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>> wrote:
> > > >
> > > >
> > > >
> > > > I think think the name “User Manager” is a misnomer and it should
> just
> > be
> > > > called “Security Manager”. It was confusing the hell out of me when
> > > reading
> > > > the AIP and I was convinced you wanted to fully manage e.g. keycloak
> > > > provided users inside Airflow (CRUD). What you are doing but aren’t
> > > exactly
> > > > saying:
> > > >
> > > >
> > > > 1 - Define API that a Security Plugin should adhere to (AuthZ/AuthN)
> > > > 2 - Have a default Security Plugin based on FAB
> > > >
> > > >
> > > > Points of improvement:
> > > >
> > > >
> > > > * is_authorized should include context so that it becomes possible
> the
> > > > determine whether a user has access to a resource based on for
> example
> > > the
> > > > location the user is at. Or in which environment. It is also useful
> for
> > > > auditing. Non implementing security managers can ignore it.
> > > > * I’d prefer some kind of ’standards’ based API. Like using Flask’s
> way
> > > of
> > > > registering endpoints or something along those lines. See also
> > questions
> > > > below.
> > > >
> > > >
> > > > Questions:
> > > >
> > > >
> > > > * What is the need of post_login? Why would you want to store the
> > access
> > > > token? The security manager should, imho, do this on its own: it
> > executes
> > > > the login flow.
> > > > * Why is get_tab_configuration special? We have this possibility in
> > > > standard “plugins”. I suggest using that framework
> > > > * What is the need for “get_user_name”? Are we going to invent our
> own
> > > > Framework? Otherwise Flask might work?
> > > >
> > > >
> > > >
> > > >
> > > > Cheers
> > > > Bolke
> > > >
> > > >
> > > > On 9 May 2023 at 19:55:32, Beck, Vincent (vincbeck@amazon.com.inva <ma...@amazon.com.inva> <mailto:vincbeck@amazon.com.inva <ma...@amazon.com.inva>>
> > > > <mailto:vincbeck@amazon.com.inva <ma...@amazon.com.inva> <mailto:vincbeck@amazon.com.inva <ma...@amazon.com.inva>>>lid)
> > > > wrote:
> > > >
> > > >
> > > > Hello all,
> > > >
> > > >
> > > > I would like to start voting on this one so please add your comments
> on
> > > the
> > > > API or by replying here if you're interested, you still have time to
> do
> > > it
> > > > :)
> > > >
> > > >
> > > > AIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > >
> > > >
> > > >
> > > > Vincent
> > > >
> > > >
> > > > On 2023-05-03, 8:03 AM, "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>>> wrote:
> > > >
> > > >
> > > >
> > > >
> > > > CAUTION: This email originated from outside of the organization. Do
> not
> > > > click links or open attachments unless you can confirm the sender and
> > > know
> > > > the content is safe.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Alsoo to add (and make more concrete) to what it could mean at the
> > > > implementation level.
> > > >
> > > >
> > > >
> > > >
> > > > It means the for our API endpoints and views is that instead of
> > > > @auth.hasaccess (for example here
> > > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680 <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>>
> > <
> > > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680&gt;>>
> > <
> > > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680&gt;>>
> > <
> > > >
> > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680&gt> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680&gt> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680&amp;gt>>
> > > > ;>)
> > > > where it check for permissions assigned to current role, airflow
> would
> > > > entirely delegate the decision to the 'user management plugin'.
> > > >
> > > >
> > > >
> > > >
> > > > We would simply delegate the decision to whatever plugin we have
> > > > configured. In case FAB pługin is used, it would do exactly the same
> > > checks
> > > > as today (and FAB pługin would contribute the 'Security' menu the
> same
> > > way
> > > > other plugins can contribute their own (as well as CLI commands and
> > > > user/role APIs - we would have to add a mechanism where plugins could
> > > > contribute CLIs and APIs on their own, but that should be easy). So
> if
> > > you
> > > > have FAB pługin chosen as default, nothing changes for the users
> > > comparing
> > > > to today. It would not require bumping Airflow Major version because
> it
> > > > would be backwards compatible. However similarly as providers it
> would
> > > move
> > > > the code of FAB plugins and roles out of the core of Airflow.
> > > >
> > > >
> > > >
> > > >
> > > > We could have keycloak 'reference' implementation where we would come
> > up
> > > > with very opinionated approach where we have simple admin/user roles
> > > > (matching roughly default set of roles and permissions we currently
> > have
> > > > with FAB). The big difference here will be that the user/role APIs,
> > > views,
> > > > CLIs will be gone entirely. And we could add there the concept of
> > > > 'tenants' or rather 'teams' giving access to 'dag groups' and
> implement
> > > > decisions ob authorisation based on 'group' assignment for dag and
> > user.
> > > > This is the main goal of this change to make it possible and easy
> > without
> > > > having to mess with FAB permission and Role model which is completely
> > not
> > > > suitable to manage 'groups of resources of the same type' with
> separate
> > > set
> > > > of permissions (think having few separate teams or tenants - each of
> > them
> > > > wanting access to separate set of DAGs).
> > > >
> > > >
> > > >
> > > >
> > > > Currently if an SSO/enterprise user wants to plug in their
> authn/authz
> > > with
> > > > group access - they have to deal with all that including cumbersome
> > > syncing
> > > > of permissions for individual DAGs matching them to users separately
> -
> > > just
> > > > to satisfy FAB permission model lacking this feature and flexibility.
> > > >
> > > >
> > > >
> > > >
> > > > No more need for that if we go the direction described here.
> > > >
> > > >
> > > >
> > > >
> > > > And that's basically it for what we would implement in Airflow
> > > > out-of-the-box: FAB minimal implementation + be keycloak opinionated
> > > > reference implementation with simple teams/tenant
> <https://teams.googleplex.com/u/tenant> <https://teams.googleplex.com/u/tenant>> <https://teams.googleplex.com/u/tenant>> <https://teams.googleplex.com/u/tenant&gt;>> support
> > > >
> > > >
> > > >
> > > >
> > > > This also makes it super flexible for those who want more flexibility
> > and
> > > > do not want to use our opinionated keycloak plugin. They will be
> > entirely
> > > > free to implement their own ways - as flexible or as opinionated they
> > > want
> > > > - without having to messwith permission syncing and the role model of
> > > FAB.
> > > > Nor keycloak integration. They could go with their own.
> > > >
> > > >
> > > >
> > > >
> > > > And with that - sky is the limit. The decision to access particular
> dag
> > > > could be made based on folder where the dag is from, id of dag, or
> dag
> > > > label or whatever other parameter (as long as we pass all that
> > > information
> > > > to the plugin). You could even - if you really want - make a decision
> > to
> > > > allow certain users (and only them) to be able to clear individual
> > tasks,
> > > > or tasks where specific operator types are used. For example you
> could
> > > > decide if you really want to only allow th google cloud admin users
> to
> > > > clear tasks that are operators coming from Google Provider (why
> not?).
> > Or
> > > > only allow to that in certain time windows, or .... whatever.
> > > >
> > > >
> > > >
> > > >
> > > > It's extremely flexible and powerful - but we will not have to deal
> > with
> > > > all the complexity in the community - we could delegate it to those
> who
> > > > mange the deployment or manage deployment of Airflow as a service and
> > > > decide they want to integrate it with the service authorisation that
> is
> > > > already there - think direct integratio with IAM ON AWS or GCP. And
> if
> > we
> > > > just provide a reference, opinionated implementation for keycloak
> with
> > > > tenant support and legacy/minimal FAB as optional plugins, that
> should
> > > also
> > > > 'good enough' for those who do not want to implement their own plugin
> > > > implementation.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > > <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>>> napisał:
> > > >
> > > >
> > > >
> > > >
> > > > > > I don't think we can do that: we still need to consider users
> when
> > > they
> > > > > are just getting started out with Airflow -- if we only cater to
> > giant
> > > > > enterprises with SSO and a central directory we remove our ability
> > for
> > > > > individual or small teams/companies
> <https://teams.googleplex.com/u/companies> <https://teams.googleplex.com/u/companies>> <https://teams.googleplex.com/u/companies>> <https://teams.googleplex.com/u/companies&gt;>> to get started with Airflow.
> > > > >
> > > > > Let me clarify this because I think we are talking about the same
> > > thing.
> > > > >
> > > > > I **think** having the "minimal FAB user manager implementation" as
> > the
> > > > > "default" option when you install Airflow serves the purpose you
> > > mention
> > > > > Ash - if we do it the way that User/Role management is coming
> > together
> > > > with
> > > > > FAB implementation (enabled by default for the initial
> installation).
> > > > What
> > > > > I am saying is that it should not be part of the "core airflow" in
> > the
> > > > > sense that one could live without it (specifically in those big
> > > > SSO/Central
> > > > > directory case).
> > > > > In my vision those two cases are separated, but the "minimal
> > > > > implementation" when enabled should act and behave "as if" nothing
> > > > changed
> > > > > compared to the current way it looks and behaves.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <ma...@apache.org>>
> > > > <mailto:ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <ma...@apache.org>>> <mailto:
> > > > ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <ma...@apache.org>> <mailto:ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > >> > - I am fine with (and actually I like it better) removing User
> and
> > > > Role
> > > > >> related Rest APIs and CLI commands from Airflow. That will indeed
> > make
> > > > the
> > > > >> AIP way simpler and shorter
> > > > >>
> > > > >> I don't think we can do that: we still need to consider users when
> > > they
> > > > >> are just getting started out with Airflow -- if we only cater to
> > giant
> > > > >> enterprises with SSO and a central directory we remove our ability
> > for
> > > > >> individual or small teams/companies
> <https://teams.googleplex.com/u/companies> <https://teams.googleplex.com/u/companies>> <https://teams.googleplex.com/u/companies>> <https://teams.googleplex.com/u/companies&gt;>> to get started with Airflow.
> > > > >> On May 2 2023, at 8:23 pm, Beck, Vincent <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > >>LID>
> > > > >> wrote:
> > > > >> > Thank you Jarek and Bolke for taking time to read and provide
> > > feedback
> > > > >> to the AIP, much appreciated! Multiple points here:
> > > > >> >
> > > > >> > - I am fine with (and actually I like it better) removing User
> and
> > > > Role
> > > > >> related Rest APIs and CLI commands from Airflow. That will indeed
> > make
> > > > the
> > > > >> AIP way simpler and shorter. As I explained in the AIP comments, I
> > am
> > > > >> suggesting though to leave an option to user managers to define
> > > > additional
> > > > >> REST APIs and CLI commands if they need. By default no additional
> > REST
> > > > API
> > > > >> and CLI command would be defined but user managers would have the
> > > option
> > > > to
> > > > >> override it. FAB user manager would be an example of user manager
> > > > >> overriding such methods. In any case, if additional Rest API or
> CLI
> > > > command
> > > > >> is defined, this would be done in user managers (as opposed to
> core
> > > > >> Airflow).
> > > > >> > - "the whole "Security" menu should be GONE completely".
> > > > >> > > I agree with this when the user manager is not the FAB one,
> > > however
> > > > >> we need to introduce a new one so Admins can easily go the
> external
> > > user
> > > > >> manager console (e.g. KeyCloak, ...). This new tab would only be a
> > > > >> link/redirect to the external console.
> > > > >> >
> > > > >> > - "All those components (and code related to those) should be
> > moved
> > > to
> > > > >> FAB minimalistic user manager (and removed from Airflow Core)".
> > > > >> > > 100%. This is the main principle of this AIP, anything related
> > to
> > > > >> user management will be moved to user managers.
> > > > >> >
> > > > >> > - "AuthN and AuthZ should be completely removed from Airflow".
> > > > >> > > 100%.
> > > > >> >
> > > > >> > - "So for a nice out-of-the-box experience (pip install
> > > > apache-airflow)
> > > > >> a lightweight User Manager could exist as a side project and have
> > some
> > > > >> menus via the plugin system until something better comes along.".
> > > > >> > > I agree with you on the long term. Though, as a first
> iteration
> > I
> > > > >> think it is better to define a user manager using FAB as default
> > user
> > > > >> manager. The benefit of doing so is to have a backward compatible
> > > > >> transition between before AIP and after AIP. This also simplifies
> > the
> > > > AIP
> > > > >> since it is easier to move files from core Airflow to user
> managers
> > > than
> > > > >> creating something new. Creating a new security model would be a
> > > > challenge
> > > > >> too. However, once the AIP implemented and merged, we can easily
> > > replace
> > > > >> this out-of-the-box FAB user manager by something "better" if the
> > > > communty
> > > > >> thinks this is the right way to go.
> > > > >> >
> > > > >> > As always, feel free to give me your thoughts.
> > > > >> > Vincent
> > > > >> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>
> > > <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>> <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>> <mailto:
> > > > >> bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>> <mailto:
> > > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>
> > > > <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>>>> wrote:
> > > > >> > I'm not sure if I read the AIP correctly. I think I concur with
> > > Jarek
> > > > >> here.
> > > > >> >
> > > > >> > AuthN and AuthZ should be completely removed from Airflow.
> Access
> > > > >> > management should be outsourced to something like Open Policy
> > Agent.
> > > > In
> > > > >> > other words there should be no user management at all within
> > > Airflow.
> > > > >> This
> > > > >> > removes the need to "sync" users (and thus become outdated) and
> > > > reduces
> > > > >> the
> > > > >> > attack surface drastically. So for a nice out-of-the-box
> > experience
> > > > (pip
> > > > >> > install apache-airflow) a lightweight User Manager could exist
> as
> > a
> > > > side
> > > > >> > project and have some menus via the plugin system until
> something
> > > > better
> > > > >> > comes along. Keycloak is great, a bit heavy for us to package
> > > > >> > unfortunately.
> > > > >> >
> > > > >> >
> > > > >> > FAB's security model is very non-standard and requires
> 'unnatural'
> > > > >> mappings
> > > > >> > in an enterprise context. It has served us well, but I would say
> > > good
> > > > >> > riddance.
> > > > >> >
> > > > >> >
> > > > >> > Bolke
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <
> jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>>
> > > > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>>>>:
> > > > >> >
> > > > >> > > I had a look and I liked what I saw for how we should
> integrate
> > > the
> > > > >> new
> > > > >> > > user manager for getting and checking the access :).
> > > > >> > >
> > > > >> > > But I think the AIP proposal is not bold enough when it comes
> to
> > > the
> > > > >> user
> > > > >> > > management part. This is one big comment - we should simplify
> it
> > > > even
> > > > >> > > further and cut more deeply.
> > > > >> > >
> > > > >> > > One of the big changes I think is super important with
> > extensible
> > > > user
> > > > >> > > management is that we should delegate even more to the
> external
> > > > >> systems
> > > > >> > > managing users. The AIP attempts to map the current API/CLI/UI
> > for
> > > > >> managing
> > > > >> > > the users and roles to the new external user management
> > services -
> > > > >> but it
> > > > >> > > is IMHO absolutely not needed.
> > > > >> > >
> > > > >> > > As I see the extensible user management is that when you
> choose
> > > > >> something
> > > > >> > > else than the legacy FAB minimalist user management. Airflow
> > > should
> > > > >> become
> > > > >> > > just "user" of that user/role information managed elsewhere.
> It
> > > > >> should not
> > > > >> > > provide ANY option to see and manage the users or roles, it
> > should
> > > > >> merely
> > > > >> > > read the information and give access to the users for
> > appropriate
> > > > >> actions,
> > > > >> > > but then all the role/user management should happen outside of
> > > > >> Airflow - in
> > > > >> > > the system that is dedicated to manage those.
> > > > >> > >
> > > > >> > > IMHO - when other than the "non FAB minimalistic user manager"
> > > > system
> > > > >> is
> > > > >> > > used. Airflow should just not have a number of functionalities
> > at
> > > > all:
> > > > >> > >
> > > > >> > > * not have "users" or "roles" CLI groups at all - they should
> be
> > > > >> missing
> > > > >> > > * the
> > > > >> > >
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>>
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>>
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User&gt> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User&gt> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User&amp;gt>>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>>
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>>
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User&gt> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User&gt> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User&amp;gt>>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > and
> > > > >> > >
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>>
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>>
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role&gt> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role&gt> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role&amp;gt>>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>>
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>>
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role&gt> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role&gt> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role&amp;gt>>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > should return 404 NOT FOUND
> > > > >> > > * the whole "Security" menu should be GONE completely.
> > > > >> > >
> > > > >> > > All those components (and code related to those) should be
> moved
> > > to
> > > > >> FAB
> > > > >> > > minimalistic user manager (and removed from Airflow Core).
> > > > Eventually
> > > > >> maybe
> > > > >> > > even moved out of the Airflow repo altogether and moved to a
> FAB
> > > > user
> > > > >> > > manager add-on to airflow.
> > > > >> > >
> > > > >> > > This might seem a drastic change, but in fact - it's not
> drastic
> > > at
> > > > >> all. It
> > > > >> > > reflects what many of the Airflow users are doing now - they
> > never
> > > > >> access
> > > > >> > > Security/Admin/Roles. Once they integrate with their LDAP or
> > other
> > > > >> > > corporate directory, the Security UI, CLI and REST API are
> > > > essentially
> > > > >> > > useless (and not used).
> > > > >> > >
> > > > >> > > And it is precisely what we want to achieve - we want to
> > delegate
> > > > the
> > > > >> > > user/role management completely out of Airflow. In fact -
> > getting
> > > > rid
> > > > >> of
> > > > >> > > those is the only reason we want to implement the change. If
> we
> > > > >> **just**
> > > > >> > > replace the current FAB and keep all the complexity of
> managing
> > > the
> > > > >> users
> > > > >> > > /roles in Airflow, I'd say we have not achieved anything but
> > > adding
> > > > >> > > complexity.
> > > > >> > >
> > > > >> > > And - eventually - it makes the AIP-56 way *smaller*. We just
> > need
> > > > to
> > > > >> make
> > > > >> > > sure that all the stuff for user management (including REST
> API,
> > > CLI
> > > > >> and
> > > > >> > > the UI) is moved to the "FAB minimalistic user manager" (and
> add
> > > > ways
> > > > >> to
> > > > >> > > plug-it in the core - possibly using existing plugins
> mechanisms
> > > > where
> > > > >> > > needed). We do not need to define new API for user/role
> > > management.
> > > > >> We only
> > > > >> > > need to describe the way how to check if the user has
> permission
> > > to
> > > > do
> > > > >> > > stuff.
> > > > >> > >
> > > > >> > > J.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <
> jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>>
> > > > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>>>> wrote:
> > > > >> > >
> > > > >> > > > I promise to provide my feedback by the end of the week,
> sorry
> > > for
> > > > >> > > > delaying it, but we had some 2.6 preparation, branching,
> > feature
> > > > >> flags
> > > > >> > > etc.
> > > > >> > > > also for AIP-44 and I am getting back on track with that one
> > > soon.
> > > > >> > > >
> > > > >> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
> > > > >> <shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>>> <mailto:
> > > > shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>>>> <mailto:
> > > > shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>>> <mailto:
> > > > shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>>>>>lid
> > > > >> > > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > >> Bumping this up for feedback!
> > > > >> > > >>
> > > > >> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite
> > busy,
> > > > but
> > > > >> we
> > > > >> > > >> would greatly appreciate your feedback here. Your inputs
> > could
> > > > >> help us
> > > > >> > > >> improve the proposal and move forward with multi-tenancy
> > work.
> > > > >> > > >>
> > > > >> > > >> Cheers,
> > > > >> > > >> Shubham
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent"
> > > <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>>
> > > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> >
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>>>
> > > > >> > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>> <mailto:
> > > > vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>> <mailto:
> > > > vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > > >>>>LID>
> > > > >> wrote:
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> CAUTION: This email originated from outside of the
> > > organization.
> > > > >> Do not
> > > > >> > > >> click links or open attachments unless you can confirm the
> > > sender
> > > > >> and
> > > > >> > > know
> > > > >> > > >> the content is safe.
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> Hi all,
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> I would like to get your feedback on an AIP I wrote:
> > > "Extensible
> > > > >> user
> > > > >> > > >> management". This AIP is a follow-up on a discussion we had
> > in
> > > > this
> > > > >> > > email
> > > > >> > > >> list on the multi tenancy topic. I decided to create a new
> > > email
> > > > >> thread
> > > > >> > > >> because I feel the topic had diverged a bit from the
> original
> > > > topic
> > > > >> > > (multi
> > > > >> > > >> tenancy).
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> As a summary, the purpose of this AIP is to extract the
> user
> > > > >> management
> > > > >> > > >> from core Airflow by introducing a user manager interface
> in
> > > the
> > > > >> core
> > > > >> > > >> Airflow that can be extended by any provider package who
> want
> > > to
> > > > >> support
> > > > >> > > >> user management natively. As a result, if this AIP gets
> > > approved
> > > > >> and go
> > > > >> > > >> through, the multi tenancy feature will be implemented as a
> > > > second
> > > > >> step
> > > > >> > > in
> > > > >> > > >> a new user manager and not in Airflow directly.
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> As always, feel very free to give your opinion on this
> email
> > > > >> thread or
> > > > >> > > on
> > > > >> > > >> the AIP by adding comments.
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> References:
> > > > >> > > >> - AIP:
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&amp;gt>>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&amp;gt>>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > >> <
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&amp;gt>>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&amp;gt>>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > >> >
> > > > >> > > >> - Initial email list discussion:
> > > > >> > > >>
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> > > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>> <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>> <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt>>
> ;>
> > <
> > > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>>
> <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt>>
> ;>
> > <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt>>
> ;>
> > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;amp;gt;&amp;gt>>
> > > ;>
> > > > <
> > > > >> > > >>
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>>
> > > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt>>
> ;>
> > > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt>>
> ;>
> > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;amp;gt;&amp;gt>>
> > > > ;>
> > > > >> <
> > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt>>
> > > ;>
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;amp;gt;&amp;gt>>
> > > ;>
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;amp;gt;&amp;gt>>
> > > ;>
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;amp;gt;&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;amp;gt;&amp;gt;&gt> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;amp;amp;gt;&amp;amp;gt;&amp;gt>>
> > > > ;>
> > > >
> > > >
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> Vincent
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> >
> > > > >> > --
> > > > >> > Bolke de Bruin
> > > > >> > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>> <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>> <mailto:
> > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>
> > > > <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>
> > > > <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>>>
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > ---------------------------------------------------------------------
> > > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>>
> > <mailto:
> > > > dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>>> <mailto:
> > > > dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > > > dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>>>>
> > > > >> > For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>>
> > > <mailto:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>>> <mailto:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>>>>
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > > > dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>>>
> > > > For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>>
> <mailto:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>>>
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> > --
> >
> > --
> > Bolke de Bruin
> > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: AIP-56 Extensible user management
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
Somehow, my last email does not show up on the Apache list web view (https://lists.apache.org/thread/ck8dsj5w82lvr0cpwr4wlptmydqwnsqc). Re-sending it to hopefully add it there. Sorry for the noise.
Hello,
I tried to answer to all open questions:
> If I understand correctly, the proposition in AIP-56 is to introduce an interface as generic as possible so that any potential federated identity protocol can be integrated in Airflow through User Manager (or as proposed in this thread, AuthManager)
You're 100% correct. This is the ultimate goal. Note on this one, based on the last feedbacks, it seems Auth manager makes more sense than User manager, which I agree. Thus, I updated the AIP to replace user manager by auth manager.
> What do you think would be the context of "current user" provided by Airflow to the implementation of AuthManager?
When a user is logged in Airflow (through an auth manager), a Flask session will be set as it is today. This will be the context of "current user". In this session, you can find user information such as user ID or OIDC token (if the auth manager used use that kind of token). This is the way I envisioned so that the auth manager can retrieve the OIDC token in order to interact with the auth manager backend. For now we know that the FAB auth manager will need the user ID as context and the KeyCloak auth manager will need a OIDC token. However, this set of information must be extensible so that new auth manager which do not use OIDC token can be added in Airflow in the future.
> Do you think it makes sense to document this in the AIP or leave decisions for later?
I think it makes total sense to add it to the AIP, which I will do, I hope it will make it clearer :)
> is_authorized should include context so that it becomes possible the determine whether a user has access to a resource based on for example the location the user is at. Or in which environment. It is also useful for auditing
I see your points. I agree then, we should come up with a context of the is_authorized API. I have not started to work on that yet but will do in the following days. If you also want to give it a shot and come up with something, that would be very much appreciated :)
> Particularly with security, I'd prefer clarity of how we intend to implement this. If we consider an access_token equivalent to an OIDC access_token that is fine, but let us be clear on what we base our design on instead of seemingly inventing our own.
I agree with you. Every time I refer to access token in the AIP, I meant OIDC access token, I'll add that clarification, thanks for the callout. However, this is very specific to each auth manager backend. In the case of KeyCloak, the access token I referred to is indeed an OIDC access token but it is impossible to assert this will be true for all auth managers implemented in the future. We need to keep in mind we cannot enforce third party auth services to implement the protocol we want but we need to be flexible/adaptable enough so we can integrate with majority of them.
> At this moment, *any* DAG has full access to the database. That means if we store user_id -> access_token any DAG can access any resource on behalf of someone else. This means the AIrflow DB is one of the most insecure places to store this kind of information.
That is correct. Though this will be fixed in the future with AIP-44 (or a following on this one) if I am not mistaken.
> post_login is not required if authentication is entirely handled by the AuthManager. If we consider post_login to be equivalent to a callback method from OIDC (again: on what do we base our security design?) then sure but then let's call it as such.
This is indeed a callback method from OIDC. Again, I did not want to be OIDC specific because we cannot assert all auth managers in the future will use OIDC protocol. That's why I came up with the name "post_login". I wanted something generic. Though, I am happy to change the name if that makes things easier to understand.
As usual, thanks for your feedbacks!
Vincent
On 2023-05-19, 11:35 AM, "Jakub Kośmider" <kosmider@google.com.INVA <ma...@google.com.INVA>LID> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hello,
If I understand correctly, the proposition in AIP-56 is to introduce an
interface as generic as possible so that any potential federated identity
protocol can be integrated in Airflow through User Manager (or as proposed
in this thread, AuthManager). In light of this, note that the methods in
AuthManager in the AIP refer to the "current user", while the example of
"is_authorized" used above is based only on specific resources and
permissions provided as method parameters.
What do you think would be the context of "current user" provided by
Airflow to the implementation of AuthManager?
Would it be the entire request or maybe just cookies/auth tokens associated
with the request or something else?
Do you think it makes sense to document this in the AIP or leave decisions
for later?
Best regards,
Jakub
On Fri, May 19, 2023 at 11:22 AM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
> > Overall: I strongly suggest basing our security design on something
> existing. For AuthN I like the OIDC scheme. Furthermore, I suggest looking
> at https://flask-login.readthedocs.io/en/latest/ <https://flask-login.readthedocs.io/en/latest/> to see how this is
> handled
> in Flask.
>
> Very much agree. Good suggestion Bolke.
>
>
> On Fri, May 19, 2023 at 9:44 AM Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>
> > Hi,
> >
> > On Wed, 17 May 2023 at 18:48, Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
> >
> > > >
> > > >
> > > > I am very open to change the name of "User Manager" but I do not
> think
> > we
> > > > should call it "Security Manager". "Security Manager" is the current
> > name
> > > > and I think this is a very bad name because: it is very generic and
> > does
> > > > not say much about what it is actually doing. By reading "Security
> > > Manager"
> > > > I would not think this is where users and roles are managed. The main
> > > goal
> > > > of AIP-56 is removing the whole user management part from core
> Airflow
> > to
> > > > providers. Keeping the name "Security Manager" would, I think, leave
> a
> > > > legacy for all future user managers (or whatever name we use) on
> > > > Flask-AppBuilder (FAB) and that's exactly what I want to avoid. I
> chose
> > > the
> > > > name “User Manager” but I think it is simple, the main goal of this
> new
> > > > module is to manage users and access for them.
> > > >
> > > >
> > > How about *AuthManager* ? (Auth standing for both Authentication and
> > > Authorisation). I do agree "Security" is much bigger promise thant what
> > we
> > > are implementing here :).
> > >
> > >
> > That works better in my mind. Note we are not just managing Users here:
> it
> > can be non personal accounts etc.
> >
> >
> > >
> > >
> > > > > is_authorized should include context so that it becomes possible
> the
> > > > determine whether a user has access to a resource based on for
> example
> > > the
> > > > location the user is at. Or in which environment. It is also useful
> for
> > > > auditing. Non implementing security managers can ignore it.
> > > >
> > > > I agree, that can be indeed a good improvement. Although I have to
> say
> > I
> > > > am pretty confident the AIPs I defined in the API will change while
> > > > implementing them and over time. Thus, another option would be to add
> > > this
> > > > context when needed. I do not think (I might be wrong) features such
> as
> > > "
> > > > determine whether a user has access to a resource based on for
> example
> > > the
> > > > location the user is at. Or in which environment" currently exist in
> > > > Airflow. The goal of the AIP is not to implement new features but to
> > > have
> > > > the same features as we have today with this separation from core
> > Airflow
> > > > to providers. Again, I am not saying this is a wrong idea (and I
> > actually
> > > > think this is a good one), I just think if this feature does not
> exist
> > in
> > > > Airflow today, we should add that later.
> > > >
> > >
> > > I think Bolke is right: we should add information that we will pass
> > context
> > > and broadly what will be in. We can leave details for the
> implementation
> > > time but I think the context should be there. We should add what will
> be
> > > part of the context:
> > >
> > > * action
> > > * dag_id
> > > * task_id
> > > * labels for the dag
> > > * location in the DAG_FOLDER
> > > * connection_id (when we change connection)
> > > * variable-id (when we change variable)
> > > * ...
> > >
> > >
> > > This should be complete enough to make any kind of decisions. Probably
> it
> > > will also evolve over time in the future versions, but we need to have
> > > something to start with that we think will be comprehensive and
> complete
> > > enough to implement usable AuthManager(s) (providing that the name will
> > > stick).
> > >
> > >
> > I think this should be extensible (= flexible). Because, I would add
> > remote_addr, proxy_ips, roles, tag etc.
> >
> > For context, I would like to be able to setup policies like this:
> >
> >
> https://towardsdatascience.com/integrating-trino-and-apache-ranger-b808f6b96ad8 <https://towardsdatascience.com/integrating-trino-and-apache-ranger-b808f6b96ad8>
> >
> > Or in open policy agent:
> >
> > # Financial department staff can view any customer dags
> >
> > allow = true {
> > input.method = "GET"
> > input.path = ["payments", customer_id]
> > finance[input.user]}
> >
> > finance = {"john","mary","peter","vivian"}
> >
> >
> >
> > >
> > > > > What is the need of post_login? Why would you want to store the
> > access
> > > > token? The security manager should, imho, do this on its own: it
> > executes
> > > > the login flow.
> > > >
> > > > The access token is needed to call any API from the user manager to
> the
> > > > third party backend (KeyCloak, ...). How would you call
> is_authorized()
> > > > without it?
> > > >
> > >
> > > Yeah. we need to map whatever authentication information comes from
> > outside
> > > (usually a token coming from external authentication information) with
> > > (likely?) flask session in Airflow and I **think** the best is that it
> > > should be kept in AIrflow's DB the session in Airflow. I consider this
> a
> > > bit of an implementation detail, but maybe this needs a bit more
> hashing
> > ou
> > > maybe others can also comment here..
> > >
> > >
> > Particularly with security, I'd prefer clarity of how we intend to
> > implement this. If we consider an access_token equivalent to an OIDC
> > access_token that is fine, but let us be clear on what we base our design
> > on instead of seemingly inventing our own.
> >
> > At this moment, *any* DAG has full access to the database. That means if
> we
> > store user_id -> access_token any DAG can access any resource on behalf
> of
> > someone else. This means the AIrflow DB is one of the most insecure
> places
> > to store this kind of information.
> >
> > post_login is not required if authentication is entirely handled by the
> > AuthManager. If we consider post_login to be equivalent to a callback
> > method from OIDC (again: on what do we base our security design?) then
> sure
> > but then let's call it as such.
> >
> > Overall: I strongly suggest basing our security design on something
> > existing. For AuthN I like the OIDC scheme. Furthermore, I suggest
> looking
> > at https://flask-login.readthedocs.io/en/latest/ <https://flask-login.readthedocs.io/en/latest/> to see how this is
> > handled
> > in Flask.
> >
> >
> > > >
> > > > > Why is get_tab_configuration special? We have this possibility in
> > > > standard “plugins”. I suggest using that framework
> > > >
> > > > I am not familiar with plugins. I'll take a look at it, thanks for
> > > > bringing it up.
> > > >
> > >
> > > Yeah. The plugin mechanism is indeed foreseen for that - and the
> minimal
> > > FAB implementation should indeed implement plugin interface to add the
> > > current Admin menu - that was what I had in mind by the current
> > "Security"
> > > tab - it can **just** be added using the current plugin mechanism.
> > >
> > > >
> > > > > What is the need for “get_user_name”? Are we going to invent our
> own
> > > > Framework? Otherwise Flask might work?
> > > >
> > > > get_user_name is needed for the UI. On the top right corner of
> Airflow
> > > > console, you can see your name. This is why it is needed. The only
> goal
> > > of
> > > > this API is to retrieve your user name from the user manager. That's
> > it.
> > > > Flask will then use this API to get the user name and display it in
> the
> > > UI.
> > > > My experience in Flask is pretty limited so I might miss something
> but
> > > the
> > > > way I see it is, you need a way to retrieve your user name from
> > whatever
> > > > user manager you are using. This is the way.
> > >
> > >
> > Please have a look at flask login (
> > https://flask-login.readthedocs.io/en/latest/ <https://flask-login.readthedocs.io/en/latest/>) or even base your design
> > upon Flask-Login. This ensures that the right information is available
> > across all components in a way that Flask components understand. The
> > question then becomes how does the DAG subsystem (which does not rely on
> > Flask) deal with this. It might need a kind of wrapper possibly.
> >
> >
> >
> > >
> > > >
> > > > I hope that helps,
> > > > Vincent
> > > >
> > > > On 2023-05-15, 5:30 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com>
> <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com>>> wrote:
> > > >
> > > >
> > > >
> > > > I think think the name “User Manager” is a misnomer and it should
> just
> > be
> > > > called “Security Manager”. It was confusing the hell out of me when
> > > reading
> > > > the AIP and I was convinced you wanted to fully manage e.g. keycloak
> > > > provided users inside Airflow (CRUD). What you are doing but aren’t
> > > exactly
> > > > saying:
> > > >
> > > >
> > > > 1 - Define API that a Security Plugin should adhere to (AuthZ/AuthN)
> > > > 2 - Have a default Security Plugin based on FAB
> > > >
> > > >
> > > > Points of improvement:
> > > >
> > > >
> > > > * is_authorized should include context so that it becomes possible
> the
> > > > determine whether a user has access to a resource based on for
> example
> > > the
> > > > location the user is at. Or in which environment. It is also useful
> for
> > > > auditing. Non implementing security managers can ignore it.
> > > > * I’d prefer some kind of ’standards’ based API. Like using Flask’s
> way
> > > of
> > > > registering endpoints or something along those lines. See also
> > questions
> > > > below.
> > > >
> > > >
> > > > Questions:
> > > >
> > > >
> > > > * What is the need of post_login? Why would you want to store the
> > access
> > > > token? The security manager should, imho, do this on its own: it
> > executes
> > > > the login flow.
> > > > * Why is get_tab_configuration special? We have this possibility in
> > > > standard “plugins”. I suggest using that framework
> > > > * What is the need for “get_user_name”? Are we going to invent our
> own
> > > > Framework? Otherwise Flask might work?
> > > >
> > > >
> > > >
> > > >
> > > > Cheers
> > > > Bolke
> > > >
> > > >
> > > > On 9 May 2023 at 19:55:32, Beck, Vincent (vincbeck@amazon.com.inva <ma...@amazon.com.inva>
> > > > <mailto:vincbeck@amazon.com.inva <ma...@amazon.com.inva>>lid)
> > > > wrote:
> > > >
> > > >
> > > > Hello all,
> > > >
> > > >
> > > > I would like to start voting on this one so please add your comments
> on
> > > the
> > > > API or by replying here if you're interested, you still have time to
> do
> > > it
> > > > :)
> > > >
> > > >
> > > > AIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > >
> > > >
> > > >
> > > > Vincent
> > > >
> > > >
> > > > On 2023-05-03, 8:03 AM, "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com>> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>> wrote:
> > > >
> > > >
> > > >
> > > >
> > > > CAUTION: This email originated from outside of the organization. Do
> not
> > > > click links or open attachments unless you can confirm the sender and
> > > know
> > > > the content is safe.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Alsoo to add (and make more concrete) to what it could mean at the
> > > > implementation level.
> > > >
> > > >
> > > >
> > > >
> > > > It means the for our API endpoints and views is that instead of
> > > > @auth.hasaccess (for example here
> > > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680 <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>
> > <
> > > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>>
> > <
> > > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>>
> > <
> > > >
> > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680&gt>
> > > > ;>)
> > > > where it check for permissions assigned to current role, airflow
> would
> > > > entirely delegate the decision to the 'user management plugin'.
> > > >
> > > >
> > > >
> > > >
> > > > We would simply delegate the decision to whatever plugin we have
> > > > configured. In case FAB pługin is used, it would do exactly the same
> > > checks
> > > > as today (and FAB pługin would contribute the 'Security' menu the
> same
> > > way
> > > > other plugins can contribute their own (as well as CLI commands and
> > > > user/role APIs - we would have to add a mechanism where plugins could
> > > > contribute CLIs and APIs on their own, but that should be easy). So
> if
> > > you
> > > > have FAB pługin chosen as default, nothing changes for the users
> > > comparing
> > > > to today. It would not require bumping Airflow Major version because
> it
> > > > would be backwards compatible. However similarly as providers it
> would
> > > move
> > > > the code of FAB plugins and roles out of the core of Airflow.
> > > >
> > > >
> > > >
> > > >
> > > > We could have keycloak 'reference' implementation where we would come
> > up
> > > > with very opinionated approach where we have simple admin/user roles
> > > > (matching roughly default set of roles and permissions we currently
> > have
> > > > with FAB). The big difference here will be that the user/role APIs,
> > > views,
> > > > CLIs will be gone entirely. And we could add there the concept of
> > > > 'tenants' or rather 'teams' giving access to 'dag groups' and
> implement
> > > > decisions ob authorisation based on 'group' assignment for dag and
> > user.
> > > > This is the main goal of this change to make it possible and easy
> > without
> > > > having to mess with FAB permission and Role model which is completely
> > not
> > > > suitable to manage 'groups of resources of the same type' with
> separate
> > > set
> > > > of permissions (think having few separate teams or tenants - each of
> > them
> > > > wanting access to separate set of DAGs).
> > > >
> > > >
> > > >
> > > >
> > > > Currently if an SSO/enterprise user wants to plug in their
> authn/authz
> > > with
> > > > group access - they have to deal with all that including cumbersome
> > > syncing
> > > > of permissions for individual DAGs matching them to users separately
> -
> > > just
> > > > to satisfy FAB permission model lacking this feature and flexibility.
> > > >
> > > >
> > > >
> > > >
> > > > No more need for that if we go the direction described here.
> > > >
> > > >
> > > >
> > > >
> > > > And that's basically it for what we would implement in Airflow
> > > > out-of-the-box: FAB minimal implementation + be keycloak opinionated
> > > > reference implementation with simple teams/tenant
> <https://teams.googleplex.com/u/tenant> <https://teams.googleplex.com/u/tenant>> support
> > > >
> > > >
> > > >
> > > >
> > > > This also makes it super flexible for those who want more flexibility
> > and
> > > > do not want to use our opinionated keycloak plugin. They will be
> > entirely
> > > > free to implement their own ways - as flexible or as opinionated they
> > > want
> > > > - without having to messwith permission syncing and the role model of
> > > FAB.
> > > > Nor keycloak integration. They could go with their own.
> > > >
> > > >
> > > >
> > > >
> > > > And with that - sky is the limit. The decision to access particular
> dag
> > > > could be made based on folder where the dag is from, id of dag, or
> dag
> > > > label or whatever other parameter (as long as we pass all that
> > > information
> > > > to the plugin). You could even - if you really want - make a decision
> > to
> > > > allow certain users (and only them) to be able to clear individual
> > tasks,
> > > > or tasks where specific operator types are used. For example you
> could
> > > > decide if you really want to only allow th google cloud admin users
> to
> > > > clear tasks that are operators coming from Google Provider (why
> not?).
> > Or
> > > > only allow to that in certain time windows, or .... whatever.
> > > >
> > > >
> > > >
> > > >
> > > > It's extremely flexible and powerful - but we will not have to deal
> > with
> > > > all the complexity in the community - we could delegate it to those
> who
> > > > mange the deployment or manage deployment of Airflow as a service and
> > > > decide they want to integrate it with the service authorisation that
> is
> > > > already there - think direct integratio with IAM ON AWS or GCP. And
> if
> > we
> > > > just provide a reference, opinionated implementation for keycloak
> with
> > > > tenant support and legacy/minimal FAB as optional plugins, that
> should
> > > also
> > > > 'good enough' for those who do not want to implement their own plugin
> > > > implementation.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>
> > > <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com>> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>> napisał:
> > > >
> > > >
> > > >
> > > >
> > > > > > I don't think we can do that: we still need to consider users
> when
> > > they
> > > > > are just getting started out with Airflow -- if we only cater to
> > giant
> > > > > enterprises with SSO and a central directory we remove our ability
> > for
> > > > > individual or small teams/companies
> <https://teams.googleplex.com/u/companies> <https://teams.googleplex.com/u/companies>> to get started with Airflow.
> > > > >
> > > > > Let me clarify this because I think we are talking about the same
> > > thing.
> > > > >
> > > > > I **think** having the "minimal FAB user manager implementation" as
> > the
> > > > > "default" option when you install Airflow serves the purpose you
> > > mention
> > > > > Ash - if we do it the way that User/Role management is coming
> > together
> > > > with
> > > > > FAB implementation (enabled by default for the initial
> installation).
> > > > What
> > > > > I am saying is that it should not be part of the "core airflow" in
> > the
> > > > > sense that one could live without it (specifically in those big
> > > > SSO/Central
> > > > > directory case).
> > > > > In my vision those two cases are separated, but the "minimal
> > > > > implementation" when enabled should act and behave "as if" nothing
> > > > changed
> > > > > compared to the current way it looks and behaves.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>
> > > > <mailto:ash@apache.org <ma...@apache.org>> <mailto:
> > > > ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <ma...@apache.org>>>> wrote:
> > > > >
> > > > >> > - I am fine with (and actually I like it better) removing User
> and
> > > > Role
> > > > >> related Rest APIs and CLI commands from Airflow. That will indeed
> > make
> > > > the
> > > > >> AIP way simpler and shorter
> > > > >>
> > > > >> I don't think we can do that: we still need to consider users when
> > > they
> > > > >> are just getting started out with Airflow -- if we only cater to
> > giant
> > > > >> enterprises with SSO and a central directory we remove our ability
> > for
> > > > >> individual or small teams/companies
> <https://teams.googleplex.com/u/companies> <https://teams.googleplex.com/u/companies>> to get started with Airflow.
> > > > >> On May 2 2023, at 8:23 pm, Beck, Vincent <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > >>LID>
> > > > >> wrote:
> > > > >> > Thank you Jarek and Bolke for taking time to read and provide
> > > feedback
> > > > >> to the AIP, much appreciated! Multiple points here:
> > > > >> >
> > > > >> > - I am fine with (and actually I like it better) removing User
> and
> > > > Role
> > > > >> related Rest APIs and CLI commands from Airflow. That will indeed
> > make
> > > > the
> > > > >> AIP way simpler and shorter. As I explained in the AIP comments, I
> > am
> > > > >> suggesting though to leave an option to user managers to define
> > > > additional
> > > > >> REST APIs and CLI commands if they need. By default no additional
> > REST
> > > > API
> > > > >> and CLI command would be defined but user managers would have the
> > > option
> > > > to
> > > > >> override it. FAB user manager would be an example of user manager
> > > > >> overriding such methods. In any case, if additional Rest API or
> CLI
> > > > command
> > > > >> is defined, this would be done in user managers (as opposed to
> core
> > > > >> Airflow).
> > > > >> > - "the whole "Security" menu should be GONE completely".
> > > > >> > > I agree with this when the user manager is not the FAB one,
> > > however
> > > > >> we need to introduce a new one so Admins can easily go the
> external
> > > user
> > > > >> manager console (e.g. KeyCloak, ...). This new tab would only be a
> > > > >> link/redirect to the external console.
> > > > >> >
> > > > >> > - "All those components (and code related to those) should be
> > moved
> > > to
> > > > >> FAB minimalistic user manager (and removed from Airflow Core)".
> > > > >> > > 100%. This is the main principle of this AIP, anything related
> > to
> > > > >> user management will be moved to user managers.
> > > > >> >
> > > > >> > - "AuthN and AuthZ should be completely removed from Airflow".
> > > > >> > > 100%.
> > > > >> >
> > > > >> > - "So for a nice out-of-the-box experience (pip install
> > > > apache-airflow)
> > > > >> a lightweight User Manager could exist as a side project and have
> > some
> > > > >> menus via the plugin system until something better comes along.".
> > > > >> > > I agree with you on the long term. Though, as a first
> iteration
> > I
> > > > >> think it is better to define a user manager using FAB as default
> > user
> > > > >> manager. The benefit of doing so is to have a backward compatible
> > > > >> transition between before AIP and after AIP. This also simplifies
> > the
> > > > AIP
> > > > >> since it is easier to move files from core Airflow to user
> managers
> > > than
> > > > >> creating something new. Creating a new security model would be a
> > > > challenge
> > > > >> too. However, once the AIP implemented and merged, we can easily
> > > replace
> > > > >> this out-of-the-box FAB user manager by something "better" if the
> > > > communty
> > > > >> thinks this is the right way to go.
> > > > >> >
> > > > >> > As always, feel free to give me your thoughts.
> > > > >> > Vincent
> > > > >> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com>
> > > <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>> <mailto:
> > > > >> bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> > > bdbruin@gmail.com <ma...@gmail.com>
> > > > <mailto:bdbruin@gmail.com <ma...@gmail.com>>>>> wrote:
> > > > >> > I'm not sure if I read the AIP correctly. I think I concur with
> > > Jarek
> > > > >> here.
> > > > >> >
> > > > >> > AuthN and AuthZ should be completely removed from Airflow.
> Access
> > > > >> > management should be outsourced to something like Open Policy
> > Agent.
> > > > In
> > > > >> > other words there should be no user management at all within
> > > Airflow.
> > > > >> This
> > > > >> > removes the need to "sync" users (and thus become outdated) and
> > > > reduces
> > > > >> the
> > > > >> > attack surface drastically. So for a nice out-of-the-box
> > experience
> > > > (pip
> > > > >> > install apache-airflow) a lightweight User Manager could exist
> as
> > a
> > > > side
> > > > >> > project and have some menus via the plugin system until
> something
> > > > better
> > > > >> > comes along. Keycloak is great, a bit heavy for us to package
> > > > >> > unfortunately.
> > > > >> >
> > > > >> >
> > > > >> > FAB's security model is very non-standard and requires
> 'unnatural'
> > > > >> mappings
> > > > >> > in an enterprise context. It has served us well, but I would say
> > > good
> > > > >> > riddance.
> > > > >> >
> > > > >> >
> > > > >> > Bolke
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <
> jarek@potiuk.com <ma...@potiuk.com>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> > > > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>>>:
> > > > >> >
> > > > >> > > I had a look and I liked what I saw for how we should
> integrate
> > > the
> > > > >> new
> > > > >> > > user manager for getting and checking the access :).
> > > > >> > >
> > > > >> > > But I think the AIP proposal is not bold enough when it comes
> to
> > > the
> > > > >> user
> > > > >> > > management part. This is one big comment - we should simplify
> it
> > > > even
> > > > >> > > further and cut more deeply.
> > > > >> > >
> > > > >> > > One of the big changes I think is super important with
> > extensible
> > > > user
> > > > >> > > management is that we should delegate even more to the
> external
> > > > >> systems
> > > > >> > > managing users. The AIP attempts to map the current API/CLI/UI
> > for
> > > > >> managing
> > > > >> > > the users and roles to the new external user management
> > services -
> > > > >> but it
> > > > >> > > is IMHO absolutely not needed.
> > > > >> > >
> > > > >> > > As I see the extensible user management is that when you
> choose
> > > > >> something
> > > > >> > > else than the legacy FAB minimalist user management. Airflow
> > > should
> > > > >> become
> > > > >> > > just "user" of that user/role information managed elsewhere.
> It
> > > > >> should not
> > > > >> > > provide ANY option to see and manage the users or roles, it
> > should
> > > > >> merely
> > > > >> > > read the information and give access to the users for
> > appropriate
> > > > >> actions,
> > > > >> > > but then all the role/user management should happen outside of
> > > > >> Airflow - in
> > > > >> > > the system that is dedicated to manage those.
> > > > >> > >
> > > > >> > > IMHO - when other than the "non FAB minimalistic user manager"
> > > > system
> > > > >> is
> > > > >> > > used. Airflow should just not have a number of functionalities
> > at
> > > > all:
> > > > >> > >
> > > > >> > > * not have "users" or "roles" CLI groups at all - they should
> be
> > > > >> missing
> > > > >> > > * the
> > > > >> > >
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User&gt>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User&gt>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > and
> > > > >> > >
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role&gt>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role&gt>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > should return 404 NOT FOUND
> > > > >> > > * the whole "Security" menu should be GONE completely.
> > > > >> > >
> > > > >> > > All those components (and code related to those) should be
> moved
> > > to
> > > > >> FAB
> > > > >> > > minimalistic user manager (and removed from Airflow Core).
> > > > Eventually
> > > > >> maybe
> > > > >> > > even moved out of the Airflow repo altogether and moved to a
> FAB
> > > > user
> > > > >> > > manager add-on to airflow.
> > > > >> > >
> > > > >> > > This might seem a drastic change, but in fact - it's not
> drastic
> > > at
> > > > >> all. It
> > > > >> > > reflects what many of the Airflow users are doing now - they
> > never
> > > > >> access
> > > > >> > > Security/Admin/Roles. Once they integrate with their LDAP or
> > other
> > > > >> > > corporate directory, the Security UI, CLI and REST API are
> > > > essentially
> > > > >> > > useless (and not used).
> > > > >> > >
> > > > >> > > And it is precisely what we want to achieve - we want to
> > delegate
> > > > the
> > > > >> > > user/role management completely out of Airflow. In fact -
> > getting
> > > > rid
> > > > >> of
> > > > >> > > those is the only reason we want to implement the change. If
> we
> > > > >> **just**
> > > > >> > > replace the current FAB and keep all the complexity of
> managing
> > > the
> > > > >> users
> > > > >> > > /roles in Airflow, I'd say we have not achieved anything but
> > > adding
> > > > >> > > complexity.
> > > > >> > >
> > > > >> > > And - eventually - it makes the AIP-56 way *smaller*. We just
> > need
> > > > to
> > > > >> make
> > > > >> > > sure that all the stuff for user management (including REST
> API,
> > > CLI
> > > > >> and
> > > > >> > > the UI) is moved to the "FAB minimalistic user manager" (and
> add
> > > > ways
> > > > >> to
> > > > >> > > plug-it in the core - possibly using existing plugins
> mechanisms
> > > > where
> > > > >> > > needed). We do not need to define new API for user/role
> > > management.
> > > > >> We only
> > > > >> > > need to describe the way how to check if the user has
> permission
> > > to
> > > > do
> > > > >> > > stuff.
> > > > >> > >
> > > > >> > > J.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <
> jarek@potiuk.com <ma...@potiuk.com>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> > > > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>>> wrote:
> > > > >> > >
> > > > >> > > > I promise to provide my feedback by the end of the week,
> sorry
> > > for
> > > > >> > > > delaying it, but we had some 2.6 preparation, branching,
> > feature
> > > > >> flags
> > > > >> > > etc.
> > > > >> > > > also for AIP-44 and I am getting back on track with that one
> > > soon.
> > > > >> > > >
> > > > >> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
> > > > >> <shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:
> > > > shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>>> <mailto:
> > > > shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:
> > > > shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>>>>lid
> > > > >> > > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > >> Bumping this up for feedback!
> > > > >> > > >>
> > > > >> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite
> > busy,
> > > > but
> > > > >> we
> > > > >> > > >> would greatly appreciate your feedback here. Your inputs
> > could
> > > > >> help us
> > > > >> > > >> improve the proposal and move forward with multi-tenancy
> > work.
> > > > >> > > >>
> > > > >> > > >> Cheers,
> > > > >> > > >> Shubham
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent"
> > > <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
> > > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> >
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>>
> > > > >> > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:
> > > > vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:
> > > > vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > > >>>>LID>
> > > > >> wrote:
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> CAUTION: This email originated from outside of the
> > > organization.
> > > > >> Do not
> > > > >> > > >> click links or open attachments unless you can confirm the
> > > sender
> > > > >> and
> > > > >> > > know
> > > > >> > > >> the content is safe.
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> Hi all,
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> I would like to get your feedback on an AIP I wrote:
> > > "Extensible
> > > > >> user
> > > > >> > > >> management". This AIP is a follow-up on a discussion we had
> > in
> > > > this
> > > > >> > > email
> > > > >> > > >> list on the multi tenancy topic. I decided to create a new
> > > email
> > > > >> thread
> > > > >> > > >> because I feel the topic had diverged a bit from the
> original
> > > > topic
> > > > >> > > (multi
> > > > >> > > >> tenancy).
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> As a summary, the purpose of this AIP is to extract the
> user
> > > > >> management
> > > > >> > > >> from core Airflow by introducing a user manager interface
> in
> > > the
> > > > >> core
> > > > >> > > >> Airflow that can be extended by any provider package who
> want
> > > to
> > > > >> support
> > > > >> > > >> user management natively. As a result, if this AIP gets
> > > approved
> > > > >> and go
> > > > >> > > >> through, the multi tenancy feature will be implemented as a
> > > > second
> > > > >> step
> > > > >> > > in
> > > > >> > > >> a new user manager and not in Airflow directly.
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> As always, feel very free to give your opinion on this
> email
> > > > >> thread or
> > > > >> > > on
> > > > >> > > >> the AIP by adding comments.
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> References:
> > > > >> > > >> - AIP:
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > >> <
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management&gt>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > >> >
> > > > >> > > >> - Initial email list discussion:
> > > > >> > > >>
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> > > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt>
> ;>
> > <
> > > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt>
> ;>
> > <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt>
> ;>
> > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt>
> > > ;>
> > > > <
> > > > >> > > >>
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> > > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt>
> ;>
> > > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt>
> ;>
> > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt>
> > > > ;>
> > > > >> <
> > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt>
> > > ;>
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt>
> > > ;>
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt>
> > > ;>
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt;> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;amp;gt;&amp;gt;&gt>
> > > > ;>
> > > >
> > > >
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> Vincent
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> >
> > > > >> > --
> > > > >> > Bolke de Bruin
> > > > >> > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>> <mailto:
> > bdbruin@gmail.com <ma...@gmail.com>
> > > > <mailto:bdbruin@gmail.com <ma...@gmail.com>>
> > > > <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>>
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > ---------------------------------------------------------------------
> > > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
> > <mailto:
> > > > dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > > > dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > > > dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>>>
> > > > >> > For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > <mailto:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>>>
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > > > dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>>
> > > > For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>
> <mailto:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>>
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> > --
> >
> > --
> > Bolke de Bruin
> > bdbruin@gmail.com <ma...@gmail.com>
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: AIP-56 Extensible user management
Posted by Jakub Kośmider <ko...@google.com.INVALID>.
Hello,
If I understand correctly, the proposition in AIP-56 is to introduce an
interface as generic as possible so that any potential federated identity
protocol can be integrated in Airflow through User Manager (or as proposed
in this thread, AuthManager). In light of this, note that the methods in
AuthManager in the AIP refer to the "current user", while the example of
"is_authorized" used above is based only on specific resources and
permissions provided as method parameters.
What do you think would be the context of "current user" provided by
Airflow to the implementation of AuthManager?
Would it be the entire request or maybe just cookies/auth tokens associated
with the request or something else?
Do you think it makes sense to document this in the AIP or leave decisions
for later?
Best regards,
Jakub
On Fri, May 19, 2023 at 11:22 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > Overall: I strongly suggest basing our security design on something
> existing. For AuthN I like the OIDC scheme. Furthermore, I suggest looking
> at https://flask-login.readthedocs.io/en/latest/ to see how this is
> handled
> in Flask.
>
> Very much agree. Good suggestion Bolke.
>
>
> On Fri, May 19, 2023 at 9:44 AM Bolke de Bruin <bd...@gmail.com> wrote:
>
> > Hi,
> >
> > On Wed, 17 May 2023 at 18:48, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > >
> > > >
> > > > I am very open to change the name of "User Manager" but I do not
> think
> > we
> > > > should call it "Security Manager". "Security Manager" is the current
> > name
> > > > and I think this is a very bad name because: it is very generic and
> > does
> > > > not say much about what it is actually doing. By reading "Security
> > > Manager"
> > > > I would not think this is where users and roles are managed. The main
> > > goal
> > > > of AIP-56 is removing the whole user management part from core
> Airflow
> > to
> > > > providers. Keeping the name "Security Manager" would, I think, leave
> a
> > > > legacy for all future user managers (or whatever name we use) on
> > > > Flask-AppBuilder (FAB) and that's exactly what I want to avoid. I
> chose
> > > the
> > > > name “User Manager” but I think it is simple, the main goal of this
> new
> > > > module is to manage users and access for them.
> > > >
> > > >
> > > How about *AuthManager* ? (Auth standing for both Authentication and
> > > Authorisation). I do agree "Security" is much bigger promise thant what
> > we
> > > are implementing here :).
> > >
> > >
> > That works better in my mind. Note we are not just managing Users here:
> it
> > can be non personal accounts etc.
> >
> >
> > >
> > >
> > > > > is_authorized should include context so that it becomes possible
> the
> > > > determine whether a user has access to a resource based on for
> example
> > > the
> > > > location the user is at. Or in which environment. It is also useful
> for
> > > > auditing. Non implementing security managers can ignore it.
> > > >
> > > > I agree, that can be indeed a good improvement. Although I have to
> say
> > I
> > > > am pretty confident the AIPs I defined in the API will change while
> > > > implementing them and over time. Thus, another option would be to add
> > > this
> > > > context when needed. I do not think (I might be wrong) features such
> as
> > > "
> > > > determine whether a user has access to a resource based on for
> example
> > > the
> > > > location the user is at. Or in which environment" currently exist in
> > > > Airflow. The goal of the AIP is not to implement new features but to
> > > have
> > > > the same features as we have today with this separation from core
> > Airflow
> > > > to providers. Again, I am not saying this is a wrong idea (and I
> > actually
> > > > think this is a good one), I just think if this feature does not
> exist
> > in
> > > > Airflow today, we should add that later.
> > > >
> > >
> > > I think Bolke is right: we should add information that we will pass
> > context
> > > and broadly what will be in. We can leave details for the
> implementation
> > > time but I think the context should be there. We should add what will
> be
> > > part of the context:
> > >
> > > * action
> > > * dag_id
> > > * task_id
> > > * labels for the dag
> > > * location in the DAG_FOLDER
> > > * connection_id (when we change connection)
> > > * variable-id (when we change variable)
> > > * ...
> > >
> > >
> > > This should be complete enough to make any kind of decisions. Probably
> it
> > > will also evolve over time in the future versions, but we need to have
> > > something to start with that we think will be comprehensive and
> complete
> > > enough to implement usable AuthManager(s) (providing that the name will
> > > stick).
> > >
> > >
> > I think this should be extensible (= flexible). Because, I would add
> > remote_addr, proxy_ips, roles, tag etc.
> >
> > For context, I would like to be able to setup policies like this:
> >
> >
> https://towardsdatascience.com/integrating-trino-and-apache-ranger-b808f6b96ad8
> >
> > Or in open policy agent:
> >
> > # Financial department staff can view any customer dags
> >
> > allow = true {
> > input.method = "GET"
> > input.path = ["payments", customer_id]
> > finance[input.user]}
> >
> > finance = {"john","mary","peter","vivian"}
> >
> >
> >
> > >
> > > > > What is the need of post_login? Why would you want to store the
> > access
> > > > token? The security manager should, imho, do this on its own: it
> > executes
> > > > the login flow.
> > > >
> > > > The access token is needed to call any API from the user manager to
> the
> > > > third party backend (KeyCloak, ...). How would you call
> is_authorized()
> > > > without it?
> > > >
> > >
> > > Yeah. we need to map whatever authentication information comes from
> > outside
> > > (usually a token coming from external authentication information) with
> > > (likely?) flask session in Airflow and I **think** the best is that it
> > > should be kept in AIrflow's DB the session in Airflow. I consider this
> a
> > > bit of an implementation detail, but maybe this needs a bit more
> hashing
> > ou
> > > maybe others can also comment here..
> > >
> > >
> > Particularly with security, I'd prefer clarity of how we intend to
> > implement this. If we consider an access_token equivalent to an OIDC
> > access_token that is fine, but let us be clear on what we base our design
> > on instead of seemingly inventing our own.
> >
> > At this moment, *any* DAG has full access to the database. That means if
> we
> > store user_id -> access_token any DAG can access any resource on behalf
> of
> > someone else. This means the AIrflow DB is one of the most insecure
> places
> > to store this kind of information.
> >
> > post_login is not required if authentication is entirely handled by the
> > AuthManager. If we consider post_login to be equivalent to a callback
> > method from OIDC (again: on what do we base our security design?) then
> sure
> > but then let's call it as such.
> >
> > Overall: I strongly suggest basing our security design on something
> > existing. For AuthN I like the OIDC scheme. Furthermore, I suggest
> looking
> > at https://flask-login.readthedocs.io/en/latest/ to see how this is
> > handled
> > in Flask.
> >
> >
> > > >
> > > > > Why is get_tab_configuration special? We have this possibility in
> > > > standard “plugins”. I suggest using that framework
> > > >
> > > > I am not familiar with plugins. I'll take a look at it, thanks for
> > > > bringing it up.
> > > >
> > >
> > > Yeah. The plugin mechanism is indeed foreseen for that - and the
> minimal
> > > FAB implementation should indeed implement plugin interface to add the
> > > current Admin menu - that was what I had in mind by the current
> > "Security"
> > > tab - it can **just** be added using the current plugin mechanism.
> > >
> > > >
> > > > > What is the need for “get_user_name”? Are we going to invent our
> own
> > > > Framework? Otherwise Flask might work?
> > > >
> > > > get_user_name is needed for the UI. On the top right corner of
> Airflow
> > > > console, you can see your name. This is why it is needed. The only
> goal
> > > of
> > > > this API is to retrieve your user name from the user manager. That's
> > it.
> > > > Flask will then use this API to get the user name and display it in
> the
> > > UI.
> > > > My experience in Flask is pretty limited so I might miss something
> but
> > > the
> > > > way I see it is, you need a way to retrieve your user name from
> > whatever
> > > > user manager you are using. This is the way.
> > >
> > >
> > Please have a look at flask login (
> > https://flask-login.readthedocs.io/en/latest/) or even base your design
> > upon Flask-Login. This ensures that the right information is available
> > across all components in a way that Flask components understand. The
> > question then becomes how does the DAG subsystem (which does not rely on
> > Flask) deal with this. It might need a kind of wrapper possibly.
> >
> >
> >
> > >
> > > >
> > > > I hope that helps,
> > > > Vincent
> > > >
> > > > On 2023-05-15, 5:30 PM, "Bolke de Bruin" <bdbruin@gmail.com
> <mailto:
> > > > bdbruin@gmail.com>> wrote:
> > > >
> > > >
> > > >
> > > > I think think the name “User Manager” is a misnomer and it should
> just
> > be
> > > > called “Security Manager”. It was confusing the hell out of me when
> > > reading
> > > > the AIP and I was convinced you wanted to fully manage e.g. keycloak
> > > > provided users inside Airflow (CRUD). What you are doing but aren’t
> > > exactly
> > > > saying:
> > > >
> > > >
> > > > 1 - Define API that a Security Plugin should adhere to (AuthZ/AuthN)
> > > > 2 - Have a default Security Plugin based on FAB
> > > >
> > > >
> > > > Points of improvement:
> > > >
> > > >
> > > > * is_authorized should include context so that it becomes possible
> the
> > > > determine whether a user has access to a resource based on for
> example
> > > the
> > > > location the user is at. Or in which environment. It is also useful
> for
> > > > auditing. Non implementing security managers can ignore it.
> > > > * I’d prefer some kind of ’standards’ based API. Like using Flask’s
> way
> > > of
> > > > registering endpoints or something along those lines. See also
> > questions
> > > > below.
> > > >
> > > >
> > > > Questions:
> > > >
> > > >
> > > > * What is the need of post_login? Why would you want to store the
> > access
> > > > token? The security manager should, imho, do this on its own: it
> > executes
> > > > the login flow.
> > > > * Why is get_tab_configuration special? We have this possibility in
> > > > standard “plugins”. I suggest using that framework
> > > > * What is the need for “get_user_name”? Are we going to invent our
> own
> > > > Framework? Otherwise Flask might work?
> > > >
> > > >
> > > >
> > > >
> > > > Cheers
> > > > Bolke
> > > >
> > > >
> > > > On 9 May 2023 at 19:55:32, Beck, Vincent (vincbeck@amazon.com.inva
> > > > <ma...@amazon.com.inva>lid)
> > > > wrote:
> > > >
> > > >
> > > > Hello all,
> > > >
> > > >
> > > > I would like to start voting on this one so please add your comments
> on
> > > the
> > > > API or by replying here if you're interested, you still have time to
> do
> > > it
> > > > :)
> > > >
> > > >
> > > > AIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > >
> > > >
> > > >
> > > > Vincent
> > > >
> > > >
> > > > On 2023-05-03, 8:03 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> > > > jarek@potiuk.com> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com>>> wrote:
> > > >
> > > >
> > > >
> > > >
> > > > CAUTION: This email originated from outside of the organization. Do
> not
> > > > click links or open attachments unless you can confirm the sender and
> > > know
> > > > the content is safe.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Alsoo to add (and make more concrete) to what it could mean at the
> > > > implementation level.
> > > >
> > > >
> > > >
> > > >
> > > > It means the for our API endpoints and views is that instead of
> > > > @auth.hasaccess (for example here
> > > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680
> > <
> > > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>
> > <
> > > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>
> > <
> > > >
> > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>
> > > > ;>)
> > > > where it check for permissions assigned to current role, airflow
> would
> > > > entirely delegate the decision to the 'user management plugin'.
> > > >
> > > >
> > > >
> > > >
> > > > We would simply delegate the decision to whatever plugin we have
> > > > configured. In case FAB pługin is used, it would do exactly the same
> > > checks
> > > > as today (and FAB pługin would contribute the 'Security' menu the
> same
> > > way
> > > > other plugins can contribute their own (as well as CLI commands and
> > > > user/role APIs - we would have to add a mechanism where plugins could
> > > > contribute CLIs and APIs on their own, but that should be easy). So
> if
> > > you
> > > > have FAB pługin chosen as default, nothing changes for the users
> > > comparing
> > > > to today. It would not require bumping Airflow Major version because
> it
> > > > would be backwards compatible. However similarly as providers it
> would
> > > move
> > > > the code of FAB plugins and roles out of the core of Airflow.
> > > >
> > > >
> > > >
> > > >
> > > > We could have keycloak 'reference' implementation where we would come
> > up
> > > > with very opinionated approach where we have simple admin/user roles
> > > > (matching roughly default set of roles and permissions we currently
> > have
> > > > with FAB). The big difference here will be that the user/role APIs,
> > > views,
> > > > CLIs will be gone entirely. And we could add there the concept of
> > > > 'tenants' or rather 'teams' giving access to 'dag groups' and
> implement
> > > > decisions ob authorisation based on 'group' assignment for dag and
> > user.
> > > > This is the main goal of this change to make it possible and easy
> > without
> > > > having to mess with FAB permission and Role model which is completely
> > not
> > > > suitable to manage 'groups of resources of the same type' with
> separate
> > > set
> > > > of permissions (think having few separate teams or tenants - each of
> > them
> > > > wanting access to separate set of DAGs).
> > > >
> > > >
> > > >
> > > >
> > > > Currently if an SSO/enterprise user wants to plug in their
> authn/authz
> > > with
> > > > group access - they have to deal with all that including cumbersome
> > > syncing
> > > > of permissions for individual DAGs matching them to users separately
> -
> > > just
> > > > to satisfy FAB permission model lacking this feature and flexibility.
> > > >
> > > >
> > > >
> > > >
> > > > No more need for that if we go the direction described here.
> > > >
> > > >
> > > >
> > > >
> > > > And that's basically it for what we would implement in Airflow
> > > > out-of-the-box: FAB minimal implementation + be keycloak opinionated
> > > > reference implementation with simple teams/tenant
> <https://teams.googleplex.com/u/tenant> support
> > > >
> > > >
> > > >
> > > >
> > > > This also makes it super flexible for those who want more flexibility
> > and
> > > > do not want to use our opinionated keycloak plugin. They will be
> > entirely
> > > > free to implement their own ways - as flexible or as opinionated they
> > > want
> > > > - without having to messwith permission syncing and the role model of
> > > FAB.
> > > > Nor keycloak integration. They could go with their own.
> > > >
> > > >
> > > >
> > > >
> > > > And with that - sky is the limit. The decision to access particular
> dag
> > > > could be made based on folder where the dag is from, id of dag, or
> dag
> > > > label or whatever other parameter (as long as we pass all that
> > > information
> > > > to the plugin). You could even - if you really want - make a decision
> > to
> > > > allow certain users (and only them) to be able to clear individual
> > tasks,
> > > > or tasks where specific operator types are used. For example you
> could
> > > > decide if you really want to only allow th google cloud admin users
> to
> > > > clear tasks that are operators coming from Google Provider (why
> not?).
> > Or
> > > > only allow to that in certain time windows, or .... whatever.
> > > >
> > > >
> > > >
> > > >
> > > > It's extremely flexible and powerful - but we will not have to deal
> > with
> > > > all the complexity in the community - we could delegate it to those
> who
> > > > mange the deployment or manage deployment of Airflow as a service and
> > > > decide they want to integrate it with the service authorisation that
> is
> > > > already there - think direct integratio with IAM ON AWS or GCP. And
> if
> > we
> > > > just provide a reference, opinionated implementation for keycloak
> with
> > > > tenant support and legacy/minimal FAB as optional plugins, that
> should
> > > also
> > > > 'good enough' for those who do not want to implement their own plugin
> > > > implementation.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <jarek@potiuk.com
> > > <mailto:
> > > > jarek@potiuk.com> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com>>> napisał:
> > > >
> > > >
> > > >
> > > >
> > > > > > I don't think we can do that: we still need to consider users
> when
> > > they
> > > > > are just getting started out with Airflow -- if we only cater to
> > giant
> > > > > enterprises with SSO and a central directory we remove our ability
> > for
> > > > > individual or small teams/companies
> <https://teams.googleplex.com/u/companies> to get started with Airflow.
> > > > >
> > > > > Let me clarify this because I think we are talking about the same
> > > thing.
> > > > >
> > > > > I **think** having the "minimal FAB user manager implementation" as
> > the
> > > > > "default" option when you install Airflow serves the purpose you
> > > mention
> > > > > Ash - if we do it the way that User/Role management is coming
> > together
> > > > with
> > > > > FAB implementation (enabled by default for the initial
> installation).
> > > > What
> > > > > I am saying is that it should not be part of the "core airflow" in
> > the
> > > > > sense that one could live without it (specifically in those big
> > > > SSO/Central
> > > > > directory case).
> > > > > In my vision those two cases are separated, but the "minimal
> > > > > implementation" when enabled should act and behave "as if" nothing
> > > > changed
> > > > > compared to the current way it looks and behaves.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <ash@apache.org
> > > > <ma...@apache.org> <mailto:
> > > > ash@apache.org <ma...@apache.org>>> wrote:
> > > > >
> > > > >> > - I am fine with (and actually I like it better) removing User
> and
> > > > Role
> > > > >> related Rest APIs and CLI commands from Airflow. That will indeed
> > make
> > > > the
> > > > >> AIP way simpler and shorter
> > > > >>
> > > > >> I don't think we can do that: we still need to consider users when
> > > they
> > > > >> are just getting started out with Airflow -- if we only cater to
> > giant
> > > > >> enterprises with SSO and a central directory we remove our ability
> > for
> > > > >> individual or small teams/companies
> <https://teams.googleplex.com/u/companies> to get started with Airflow.
> > > > >> On May 2 2023, at 8:23 pm, Beck, Vincent <vincbeck@amazon.com.INVA
> > > > <ma...@amazon.com.INVA>
> > > > <mailto:vincbeck@amazon.com.INVA <mailto:vincbeck@amazon.com.INVA
> > >>LID>
> > > > >> wrote:
> > > > >> > Thank you Jarek and Bolke for taking time to read and provide
> > > feedback
> > > > >> to the AIP, much appreciated! Multiple points here:
> > > > >> >
> > > > >> > - I am fine with (and actually I like it better) removing User
> and
> > > > Role
> > > > >> related Rest APIs and CLI commands from Airflow. That will indeed
> > make
> > > > the
> > > > >> AIP way simpler and shorter. As I explained in the AIP comments, I
> > am
> > > > >> suggesting though to leave an option to user managers to define
> > > > additional
> > > > >> REST APIs and CLI commands if they need. By default no additional
> > REST
> > > > API
> > > > >> and CLI command would be defined but user managers would have the
> > > option
> > > > to
> > > > >> override it. FAB user manager would be an example of user manager
> > > > >> overriding such methods. In any case, if additional Rest API or
> CLI
> > > > command
> > > > >> is defined, this would be done in user managers (as opposed to
> core
> > > > >> Airflow).
> > > > >> > - "the whole "Security" menu should be GONE completely".
> > > > >> > > I agree with this when the user manager is not the FAB one,
> > > however
> > > > >> we need to introduce a new one so Admins can easily go the
> external
> > > user
> > > > >> manager console (e.g. KeyCloak, ...). This new tab would only be a
> > > > >> link/redirect to the external console.
> > > > >> >
> > > > >> > - "All those components (and code related to those) should be
> > moved
> > > to
> > > > >> FAB minimalistic user manager (and removed from Airflow Core)".
> > > > >> > > 100%. This is the main principle of this AIP, anything related
> > to
> > > > >> user management will be moved to user managers.
> > > > >> >
> > > > >> > - "AuthN and AuthZ should be completely removed from Airflow".
> > > > >> > > 100%.
> > > > >> >
> > > > >> > - "So for a nice out-of-the-box experience (pip install
> > > > apache-airflow)
> > > > >> a lightweight User Manager could exist as a side project and have
> > some
> > > > >> menus via the plugin system until something better comes along.".
> > > > >> > > I agree with you on the long term. Though, as a first
> iteration
> > I
> > > > >> think it is better to define a user manager using FAB as default
> > user
> > > > >> manager. The benefit of doing so is to have a backward compatible
> > > > >> transition between before AIP and after AIP. This also simplifies
> > the
> > > > AIP
> > > > >> since it is easier to move files from core Airflow to user
> managers
> > > than
> > > > >> creating something new. Creating a new security model would be a
> > > > challenge
> > > > >> too. However, once the AIP implemented and merged, we can easily
> > > replace
> > > > >> this out-of-the-box FAB user manager by something "better" if the
> > > > communty
> > > > >> thinks this is the right way to go.
> > > > >> >
> > > > >> > As always, feel free to give me your thoughts.
> > > > >> > Vincent
> > > > >> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com
> > > <mailto:
> > > > bdbruin@gmail.com> <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> > > > >> bdbruin@gmail.com <ma...@gmail.com> <mailto:
> > > bdbruin@gmail.com
> > > > <ma...@gmail.com>>>> wrote:
> > > > >> > I'm not sure if I read the AIP correctly. I think I concur with
> > > Jarek
> > > > >> here.
> > > > >> >
> > > > >> > AuthN and AuthZ should be completely removed from Airflow.
> Access
> > > > >> > management should be outsourced to something like Open Policy
> > Agent.
> > > > In
> > > > >> > other words there should be no user management at all within
> > > Airflow.
> > > > >> This
> > > > >> > removes the need to "sync" users (and thus become outdated) and
> > > > reduces
> > > > >> the
> > > > >> > attack surface drastically. So for a nice out-of-the-box
> > experience
> > > > (pip
> > > > >> > install apache-airflow) a lightweight User Manager could exist
> as
> > a
> > > > side
> > > > >> > project and have some menus via the plugin system until
> something
> > > > better
> > > > >> > comes along. Keycloak is great, a bit heavy for us to package
> > > > >> > unfortunately.
> > > > >> >
> > > > >> >
> > > > >> > FAB's security model is very non-standard and requires
> 'unnatural'
> > > > >> mappings
> > > > >> > in an enterprise context. It has served us well, but I would say
> > > good
> > > > >> > riddance.
> > > > >> >
> > > > >> >
> > > > >> > Bolke
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <
> jarek@potiuk.com
> > > > <ma...@potiuk.com>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > > > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com>>>>:
> > > > >> >
> > > > >> > > I had a look and I liked what I saw for how we should
> integrate
> > > the
> > > > >> new
> > > > >> > > user manager for getting and checking the access :).
> > > > >> > >
> > > > >> > > But I think the AIP proposal is not bold enough when it comes
> to
> > > the
> > > > >> user
> > > > >> > > management part. This is one big comment - we should simplify
> it
> > > > even
> > > > >> > > further and cut more deeply.
> > > > >> > >
> > > > >> > > One of the big changes I think is super important with
> > extensible
> > > > user
> > > > >> > > management is that we should delegate even more to the
> external
> > > > >> systems
> > > > >> > > managing users. The AIP attempts to map the current API/CLI/UI
> > for
> > > > >> managing
> > > > >> > > the users and roles to the new external user management
> > services -
> > > > >> but it
> > > > >> > > is IMHO absolutely not needed.
> > > > >> > >
> > > > >> > > As I see the extensible user management is that when you
> choose
> > > > >> something
> > > > >> > > else than the legacy FAB minimalist user management. Airflow
> > > should
> > > > >> become
> > > > >> > > just "user" of that user/role information managed elsewhere.
> It
> > > > >> should not
> > > > >> > > provide ANY option to see and manage the users or roles, it
> > should
> > > > >> merely
> > > > >> > > read the information and give access to the users for
> > appropriate
> > > > >> actions,
> > > > >> > > but then all the role/user management should happen outside of
> > > > >> Airflow - in
> > > > >> > > the system that is dedicated to manage those.
> > > > >> > >
> > > > >> > > IMHO - when other than the "non FAB minimalistic user manager"
> > > > system
> > > > >> is
> > > > >> > > used. Airflow should just not have a number of functionalities
> > at
> > > > all:
> > > > >> > >
> > > > >> > > * not have "users" or "roles" CLI groups at all - they should
> be
> > > > >> missing
> > > > >> > > * the
> > > > >> > >
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > and
> > > > >> > >
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > >
> > > > <
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > should return 404 NOT FOUND
> > > > >> > > * the whole "Security" menu should be GONE completely.
> > > > >> > >
> > > > >> > > All those components (and code related to those) should be
> moved
> > > to
> > > > >> FAB
> > > > >> > > minimalistic user manager (and removed from Airflow Core).
> > > > Eventually
> > > > >> maybe
> > > > >> > > even moved out of the Airflow repo altogether and moved to a
> FAB
> > > > user
> > > > >> > > manager add-on to airflow.
> > > > >> > >
> > > > >> > > This might seem a drastic change, but in fact - it's not
> drastic
> > > at
> > > > >> all. It
> > > > >> > > reflects what many of the Airflow users are doing now - they
> > never
> > > > >> access
> > > > >> > > Security/Admin/Roles. Once they integrate with their LDAP or
> > other
> > > > >> > > corporate directory, the Security UI, CLI and REST API are
> > > > essentially
> > > > >> > > useless (and not used).
> > > > >> > >
> > > > >> > > And it is precisely what we want to achieve - we want to
> > delegate
> > > > the
> > > > >> > > user/role management completely out of Airflow. In fact -
> > getting
> > > > rid
> > > > >> of
> > > > >> > > those is the only reason we want to implement the change. If
> we
> > > > >> **just**
> > > > >> > > replace the current FAB and keep all the complexity of
> managing
> > > the
> > > > >> users
> > > > >> > > /roles in Airflow, I'd say we have not achieved anything but
> > > adding
> > > > >> > > complexity.
> > > > >> > >
> > > > >> > > And - eventually - it makes the AIP-56 way *smaller*. We just
> > need
> > > > to
> > > > >> make
> > > > >> > > sure that all the stuff for user management (including REST
> API,
> > > CLI
> > > > >> and
> > > > >> > > the UI) is moved to the "FAB minimalistic user manager" (and
> add
> > > > ways
> > > > >> to
> > > > >> > > plug-it in the core - possibly using existing plugins
> mechanisms
> > > > where
> > > > >> > > needed). We do not need to define new API for user/role
> > > management.
> > > > >> We only
> > > > >> > > need to describe the way how to check if the user has
> permission
> > > to
> > > > do
> > > > >> > > stuff.
> > > > >> > >
> > > > >> > > J.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <
> jarek@potiuk.com
> > > > <ma...@potiuk.com>
> > > > <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > > > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:
> > > > jarek@potiuk.com <ma...@potiuk.com>>>> wrote:
> > > > >> > >
> > > > >> > > > I promise to provide my feedback by the end of the week,
> sorry
> > > for
> > > > >> > > > delaying it, but we had some 2.6 preparation, branching,
> > feature
> > > > >> flags
> > > > >> > > etc.
> > > > >> > > > also for AIP-44 and I am getting back on track with that one
> > > soon.
> > > > >> > > >
> > > > >> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
> > > > >> <shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:
> > > > shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:
> > > > shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:
> > > > shubhx@amazon.com.inva <ma...@amazon.com.inva>>>lid
> > > > >> > > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > >> Bumping this up for feedback!
> > > > >> > > >>
> > > > >> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite
> > busy,
> > > > but
> > > > >> we
> > > > >> > > >> would greatly appreciate your feedback here. Your inputs
> > could
> > > > >> help us
> > > > >> > > >> improve the proposal and move forward with multi-tenancy
> > work.
> > > > >> > > >>
> > > > >> > > >> Cheers,
> > > > >> > > >> Shubham
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent"
> > > <vincbeck@amazon.com.INVA
> > > > <ma...@amazon.com.INVA>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > > > >> <mailto:vincbeck@amazon.com.INVA <mailto:vincbeck@amazon.com.INVA
> >
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
> > > > >> > > >> <mailto:vincbeck@amazon.com.INVA <mailto:
> > > > vincbeck@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <mailto:
> > > > vincbeck@amazon.com.INVA>>
> > > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > > > <mailto:vincbeck@amazon.com.INVA <mailto:vincbeck@amazon.com.INVA
> > > >>>>LID>
> > > > >> wrote:
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> CAUTION: This email originated from outside of the
> > > organization.
> > > > >> Do not
> > > > >> > > >> click links or open attachments unless you can confirm the
> > > sender
> > > > >> and
> > > > >> > > know
> > > > >> > > >> the content is safe.
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> Hi all,
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> I would like to get your feedback on an AIP I wrote:
> > > "Extensible
> > > > >> user
> > > > >> > > >> management". This AIP is a follow-up on a discussion we had
> > in
> > > > this
> > > > >> > > email
> > > > >> > > >> list on the multi tenancy topic. I decided to create a new
> > > email
> > > > >> thread
> > > > >> > > >> because I feel the topic had diverged a bit from the
> original
> > > > topic
> > > > >> > > (multi
> > > > >> > > >> tenancy).
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> As a summary, the purpose of this AIP is to extract the
> user
> > > > >> management
> > > > >> > > >> from core Airflow by introducing a user manager interface
> in
> > > the
> > > > >> core
> > > > >> > > >> Airflow that can be extended by any provider package who
> want
> > > to
> > > > >> support
> > > > >> > > >> user management natively. As a result, if this AIP gets
> > > approved
> > > > >> and go
> > > > >> > > >> through, the multi tenancy feature will be implemented as a
> > > > second
> > > > >> step
> > > > >> > > in
> > > > >> > > >> a new user manager and not in Airflow directly.
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> As always, feel very free to give your opinion on this
> email
> > > > >> thread or
> > > > >> > > on
> > > > >> > > >> the AIP by adding comments.
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> References:
> > > > >> > > >> - AIP:
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > >> <
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > ;>
> > > >
> > > >
> > > > >> <
> > > > >>
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > >
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > > ;>
> > > >
> > > >
> > > > >> >
> > > > >> > > >> >
> > > > >> > > >> - Initial email list discussion:
> > > > >> > > >>
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61
> > > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> ;>
> > <
> > > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> ;>
> > <
> > > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> ;>
> > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> > > ;>
> > > > <
> > > > >> > > >>
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> > > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> ;>
> > > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> ;>
> > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> > > > ;>
> > > > >> <
> > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> > > ;>
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> > > ;>
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> > > ;>
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt;>
> > > > ;>
> > > >
> > > >
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >> Vincent
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> >
> > > > >> > --
> > > > >> > Bolke de Bruin
> > > > >> > bdbruin@gmail.com <ma...@gmail.com> <mailto:
> > > > bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> > bdbruin@gmail.com
> > > > <ma...@gmail.com>
> > > > <mailto:bdbruin@gmail.com <ma...@gmail.com>>>
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > ---------------------------------------------------------------------
> > > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > <mailto:
> > > > dev-unsubscribe@airflow.apache.org> <mailto:
> > > > dev-unsubscribe@airflow.apache.org <mailto:
> > > > dev-unsubscribe@airflow.apache.org>>
> > > > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> > > <mailto:
> > > > dev-help@airflow.apache.org> <mailto:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>>
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> > > > dev-unsubscribe@airflow.apache.org>
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> <mailto:
> > > > dev-help@airflow.apache.org>
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> > --
> >
> > --
> > Bolke de Bruin
> > bdbruin@gmail.com
> >
>
Re: AIP-56 Extensible user management
Posted by Jarek Potiuk <ja...@potiuk.com>.
> Overall: I strongly suggest basing our security design on something
existing. For AuthN I like the OIDC scheme. Furthermore, I suggest looking
at https://flask-login.readthedocs.io/en/latest/ to see how this is handled
in Flask.
Very much agree. Good suggestion Bolke.
On Fri, May 19, 2023 at 9:44 AM Bolke de Bruin <bd...@gmail.com> wrote:
> Hi,
>
> On Wed, 17 May 2023 at 18:48, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > >
> > >
> > > I am very open to change the name of "User Manager" but I do not think
> we
> > > should call it "Security Manager". "Security Manager" is the current
> name
> > > and I think this is a very bad name because: it is very generic and
> does
> > > not say much about what it is actually doing. By reading "Security
> > Manager"
> > > I would not think this is where users and roles are managed. The main
> > goal
> > > of AIP-56 is removing the whole user management part from core Airflow
> to
> > > providers. Keeping the name "Security Manager" would, I think, leave a
> > > legacy for all future user managers (or whatever name we use) on
> > > Flask-AppBuilder (FAB) and that's exactly what I want to avoid. I chose
> > the
> > > name “User Manager” but I think it is simple, the main goal of this new
> > > module is to manage users and access for them.
> > >
> > >
> > How about *AuthManager* ? (Auth standing for both Authentication and
> > Authorisation). I do agree "Security" is much bigger promise thant what
> we
> > are implementing here :).
> >
> >
> That works better in my mind. Note we are not just managing Users here: it
> can be non personal accounts etc.
>
>
> >
> >
> > > > is_authorized should include context so that it becomes possible the
> > > determine whether a user has access to a resource based on for example
> > the
> > > location the user is at. Or in which environment. It is also useful for
> > > auditing. Non implementing security managers can ignore it.
> > >
> > > I agree, that can be indeed a good improvement. Although I have to say
> I
> > > am pretty confident the AIPs I defined in the API will change while
> > > implementing them and over time. Thus, another option would be to add
> > this
> > > context when needed. I do not think (I might be wrong) features such as
> > "
> > > determine whether a user has access to a resource based on for example
> > the
> > > location the user is at. Or in which environment" currently exist in
> > > Airflow. The goal of the AIP is not to implement new features but to
> > have
> > > the same features as we have today with this separation from core
> Airflow
> > > to providers. Again, I am not saying this is a wrong idea (and I
> actually
> > > think this is a good one), I just think if this feature does not exist
> in
> > > Airflow today, we should add that later.
> > >
> >
> > I think Bolke is right: we should add information that we will pass
> context
> > and broadly what will be in. We can leave details for the implementation
> > time but I think the context should be there. We should add what will be
> > part of the context:
> >
> > * action
> > * dag_id
> > * task_id
> > * labels for the dag
> > * location in the DAG_FOLDER
> > * connection_id (when we change connection)
> > * variable-id (when we change variable)
> > * ...
> >
> >
> > This should be complete enough to make any kind of decisions. Probably it
> > will also evolve over time in the future versions, but we need to have
> > something to start with that we think will be comprehensive and complete
> > enough to implement usable AuthManager(s) (providing that the name will
> > stick).
> >
> >
> I think this should be extensible (= flexible). Because, I would add
> remote_addr, proxy_ips, roles, tag etc.
>
> For context, I would like to be able to setup policies like this:
>
> https://towardsdatascience.com/integrating-trino-and-apache-ranger-b808f6b96ad8
>
> Or in open policy agent:
>
> # Financial department staff can view any customer dags
>
> allow = true {
> input.method = "GET"
> input.path = ["payments", customer_id]
> finance[input.user]}
>
> finance = {"john","mary","peter","vivian"}
>
>
>
> >
> > > > What is the need of post_login? Why would you want to store the
> access
> > > token? The security manager should, imho, do this on its own: it
> executes
> > > the login flow.
> > >
> > > The access token is needed to call any API from the user manager to the
> > > third party backend (KeyCloak, ...). How would you call is_authorized()
> > > without it?
> > >
> >
> > Yeah. we need to map whatever authentication information comes from
> outside
> > (usually a token coming from external authentication information) with
> > (likely?) flask session in Airflow and I **think** the best is that it
> > should be kept in AIrflow's DB the session in Airflow. I consider this a
> > bit of an implementation detail, but maybe this needs a bit more hashing
> ou
> > maybe others can also comment here..
> >
> >
> Particularly with security, I'd prefer clarity of how we intend to
> implement this. If we consider an access_token equivalent to an OIDC
> access_token that is fine, but let us be clear on what we base our design
> on instead of seemingly inventing our own.
>
> At this moment, *any* DAG has full access to the database. That means if we
> store user_id -> access_token any DAG can access any resource on behalf of
> someone else. This means the AIrflow DB is one of the most insecure places
> to store this kind of information.
>
> post_login is not required if authentication is entirely handled by the
> AuthManager. If we consider post_login to be equivalent to a callback
> method from OIDC (again: on what do we base our security design?) then sure
> but then let's call it as such.
>
> Overall: I strongly suggest basing our security design on something
> existing. For AuthN I like the OIDC scheme. Furthermore, I suggest looking
> at https://flask-login.readthedocs.io/en/latest/ to see how this is
> handled
> in Flask.
>
>
> > >
> > > > Why is get_tab_configuration special? We have this possibility in
> > > standard “plugins”. I suggest using that framework
> > >
> > > I am not familiar with plugins. I'll take a look at it, thanks for
> > > bringing it up.
> > >
> >
> > Yeah. The plugin mechanism is indeed foreseen for that - and the minimal
> > FAB implementation should indeed implement plugin interface to add the
> > current Admin menu - that was what I had in mind by the current
> "Security"
> > tab - it can **just** be added using the current plugin mechanism.
> >
> > >
> > > > What is the need for “get_user_name”? Are we going to invent our own
> > > Framework? Otherwise Flask might work?
> > >
> > > get_user_name is needed for the UI. On the top right corner of Airflow
> > > console, you can see your name. This is why it is needed. The only goal
> > of
> > > this API is to retrieve your user name from the user manager. That's
> it.
> > > Flask will then use this API to get the user name and display it in the
> > UI.
> > > My experience in Flask is pretty limited so I might miss something but
> > the
> > > way I see it is, you need a way to retrieve your user name from
> whatever
> > > user manager you are using. This is the way.
> >
> >
> Please have a look at flask login (
> https://flask-login.readthedocs.io/en/latest/) or even base your design
> upon Flask-Login. This ensures that the right information is available
> across all components in a way that Flask components understand. The
> question then becomes how does the DAG subsystem (which does not rely on
> Flask) deal with this. It might need a kind of wrapper possibly.
>
>
>
> >
> > >
> > > I hope that helps,
> > > Vincent
> > >
> > > On 2023-05-15, 5:30 PM, "Bolke de Bruin" <bdbruin@gmail.com <mailto:
> > > bdbruin@gmail.com>> wrote:
> > >
> > >
> > >
> > > I think think the name “User Manager” is a misnomer and it should just
> be
> > > called “Security Manager”. It was confusing the hell out of me when
> > reading
> > > the AIP and I was convinced you wanted to fully manage e.g. keycloak
> > > provided users inside Airflow (CRUD). What you are doing but aren’t
> > exactly
> > > saying:
> > >
> > >
> > > 1 - Define API that a Security Plugin should adhere to (AuthZ/AuthN)
> > > 2 - Have a default Security Plugin based on FAB
> > >
> > >
> > > Points of improvement:
> > >
> > >
> > > * is_authorized should include context so that it becomes possible the
> > > determine whether a user has access to a resource based on for example
> > the
> > > location the user is at. Or in which environment. It is also useful for
> > > auditing. Non implementing security managers can ignore it.
> > > * I’d prefer some kind of ’standards’ based API. Like using Flask’s way
> > of
> > > registering endpoints or something along those lines. See also
> questions
> > > below.
> > >
> > >
> > > Questions:
> > >
> > >
> > > * What is the need of post_login? Why would you want to store the
> access
> > > token? The security manager should, imho, do this on its own: it
> executes
> > > the login flow.
> > > * Why is get_tab_configuration special? We have this possibility in
> > > standard “plugins”. I suggest using that framework
> > > * What is the need for “get_user_name”? Are we going to invent our own
> > > Framework? Otherwise Flask might work?
> > >
> > >
> > >
> > >
> > > Cheers
> > > Bolke
> > >
> > >
> > > On 9 May 2023 at 19:55:32, Beck, Vincent (vincbeck@amazon.com.inva
> > > <ma...@amazon.com.inva>lid)
> > > wrote:
> > >
> > >
> > > Hello all,
> > >
> > >
> > > I would like to start voting on this one so please add your comments on
> > the
> > > API or by replying here if you're interested, you still have time to do
> > it
> > > :)
> > >
> > >
> > > AIP:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > >
> > >
> > >
> > > Vincent
> > >
> > >
> > > On 2023-05-03, 8:03 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> > > jarek@potiuk.com> <mailto:
> > > jarek@potiuk.com <ma...@potiuk.com>>> wrote:
> > >
> > >
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > > click links or open attachments unless you can confirm the sender and
> > know
> > > the content is safe.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Alsoo to add (and make more concrete) to what it could mean at the
> > > implementation level.
> > >
> > >
> > >
> > >
> > > It means the for our API endpoints and views is that instead of
> > > @auth.hasaccess (for example here
> > > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680
> <
> > > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>
> <
> > > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>
> <
> > >
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>
> > > ;>)
> > > where it check for permissions assigned to current role, airflow would
> > > entirely delegate the decision to the 'user management plugin'.
> > >
> > >
> > >
> > >
> > > We would simply delegate the decision to whatever plugin we have
> > > configured. In case FAB pługin is used, it would do exactly the same
> > checks
> > > as today (and FAB pługin would contribute the 'Security' menu the same
> > way
> > > other plugins can contribute their own (as well as CLI commands and
> > > user/role APIs - we would have to add a mechanism where plugins could
> > > contribute CLIs and APIs on their own, but that should be easy). So if
> > you
> > > have FAB pługin chosen as default, nothing changes for the users
> > comparing
> > > to today. It would not require bumping Airflow Major version because it
> > > would be backwards compatible. However similarly as providers it would
> > move
> > > the code of FAB plugins and roles out of the core of Airflow.
> > >
> > >
> > >
> > >
> > > We could have keycloak 'reference' implementation where we would come
> up
> > > with very opinionated approach where we have simple admin/user roles
> > > (matching roughly default set of roles and permissions we currently
> have
> > > with FAB). The big difference here will be that the user/role APIs,
> > views,
> > > CLIs will be gone entirely. And we could add there the concept of
> > > 'tenants' or rather 'teams' giving access to 'dag groups' and implement
> > > decisions ob authorisation based on 'group' assignment for dag and
> user.
> > > This is the main goal of this change to make it possible and easy
> without
> > > having to mess with FAB permission and Role model which is completely
> not
> > > suitable to manage 'groups of resources of the same type' with separate
> > set
> > > of permissions (think having few separate teams or tenants - each of
> them
> > > wanting access to separate set of DAGs).
> > >
> > >
> > >
> > >
> > > Currently if an SSO/enterprise user wants to plug in their authn/authz
> > with
> > > group access - they have to deal with all that including cumbersome
> > syncing
> > > of permissions for individual DAGs matching them to users separately -
> > just
> > > to satisfy FAB permission model lacking this feature and flexibility.
> > >
> > >
> > >
> > >
> > > No more need for that if we go the direction described here.
> > >
> > >
> > >
> > >
> > > And that's basically it for what we would implement in Airflow
> > > out-of-the-box: FAB minimal implementation + be keycloak opinionated
> > > reference implementation with simple teams/tenant support
> > >
> > >
> > >
> > >
> > > This also makes it super flexible for those who want more flexibility
> and
> > > do not want to use our opinionated keycloak plugin. They will be
> entirely
> > > free to implement their own ways - as flexible or as opinionated they
> > want
> > > - without having to messwith permission syncing and the role model of
> > FAB.
> > > Nor keycloak integration. They could go with their own.
> > >
> > >
> > >
> > >
> > > And with that - sky is the limit. The decision to access particular dag
> > > could be made based on folder where the dag is from, id of dag, or dag
> > > label or whatever other parameter (as long as we pass all that
> > information
> > > to the plugin). You could even - if you really want - make a decision
> to
> > > allow certain users (and only them) to be able to clear individual
> tasks,
> > > or tasks where specific operator types are used. For example you could
> > > decide if you really want to only allow th google cloud admin users to
> > > clear tasks that are operators coming from Google Provider (why not?).
> Or
> > > only allow to that in certain time windows, or .... whatever.
> > >
> > >
> > >
> > >
> > > It's extremely flexible and powerful - but we will not have to deal
> with
> > > all the complexity in the community - we could delegate it to those who
> > > mange the deployment or manage deployment of Airflow as a service and
> > > decide they want to integrate it with the service authorisation that is
> > > already there - think direct integratio with IAM ON AWS or GCP. And if
> we
> > > just provide a reference, opinionated implementation for keycloak with
> > > tenant support and legacy/minimal FAB as optional plugins, that should
> > also
> > > 'good enough' for those who do not want to implement their own plugin
> > > implementation.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > J.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <jarek@potiuk.com
> > <mailto:
> > > jarek@potiuk.com> <mailto:
> > > jarek@potiuk.com <ma...@potiuk.com>>> napisał:
> > >
> > >
> > >
> > >
> > > > > I don't think we can do that: we still need to consider users when
> > they
> > > > are just getting started out with Airflow -- if we only cater to
> giant
> > > > enterprises with SSO and a central directory we remove our ability
> for
> > > > individual or small teams/companies to get started with Airflow.
> > > >
> > > > Let me clarify this because I think we are talking about the same
> > thing.
> > > >
> > > > I **think** having the "minimal FAB user manager implementation" as
> the
> > > > "default" option when you install Airflow serves the purpose you
> > mention
> > > > Ash - if we do it the way that User/Role management is coming
> together
> > > with
> > > > FAB implementation (enabled by default for the initial installation).
> > > What
> > > > I am saying is that it should not be part of the "core airflow" in
> the
> > > > sense that one could live without it (specifically in those big
> > > SSO/Central
> > > > directory case).
> > > > In my vision those two cases are separated, but the "minimal
> > > > implementation" when enabled should act and behave "as if" nothing
> > > changed
> > > > compared to the current way it looks and behaves.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <ash@apache.org
> > > <ma...@apache.org> <mailto:
> > > ash@apache.org <ma...@apache.org>>> wrote:
> > > >
> > > >> > - I am fine with (and actually I like it better) removing User and
> > > Role
> > > >> related Rest APIs and CLI commands from Airflow. That will indeed
> make
> > > the
> > > >> AIP way simpler and shorter
> > > >>
> > > >> I don't think we can do that: we still need to consider users when
> > they
> > > >> are just getting started out with Airflow -- if we only cater to
> giant
> > > >> enterprises with SSO and a central directory we remove our ability
> for
> > > >> individual or small teams/companies to get started with Airflow.
> > > >> On May 2 2023, at 8:23 pm, Beck, Vincent <vincbeck@amazon.com.INVA
> > > <ma...@amazon.com.INVA>
> > > <mailto:vincbeck@amazon.com.INVA <mailto:vincbeck@amazon.com.INVA
> >>LID>
> > > >> wrote:
> > > >> > Thank you Jarek and Bolke for taking time to read and provide
> > feedback
> > > >> to the AIP, much appreciated! Multiple points here:
> > > >> >
> > > >> > - I am fine with (and actually I like it better) removing User and
> > > Role
> > > >> related Rest APIs and CLI commands from Airflow. That will indeed
> make
> > > the
> > > >> AIP way simpler and shorter. As I explained in the AIP comments, I
> am
> > > >> suggesting though to leave an option to user managers to define
> > > additional
> > > >> REST APIs and CLI commands if they need. By default no additional
> REST
> > > API
> > > >> and CLI command would be defined but user managers would have the
> > option
> > > to
> > > >> override it. FAB user manager would be an example of user manager
> > > >> overriding such methods. In any case, if additional Rest API or CLI
> > > command
> > > >> is defined, this would be done in user managers (as opposed to core
> > > >> Airflow).
> > > >> > - "the whole "Security" menu should be GONE completely".
> > > >> > > I agree with this when the user manager is not the FAB one,
> > however
> > > >> we need to introduce a new one so Admins can easily go the external
> > user
> > > >> manager console (e.g. KeyCloak, ...). This new tab would only be a
> > > >> link/redirect to the external console.
> > > >> >
> > > >> > - "All those components (and code related to those) should be
> moved
> > to
> > > >> FAB minimalistic user manager (and removed from Airflow Core)".
> > > >> > > 100%. This is the main principle of this AIP, anything related
> to
> > > >> user management will be moved to user managers.
> > > >> >
> > > >> > - "AuthN and AuthZ should be completely removed from Airflow".
> > > >> > > 100%.
> > > >> >
> > > >> > - "So for a nice out-of-the-box experience (pip install
> > > apache-airflow)
> > > >> a lightweight User Manager could exist as a side project and have
> some
> > > >> menus via the plugin system until something better comes along.".
> > > >> > > I agree with you on the long term. Though, as a first iteration
> I
> > > >> think it is better to define a user manager using FAB as default
> user
> > > >> manager. The benefit of doing so is to have a backward compatible
> > > >> transition between before AIP and after AIP. This also simplifies
> the
> > > AIP
> > > >> since it is easier to move files from core Airflow to user managers
> > than
> > > >> creating something new. Creating a new security model would be a
> > > challenge
> > > >> too. However, once the AIP implemented and merged, we can easily
> > replace
> > > >> this out-of-the-box FAB user manager by something "better" if the
> > > communty
> > > >> thinks this is the right way to go.
> > > >> >
> > > >> > As always, feel free to give me your thoughts.
> > > >> > Vincent
> > > >> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com
> > <mailto:
> > > bdbruin@gmail.com> <mailto:
> > > bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> > > >> bdbruin@gmail.com <ma...@gmail.com> <mailto:
> > bdbruin@gmail.com
> > > <ma...@gmail.com>>>> wrote:
> > > >> > I'm not sure if I read the AIP correctly. I think I concur with
> > Jarek
> > > >> here.
> > > >> >
> > > >> > AuthN and AuthZ should be completely removed from Airflow. Access
> > > >> > management should be outsourced to something like Open Policy
> Agent.
> > > In
> > > >> > other words there should be no user management at all within
> > Airflow.
> > > >> This
> > > >> > removes the need to "sync" users (and thus become outdated) and
> > > reduces
> > > >> the
> > > >> > attack surface drastically. So for a nice out-of-the-box
> experience
> > > (pip
> > > >> > install apache-airflow) a lightweight User Manager could exist as
> a
> > > side
> > > >> > project and have some menus via the plugin system until something
> > > better
> > > >> > comes along. Keycloak is great, a bit heavy for us to package
> > > >> > unfortunately.
> > > >> >
> > > >> >
> > > >> > FAB's security model is very non-standard and requires 'unnatural'
> > > >> mappings
> > > >> > in an enterprise context. It has served us well, but I would say
> > good
> > > >> > riddance.
> > > >> >
> > > >> >
> > > >> > Bolke
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com
> > > <ma...@potiuk.com>
> > > <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:
> > > jarek@potiuk.com <ma...@potiuk.com>>>>:
> > > >> >
> > > >> > > I had a look and I liked what I saw for how we should integrate
> > the
> > > >> new
> > > >> > > user manager for getting and checking the access :).
> > > >> > >
> > > >> > > But I think the AIP proposal is not bold enough when it comes to
> > the
> > > >> user
> > > >> > > management part. This is one big comment - we should simplify it
> > > even
> > > >> > > further and cut more deeply.
> > > >> > >
> > > >> > > One of the big changes I think is super important with
> extensible
> > > user
> > > >> > > management is that we should delegate even more to the external
> > > >> systems
> > > >> > > managing users. The AIP attempts to map the current API/CLI/UI
> for
> > > >> managing
> > > >> > > the users and roles to the new external user management
> services -
> > > >> but it
> > > >> > > is IMHO absolutely not needed.
> > > >> > >
> > > >> > > As I see the extensible user management is that when you choose
> > > >> something
> > > >> > > else than the legacy FAB minimalist user management. Airflow
> > should
> > > >> become
> > > >> > > just "user" of that user/role information managed elsewhere. It
> > > >> should not
> > > >> > > provide ANY option to see and manage the users or roles, it
> should
> > > >> merely
> > > >> > > read the information and give access to the users for
> appropriate
> > > >> actions,
> > > >> > > but then all the role/user management should happen outside of
> > > >> Airflow - in
> > > >> > > the system that is dedicated to manage those.
> > > >> > >
> > > >> > > IMHO - when other than the "non FAB minimalistic user manager"
> > > system
> > > >> is
> > > >> > > used. Airflow should just not have a number of functionalities
> at
> > > all:
> > > >> > >
> > > >> > > * not have "users" or "roles" CLI groups at all - they should be
> > > >> missing
> > > >> > > * the
> > > >> > >
> > > >> > >
> > > >>
> > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > >
> > > <
> > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > >
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > ;>
> > >
> > >
> > > >> <
> > > >>
> > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > > >
> > > <
> > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > >
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > > ;>
> > >
> > >
> > > >> >
> > > >> > > and
> > > >> > >
> > > >> > >
> > > >>
> > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > >
> > > <
> > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > >
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > ;>
> > >
> > >
> > > >> <
> > > >>
> > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > > >
> > > <
> > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > >
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > > ;>
> > >
> > >
> > > >> >
> > > >> > > should return 404 NOT FOUND
> > > >> > > * the whole "Security" menu should be GONE completely.
> > > >> > >
> > > >> > > All those components (and code related to those) should be moved
> > to
> > > >> FAB
> > > >> > > minimalistic user manager (and removed from Airflow Core).
> > > Eventually
> > > >> maybe
> > > >> > > even moved out of the Airflow repo altogether and moved to a FAB
> > > user
> > > >> > > manager add-on to airflow.
> > > >> > >
> > > >> > > This might seem a drastic change, but in fact - it's not drastic
> > at
> > > >> all. It
> > > >> > > reflects what many of the Airflow users are doing now - they
> never
> > > >> access
> > > >> > > Security/Admin/Roles. Once they integrate with their LDAP or
> other
> > > >> > > corporate directory, the Security UI, CLI and REST API are
> > > essentially
> > > >> > > useless (and not used).
> > > >> > >
> > > >> > > And it is precisely what we want to achieve - we want to
> delegate
> > > the
> > > >> > > user/role management completely out of Airflow. In fact -
> getting
> > > rid
> > > >> of
> > > >> > > those is the only reason we want to implement the change. If we
> > > >> **just**
> > > >> > > replace the current FAB and keep all the complexity of managing
> > the
> > > >> users
> > > >> > > /roles in Airflow, I'd say we have not achieved anything but
> > adding
> > > >> > > complexity.
> > > >> > >
> > > >> > > And - eventually - it makes the AIP-56 way *smaller*. We just
> need
> > > to
> > > >> make
> > > >> > > sure that all the stuff for user management (including REST API,
> > CLI
> > > >> and
> > > >> > > the UI) is moved to the "FAB minimalistic user manager" (and add
> > > ways
> > > >> to
> > > >> > > plug-it in the core - possibly using existing plugins mechanisms
> > > where
> > > >> > > needed). We do not need to define new API for user/role
> > management.
> > > >> We only
> > > >> > > need to describe the way how to check if the user has permission
> > to
> > > do
> > > >> > > stuff.
> > > >> > >
> > > >> > > J.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com
> > > <ma...@potiuk.com>
> > > <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:
> > > jarek@potiuk.com <ma...@potiuk.com>>>> wrote:
> > > >> > >
> > > >> > > > I promise to provide my feedback by the end of the week, sorry
> > for
> > > >> > > > delaying it, but we had some 2.6 preparation, branching,
> feature
> > > >> flags
> > > >> > > etc.
> > > >> > > > also for AIP-44 and I am getting back on track with that one
> > soon.
> > > >> > > >
> > > >> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
> > > >> <shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:
> > > shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:
> > > shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:
> > > shubhx@amazon.com.inva <ma...@amazon.com.inva>>>lid
> > > >> > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > >> Bumping this up for feedback!
> > > >> > > >>
> > > >> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite
> busy,
> > > but
> > > >> we
> > > >> > > >> would greatly appreciate your feedback here. Your inputs
> could
> > > >> help us
> > > >> > > >> improve the proposal and move forward with multi-tenancy
> work.
> > > >> > > >>
> > > >> > > >> Cheers,
> > > >> > > >> Shubham
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent"
> > <vincbeck@amazon.com.INVA
> > > <ma...@amazon.com.INVA>
> > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
> > > >> > > >> <mailto:vincbeck@amazon.com.INVA <mailto:
> > > vincbeck@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <mailto:
> > > vincbeck@amazon.com.INVA>>
> > > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > > <mailto:vincbeck@amazon.com.INVA <mailto:vincbeck@amazon.com.INVA
> > >>>>LID>
> > > >> wrote:
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> CAUTION: This email originated from outside of the
> > organization.
> > > >> Do not
> > > >> > > >> click links or open attachments unless you can confirm the
> > sender
> > > >> and
> > > >> > > know
> > > >> > > >> the content is safe.
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> Hi all,
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> I would like to get your feedback on an AIP I wrote:
> > "Extensible
> > > >> user
> > > >> > > >> management". This AIP is a follow-up on a discussion we had
> in
> > > this
> > > >> > > email
> > > >> > > >> list on the multi tenancy topic. I decided to create a new
> > email
> > > >> thread
> > > >> > > >> because I feel the topic had diverged a bit from the original
> > > topic
> > > >> > > (multi
> > > >> > > >> tenancy).
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> As a summary, the purpose of this AIP is to extract the user
> > > >> management
> > > >> > > >> from core Airflow by introducing a user manager interface in
> > the
> > > >> core
> > > >> > > >> Airflow that can be extended by any provider package who want
> > to
> > > >> support
> > > >> > > >> user management natively. As a result, if this AIP gets
> > approved
> > > >> and go
> > > >> > > >> through, the multi tenancy feature will be implemented as a
> > > second
> > > >> step
> > > >> > > in
> > > >> > > >> a new user manager and not in Airflow directly.
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> As always, feel very free to give your opinion on this email
> > > >> thread or
> > > >> > > on
> > > >> > > >> the AIP by adding comments.
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> References:
> > > >> > > >> - AIP:
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > >
> > > <
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > >
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > ;>
> > >
> > >
> > > >> <
> > > >>
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > >
> > > <
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > >
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > ;>
> > >
> > >
> > > >> >
> > > >> > > >> <
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > >
> > > <
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > >
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > ;>
> > >
> > >
> > > >> <
> > > >>
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > > >
> > > <
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > >
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > > ;>
> > >
> > >
> > > >> >
> > > >> > > >> >
> > > >> > > >> - Initial email list discussion:
> > > >> > > >>
> > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61
> > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> <
> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> <
> > > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> <
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> > ;>
> > > <
> > > >> > > >>
> > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> > > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> <
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> > > ;>
> > > >> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> > ;>
> > > <
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> > ;>
> > > <
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> > ;>
> > > <
> > >
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt;>
> > > ;>
> > >
> > >
> > > >> > > >>
> > > >> > > >>
> > > >> > > >> Vincent
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > --
> > > >> > Bolke de Bruin
> > > >> > bdbruin@gmail.com <ma...@gmail.com> <mailto:
> > > bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> bdbruin@gmail.com
> > > <ma...@gmail.com>
> > > <mailto:bdbruin@gmail.com <ma...@gmail.com>>>
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > ---------------------------------------------------------------------
> > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> <mailto:
> > > dev-unsubscribe@airflow.apache.org> <mailto:
> > > dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org>>
> > > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> > <mailto:
> > > dev-help@airflow.apache.org> <mailto:
> > > dev-help@airflow.apache.org <ma...@airflow.apache.org>>
> > > >> >
> > > >>
> > > >>
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org>
> > > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> > > dev-help@airflow.apache.org>
> > >
> > >
> > >
> > >
> >
>
>
> --
>
> --
> Bolke de Bruin
> bdbruin@gmail.com
>
Re: AIP-56 Extensible user management
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi,
On Wed, 17 May 2023 at 18:48, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >
> > I am very open to change the name of "User Manager" but I do not think we
> > should call it "Security Manager". "Security Manager" is the current name
> > and I think this is a very bad name because: it is very generic and does
> > not say much about what it is actually doing. By reading "Security
> Manager"
> > I would not think this is where users and roles are managed. The main
> goal
> > of AIP-56 is removing the whole user management part from core Airflow to
> > providers. Keeping the name "Security Manager" would, I think, leave a
> > legacy for all future user managers (or whatever name we use) on
> > Flask-AppBuilder (FAB) and that's exactly what I want to avoid. I chose
> the
> > name “User Manager” but I think it is simple, the main goal of this new
> > module is to manage users and access for them.
> >
> >
> How about *AuthManager* ? (Auth standing for both Authentication and
> Authorisation). I do agree "Security" is much bigger promise thant what we
> are implementing here :).
>
>
That works better in my mind. Note we are not just managing Users here: it
can be non personal accounts etc.
>
>
> > > is_authorized should include context so that it becomes possible the
> > determine whether a user has access to a resource based on for example
> the
> > location the user is at. Or in which environment. It is also useful for
> > auditing. Non implementing security managers can ignore it.
> >
> > I agree, that can be indeed a good improvement. Although I have to say I
> > am pretty confident the AIPs I defined in the API will change while
> > implementing them and over time. Thus, another option would be to add
> this
> > context when needed. I do not think (I might be wrong) features such as
> "
> > determine whether a user has access to a resource based on for example
> the
> > location the user is at. Or in which environment" currently exist in
> > Airflow. The goal of the AIP is not to implement new features but to
> have
> > the same features as we have today with this separation from core Airflow
> > to providers. Again, I am not saying this is a wrong idea (and I actually
> > think this is a good one), I just think if this feature does not exist in
> > Airflow today, we should add that later.
> >
>
> I think Bolke is right: we should add information that we will pass context
> and broadly what will be in. We can leave details for the implementation
> time but I think the context should be there. We should add what will be
> part of the context:
>
> * action
> * dag_id
> * task_id
> * labels for the dag
> * location in the DAG_FOLDER
> * connection_id (when we change connection)
> * variable-id (when we change variable)
> * ...
>
>
> This should be complete enough to make any kind of decisions. Probably it
> will also evolve over time in the future versions, but we need to have
> something to start with that we think will be comprehensive and complete
> enough to implement usable AuthManager(s) (providing that the name will
> stick).
>
>
I think this should be extensible (= flexible). Because, I would add
remote_addr, proxy_ips, roles, tag etc.
For context, I would like to be able to setup policies like this:
https://towardsdatascience.com/integrating-trino-and-apache-ranger-b808f6b96ad8
Or in open policy agent:
# Financial department staff can view any customer dags
allow = true {
input.method = "GET"
input.path = ["payments", customer_id]
finance[input.user]}
finance = {"john","mary","peter","vivian"}
>
> > > What is the need of post_login? Why would you want to store the access
> > token? The security manager should, imho, do this on its own: it executes
> > the login flow.
> >
> > The access token is needed to call any API from the user manager to the
> > third party backend (KeyCloak, ...). How would you call is_authorized()
> > without it?
> >
>
> Yeah. we need to map whatever authentication information comes from outside
> (usually a token coming from external authentication information) with
> (likely?) flask session in Airflow and I **think** the best is that it
> should be kept in AIrflow's DB the session in Airflow. I consider this a
> bit of an implementation detail, but maybe this needs a bit more hashing ou
> maybe others can also comment here..
>
>
Particularly with security, I'd prefer clarity of how we intend to
implement this. If we consider an access_token equivalent to an OIDC
access_token that is fine, but let us be clear on what we base our design
on instead of seemingly inventing our own.
At this moment, *any* DAG has full access to the database. That means if we
store user_id -> access_token any DAG can access any resource on behalf of
someone else. This means the AIrflow DB is one of the most insecure places
to store this kind of information.
post_login is not required if authentication is entirely handled by the
AuthManager. If we consider post_login to be equivalent to a callback
method from OIDC (again: on what do we base our security design?) then sure
but then let's call it as such.
Overall: I strongly suggest basing our security design on something
existing. For AuthN I like the OIDC scheme. Furthermore, I suggest looking
at https://flask-login.readthedocs.io/en/latest/ to see how this is handled
in Flask.
> >
> > > Why is get_tab_configuration special? We have this possibility in
> > standard “plugins”. I suggest using that framework
> >
> > I am not familiar with plugins. I'll take a look at it, thanks for
> > bringing it up.
> >
>
> Yeah. The plugin mechanism is indeed foreseen for that - and the minimal
> FAB implementation should indeed implement plugin interface to add the
> current Admin menu - that was what I had in mind by the current "Security"
> tab - it can **just** be added using the current plugin mechanism.
>
> >
> > > What is the need for “get_user_name”? Are we going to invent our own
> > Framework? Otherwise Flask might work?
> >
> > get_user_name is needed for the UI. On the top right corner of Airflow
> > console, you can see your name. This is why it is needed. The only goal
> of
> > this API is to retrieve your user name from the user manager. That's it.
> > Flask will then use this API to get the user name and display it in the
> UI.
> > My experience in Flask is pretty limited so I might miss something but
> the
> > way I see it is, you need a way to retrieve your user name from whatever
> > user manager you are using. This is the way.
>
>
Please have a look at flask login (
https://flask-login.readthedocs.io/en/latest/) or even base your design
upon Flask-Login. This ensures that the right information is available
across all components in a way that Flask components understand. The
question then becomes how does the DAG subsystem (which does not rely on
Flask) deal with this. It might need a kind of wrapper possibly.
>
> >
> > I hope that helps,
> > Vincent
> >
> > On 2023-05-15, 5:30 PM, "Bolke de Bruin" <bdbruin@gmail.com <mailto:
> > bdbruin@gmail.com>> wrote:
> >
> >
> >
> > I think think the name “User Manager” is a misnomer and it should just be
> > called “Security Manager”. It was confusing the hell out of me when
> reading
> > the AIP and I was convinced you wanted to fully manage e.g. keycloak
> > provided users inside Airflow (CRUD). What you are doing but aren’t
> exactly
> > saying:
> >
> >
> > 1 - Define API that a Security Plugin should adhere to (AuthZ/AuthN)
> > 2 - Have a default Security Plugin based on FAB
> >
> >
> > Points of improvement:
> >
> >
> > * is_authorized should include context so that it becomes possible the
> > determine whether a user has access to a resource based on for example
> the
> > location the user is at. Or in which environment. It is also useful for
> > auditing. Non implementing security managers can ignore it.
> > * I’d prefer some kind of ’standards’ based API. Like using Flask’s way
> of
> > registering endpoints or something along those lines. See also questions
> > below.
> >
> >
> > Questions:
> >
> >
> > * What is the need of post_login? Why would you want to store the access
> > token? The security manager should, imho, do this on its own: it executes
> > the login flow.
> > * Why is get_tab_configuration special? We have this possibility in
> > standard “plugins”. I suggest using that framework
> > * What is the need for “get_user_name”? Are we going to invent our own
> > Framework? Otherwise Flask might work?
> >
> >
> >
> >
> > Cheers
> > Bolke
> >
> >
> > On 9 May 2023 at 19:55:32, Beck, Vincent (vincbeck@amazon.com.inva
> > <ma...@amazon.com.inva>lid)
> > wrote:
> >
> >
> > Hello all,
> >
> >
> > I would like to start voting on this one so please add your comments on
> the
> > API or by replying here if you're interested, you still have time to do
> it
> > :)
> >
> >
> > AIP:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > >
> >
> >
> > Vincent
> >
> >
> > On 2023-05-03, 8:03 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> > jarek@potiuk.com> <mailto:
> > jarek@potiuk.com <ma...@potiuk.com>>> wrote:
> >
> >
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Alsoo to add (and make more concrete) to what it could mean at the
> > implementation level.
> >
> >
> >
> >
> > It means the for our API endpoints and views is that instead of
> > @auth.hasaccess (for example here
> > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680 <
> > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <
> > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <
> > https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>
> > ;>)
> > where it check for permissions assigned to current role, airflow would
> > entirely delegate the decision to the 'user management plugin'.
> >
> >
> >
> >
> > We would simply delegate the decision to whatever plugin we have
> > configured. In case FAB pługin is used, it would do exactly the same
> checks
> > as today (and FAB pługin would contribute the 'Security' menu the same
> way
> > other plugins can contribute their own (as well as CLI commands and
> > user/role APIs - we would have to add a mechanism where plugins could
> > contribute CLIs and APIs on their own, but that should be easy). So if
> you
> > have FAB pługin chosen as default, nothing changes for the users
> comparing
> > to today. It would not require bumping Airflow Major version because it
> > would be backwards compatible. However similarly as providers it would
> move
> > the code of FAB plugins and roles out of the core of Airflow.
> >
> >
> >
> >
> > We could have keycloak 'reference' implementation where we would come up
> > with very opinionated approach where we have simple admin/user roles
> > (matching roughly default set of roles and permissions we currently have
> > with FAB). The big difference here will be that the user/role APIs,
> views,
> > CLIs will be gone entirely. And we could add there the concept of
> > 'tenants' or rather 'teams' giving access to 'dag groups' and implement
> > decisions ob authorisation based on 'group' assignment for dag and user.
> > This is the main goal of this change to make it possible and easy without
> > having to mess with FAB permission and Role model which is completely not
> > suitable to manage 'groups of resources of the same type' with separate
> set
> > of permissions (think having few separate teams or tenants - each of them
> > wanting access to separate set of DAGs).
> >
> >
> >
> >
> > Currently if an SSO/enterprise user wants to plug in their authn/authz
> with
> > group access - they have to deal with all that including cumbersome
> syncing
> > of permissions for individual DAGs matching them to users separately -
> just
> > to satisfy FAB permission model lacking this feature and flexibility.
> >
> >
> >
> >
> > No more need for that if we go the direction described here.
> >
> >
> >
> >
> > And that's basically it for what we would implement in Airflow
> > out-of-the-box: FAB minimal implementation + be keycloak opinionated
> > reference implementation with simple teams/tenant support
> >
> >
> >
> >
> > This also makes it super flexible for those who want more flexibility and
> > do not want to use our opinionated keycloak plugin. They will be entirely
> > free to implement their own ways - as flexible or as opinionated they
> want
> > - without having to messwith permission syncing and the role model of
> FAB.
> > Nor keycloak integration. They could go with their own.
> >
> >
> >
> >
> > And with that - sky is the limit. The decision to access particular dag
> > could be made based on folder where the dag is from, id of dag, or dag
> > label or whatever other parameter (as long as we pass all that
> information
> > to the plugin). You could even - if you really want - make a decision to
> > allow certain users (and only them) to be able to clear individual tasks,
> > or tasks where specific operator types are used. For example you could
> > decide if you really want to only allow th google cloud admin users to
> > clear tasks that are operators coming from Google Provider (why not?). Or
> > only allow to that in certain time windows, or .... whatever.
> >
> >
> >
> >
> > It's extremely flexible and powerful - but we will not have to deal with
> > all the complexity in the community - we could delegate it to those who
> > mange the deployment or manage deployment of Airflow as a service and
> > decide they want to integrate it with the service authorisation that is
> > already there - think direct integratio with IAM ON AWS or GCP. And if we
> > just provide a reference, opinionated implementation for keycloak with
> > tenant support and legacy/minimal FAB as optional plugins, that should
> also
> > 'good enough' for those who do not want to implement their own plugin
> > implementation.
> >
> >
> >
> >
> >
> >
> >
> >
> > J.
> >
> >
> >
> >
> >
> >
> >
> >
> > śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <jarek@potiuk.com
> <mailto:
> > jarek@potiuk.com> <mailto:
> > jarek@potiuk.com <ma...@potiuk.com>>> napisał:
> >
> >
> >
> >
> > > > I don't think we can do that: we still need to consider users when
> they
> > > are just getting started out with Airflow -- if we only cater to giant
> > > enterprises with SSO and a central directory we remove our ability for
> > > individual or small teams/companies to get started with Airflow.
> > >
> > > Let me clarify this because I think we are talking about the same
> thing.
> > >
> > > I **think** having the "minimal FAB user manager implementation" as the
> > > "default" option when you install Airflow serves the purpose you
> mention
> > > Ash - if we do it the way that User/Role management is coming together
> > with
> > > FAB implementation (enabled by default for the initial installation).
> > What
> > > I am saying is that it should not be part of the "core airflow" in the
> > > sense that one could live without it (specifically in those big
> > SSO/Central
> > > directory case).
> > > In my vision those two cases are separated, but the "minimal
> > > implementation" when enabled should act and behave "as if" nothing
> > changed
> > > compared to the current way it looks and behaves.
> > >
> > > J.
> > >
> > >
> > > On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <ash@apache.org
> > <ma...@apache.org> <mailto:
> > ash@apache.org <ma...@apache.org>>> wrote:
> > >
> > >> > - I am fine with (and actually I like it better) removing User and
> > Role
> > >> related Rest APIs and CLI commands from Airflow. That will indeed make
> > the
> > >> AIP way simpler and shorter
> > >>
> > >> I don't think we can do that: we still need to consider users when
> they
> > >> are just getting started out with Airflow -- if we only cater to giant
> > >> enterprises with SSO and a central directory we remove our ability for
> > >> individual or small teams/companies to get started with Airflow.
> > >> On May 2 2023, at 8:23 pm, Beck, Vincent <vincbeck@amazon.com.INVA
> > <ma...@amazon.com.INVA>
> > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>LID>
> > >> wrote:
> > >> > Thank you Jarek and Bolke for taking time to read and provide
> feedback
> > >> to the AIP, much appreciated! Multiple points here:
> > >> >
> > >> > - I am fine with (and actually I like it better) removing User and
> > Role
> > >> related Rest APIs and CLI commands from Airflow. That will indeed make
> > the
> > >> AIP way simpler and shorter. As I explained in the AIP comments, I am
> > >> suggesting though to leave an option to user managers to define
> > additional
> > >> REST APIs and CLI commands if they need. By default no additional REST
> > API
> > >> and CLI command would be defined but user managers would have the
> option
> > to
> > >> override it. FAB user manager would be an example of user manager
> > >> overriding such methods. In any case, if additional Rest API or CLI
> > command
> > >> is defined, this would be done in user managers (as opposed to core
> > >> Airflow).
> > >> > - "the whole "Security" menu should be GONE completely".
> > >> > > I agree with this when the user manager is not the FAB one,
> however
> > >> we need to introduce a new one so Admins can easily go the external
> user
> > >> manager console (e.g. KeyCloak, ...). This new tab would only be a
> > >> link/redirect to the external console.
> > >> >
> > >> > - "All those components (and code related to those) should be moved
> to
> > >> FAB minimalistic user manager (and removed from Airflow Core)".
> > >> > > 100%. This is the main principle of this AIP, anything related to
> > >> user management will be moved to user managers.
> > >> >
> > >> > - "AuthN and AuthZ should be completely removed from Airflow".
> > >> > > 100%.
> > >> >
> > >> > - "So for a nice out-of-the-box experience (pip install
> > apache-airflow)
> > >> a lightweight User Manager could exist as a side project and have some
> > >> menus via the plugin system until something better comes along.".
> > >> > > I agree with you on the long term. Though, as a first iteration I
> > >> think it is better to define a user manager using FAB as default user
> > >> manager. The benefit of doing so is to have a backward compatible
> > >> transition between before AIP and after AIP. This also simplifies the
> > AIP
> > >> since it is easier to move files from core Airflow to user managers
> than
> > >> creating something new. Creating a new security model would be a
> > challenge
> > >> too. However, once the AIP implemented and merged, we can easily
> replace
> > >> this out-of-the-box FAB user manager by something "better" if the
> > communty
> > >> thinks this is the right way to go.
> > >> >
> > >> > As always, feel free to give me your thoughts.
> > >> > Vincent
> > >> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com
> <mailto:
> > bdbruin@gmail.com> <mailto:
> > bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> > >> bdbruin@gmail.com <ma...@gmail.com> <mailto:
> bdbruin@gmail.com
> > <ma...@gmail.com>>>> wrote:
> > >> > I'm not sure if I read the AIP correctly. I think I concur with
> Jarek
> > >> here.
> > >> >
> > >> > AuthN and AuthZ should be completely removed from Airflow. Access
> > >> > management should be outsourced to something like Open Policy Agent.
> > In
> > >> > other words there should be no user management at all within
> Airflow.
> > >> This
> > >> > removes the need to "sync" users (and thus become outdated) and
> > reduces
> > >> the
> > >> > attack surface drastically. So for a nice out-of-the-box experience
> > (pip
> > >> > install apache-airflow) a lightweight User Manager could exist as a
> > side
> > >> > project and have some menus via the plugin system until something
> > better
> > >> > comes along. Keycloak is great, a bit heavy for us to package
> > >> > unfortunately.
> > >> >
> > >> >
> > >> > FAB's security model is very non-standard and requires 'unnatural'
> > >> mappings
> > >> > in an enterprise context. It has served us well, but I would say
> good
> > >> > riddance.
> > >> >
> > >> >
> > >> > Bolke
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com
> > <ma...@potiuk.com>
> > <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:
> > jarek@potiuk.com <ma...@potiuk.com>>>>:
> > >> >
> > >> > > I had a look and I liked what I saw for how we should integrate
> the
> > >> new
> > >> > > user manager for getting and checking the access :).
> > >> > >
> > >> > > But I think the AIP proposal is not bold enough when it comes to
> the
> > >> user
> > >> > > management part. This is one big comment - we should simplify it
> > even
> > >> > > further and cut more deeply.
> > >> > >
> > >> > > One of the big changes I think is super important with extensible
> > user
> > >> > > management is that we should delegate even more to the external
> > >> systems
> > >> > > managing users. The AIP attempts to map the current API/CLI/UI for
> > >> managing
> > >> > > the users and roles to the new external user management services -
> > >> but it
> > >> > > is IMHO absolutely not needed.
> > >> > >
> > >> > > As I see the extensible user management is that when you choose
> > >> something
> > >> > > else than the legacy FAB minimalist user management. Airflow
> should
> > >> become
> > >> > > just "user" of that user/role information managed elsewhere. It
> > >> should not
> > >> > > provide ANY option to see and manage the users or roles, it should
> > >> merely
> > >> > > read the information and give access to the users for appropriate
> > >> actions,
> > >> > > but then all the role/user management should happen outside of
> > >> Airflow - in
> > >> > > the system that is dedicated to manage those.
> > >> > >
> > >> > > IMHO - when other than the "non FAB minimalistic user manager"
> > system
> > >> is
> > >> > > used. Airflow should just not have a number of functionalities at
> > all:
> > >> > >
> > >> > > * not have "users" or "roles" CLI groups at all - they should be
> > >> missing
> > >> > > * the
> > >> > >
> > >> > >
> > >>
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > >
> > <
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> >
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > ;>
> >
> >
> > >> <
> > >>
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> > >
> > <
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> >
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > ;>
> >
> >
> > >> >
> > >> > > and
> > >> > >
> > >> > >
> > >>
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > >
> > <
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> >
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > ;>
> >
> >
> > >> <
> > >>
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> > >
> > <
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> >
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > ;>
> >
> >
> > >> >
> > >> > > should return 404 NOT FOUND
> > >> > > * the whole "Security" menu should be GONE completely.
> > >> > >
> > >> > > All those components (and code related to those) should be moved
> to
> > >> FAB
> > >> > > minimalistic user manager (and removed from Airflow Core).
> > Eventually
> > >> maybe
> > >> > > even moved out of the Airflow repo altogether and moved to a FAB
> > user
> > >> > > manager add-on to airflow.
> > >> > >
> > >> > > This might seem a drastic change, but in fact - it's not drastic
> at
> > >> all. It
> > >> > > reflects what many of the Airflow users are doing now - they never
> > >> access
> > >> > > Security/Admin/Roles. Once they integrate with their LDAP or other
> > >> > > corporate directory, the Security UI, CLI and REST API are
> > essentially
> > >> > > useless (and not used).
> > >> > >
> > >> > > And it is precisely what we want to achieve - we want to delegate
> > the
> > >> > > user/role management completely out of Airflow. In fact - getting
> > rid
> > >> of
> > >> > > those is the only reason we want to implement the change. If we
> > >> **just**
> > >> > > replace the current FAB and keep all the complexity of managing
> the
> > >> users
> > >> > > /roles in Airflow, I'd say we have not achieved anything but
> adding
> > >> > > complexity.
> > >> > >
> > >> > > And - eventually - it makes the AIP-56 way *smaller*. We just need
> > to
> > >> make
> > >> > > sure that all the stuff for user management (including REST API,
> CLI
> > >> and
> > >> > > the UI) is moved to the "FAB minimalistic user manager" (and add
> > ways
> > >> to
> > >> > > plug-it in the core - possibly using existing plugins mechanisms
> > where
> > >> > > needed). We do not need to define new API for user/role
> management.
> > >> We only
> > >> > > need to describe the way how to check if the user has permission
> to
> > do
> > >> > > stuff.
> > >> > >
> > >> > > J.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com
> > <ma...@potiuk.com>
> > <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> > >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:
> > jarek@potiuk.com <ma...@potiuk.com>>>> wrote:
> > >> > >
> > >> > > > I promise to provide my feedback by the end of the week, sorry
> for
> > >> > > > delaying it, but we had some 2.6 preparation, branching, feature
> > >> flags
> > >> > > etc.
> > >> > > > also for AIP-44 and I am getting back on track with that one
> soon.
> > >> > > >
> > >> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
> > >> <shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:
> > shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:
> > shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:
> > shubhx@amazon.com.inva <ma...@amazon.com.inva>>>lid
> > >> > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > >> Bumping this up for feedback!
> > >> > > >>
> > >> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite busy,
> > but
> > >> we
> > >> > > >> would greatly appreciate your feedback here. Your inputs could
> > >> help us
> > >> > > >> improve the proposal and move forward with multi-tenancy work.
> > >> > > >>
> > >> > > >> Cheers,
> > >> > > >> Shubham
> > >> > > >>
> > >> > > >>
> > >> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent"
> <vincbeck@amazon.com.INVA
> > <ma...@amazon.com.INVA>
> > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
> > >> > > >> <mailto:vincbeck@amazon.com.INVA <mailto:
> > vincbeck@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <mailto:
> > vincbeck@amazon.com.INVA>>
> > <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > <mailto:vincbeck@amazon.com.INVA <mailto:vincbeck@amazon.com.INVA
> >>>>LID>
> > >> wrote:
> > >> > > >>
> > >> > > >>
> > >> > > >> CAUTION: This email originated from outside of the
> organization.
> > >> Do not
> > >> > > >> click links or open attachments unless you can confirm the
> sender
> > >> and
> > >> > > know
> > >> > > >> the content is safe.
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> Hi all,
> > >> > > >>
> > >> > > >>
> > >> > > >> I would like to get your feedback on an AIP I wrote:
> "Extensible
> > >> user
> > >> > > >> management". This AIP is a follow-up on a discussion we had in
> > this
> > >> > > email
> > >> > > >> list on the multi tenancy topic. I decided to create a new
> email
> > >> thread
> > >> > > >> because I feel the topic had diverged a bit from the original
> > topic
> > >> > > (multi
> > >> > > >> tenancy).
> > >> > > >>
> > >> > > >>
> > >> > > >> As a summary, the purpose of this AIP is to extract the user
> > >> management
> > >> > > >> from core Airflow by introducing a user manager interface in
> the
> > >> core
> > >> > > >> Airflow that can be extended by any provider package who want
> to
> > >> support
> > >> > > >> user management natively. As a result, if this AIP gets
> approved
> > >> and go
> > >> > > >> through, the multi tenancy feature will be implemented as a
> > second
> > >> step
> > >> > > in
> > >> > > >> a new user manager and not in Airflow directly.
> > >> > > >>
> > >> > > >>
> > >> > > >> As always, feel very free to give your opinion on this email
> > >> thread or
> > >> > > on
> > >> > > >> the AIP by adding comments.
> > >> > > >>
> > >> > > >>
> > >> > > >> References:
> > >> > > >> - AIP:
> > >> > > >>
> > >> > >
> > >>
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > >
> > <
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > ;>
> >
> >
> > >> <
> > >>
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > >
> > <
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > ;>
> >
> >
> > >> >
> > >> > > >> <
> > >> > > >>
> > >> > >
> > >>
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > >
> > <
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > ;>
> >
> >
> > >> <
> > >>
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> > >
> > <
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > ;>
> >
> >
> > >> >
> > >> > > >> >
> > >> > > >> - Initial email list discussion:
> > >> > > >>
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61
> > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> > https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> ;>
> > <
> > >> > > >>
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> > <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> > ;>
> > >> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> ;>
> > <
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> ;>
> > <
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> ;>
> > <
> >
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt;>
> > ;>
> >
> >
> > >> > > >>
> > >> > > >>
> > >> > > >> Vincent
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > --
> > >> > Bolke de Bruin
> > >> > bdbruin@gmail.com <ma...@gmail.com> <mailto:
> > bdbruin@gmail.com <ma...@gmail.com>> <mailto:bdbruin@gmail.com
> > <ma...@gmail.com>
> > <mailto:bdbruin@gmail.com <ma...@gmail.com>>>
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> ---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org> <mailto:
> > dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>>
> > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> <mailto:
> > dev-help@airflow.apache.org> <mailto:
> > dev-help@airflow.apache.org <ma...@airflow.apache.org>>
> > >> >
> > >>
> > >>
> >
> >
> >
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>
> > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> > dev-help@airflow.apache.org>
> >
> >
> >
> >
>
--
--
Bolke de Bruin
bdbruin@gmail.com
Re: AIP-56 Extensible user management
Posted by Jarek Potiuk <ja...@potiuk.com>.
>
>
> I am very open to change the name of "User Manager" but I do not think we
> should call it "Security Manager". "Security Manager" is the current name
> and I think this is a very bad name because: it is very generic and does
> not say much about what it is actually doing. By reading "Security Manager"
> I would not think this is where users and roles are managed. The main goal
> of AIP-56 is removing the whole user management part from core Airflow to
> providers. Keeping the name "Security Manager" would, I think, leave a
> legacy for all future user managers (or whatever name we use) on
> Flask-AppBuilder (FAB) and that's exactly what I want to avoid. I chose the
> name “User Manager” but I think it is simple, the main goal of this new
> module is to manage users and access for them.
>
>
How about *AuthManager* ? (Auth standing for both Authentication and
Authorisation). I do agree "Security" is much bigger promise thant what we
are implementing here :).
> > is_authorized should include context so that it becomes possible the
> determine whether a user has access to a resource based on for example the
> location the user is at. Or in which environment. It is also useful for
> auditing. Non implementing security managers can ignore it.
>
> I agree, that can be indeed a good improvement. Although I have to say I
> am pretty confident the AIPs I defined in the API will change while
> implementing them and over time. Thus, another option would be to add this
> context when needed. I do not think (I might be wrong) features such as "
> determine whether a user has access to a resource based on for example the
> location the user is at. Or in which environment" currently exist in
> Airflow. The goal of the AIP is not to implement new features but to have
> the same features as we have today with this separation from core Airflow
> to providers. Again, I am not saying this is a wrong idea (and I actually
> think this is a good one), I just think if this feature does not exist in
> Airflow today, we should add that later.
>
I think Bolke is right: we should add information that we will pass context
and broadly what will be in. We can leave details for the implementation
time but I think the context should be there. We should add what will be
part of the context:
* action
* dag_id
* task_id
* labels for the dag
* location in the DAG_FOLDER
* connection_id (when we change connection)
* variable-id (when we change variable)
* ...
This should be complete enough to make any kind of decisions. Probably it
will also evolve over time in the future versions, but we need to have
something to start with that we think will be comprehensive and complete
enough to implement usable AuthManager(s) (providing that the name will
stick).
> > What is the need of post_login? Why would you want to store the access
> token? The security manager should, imho, do this on its own: it executes
> the login flow.
>
> The access token is needed to call any API from the user manager to the
> third party backend (KeyCloak, ...). How would you call is_authorized()
> without it?
>
Yeah. we need to map whatever authentication information comes from outside
(usually a token coming from external authentication information) with
(likely?) flask session in Airflow and I **think** the best is that it
should be kept in AIrflow's DB the session in Airflow. I consider this a
bit of an implementation detail, but maybe this needs a bit more hashing ou
maybe others can also comment here..
>
> > Why is get_tab_configuration special? We have this possibility in
> standard “plugins”. I suggest using that framework
>
> I am not familiar with plugins. I'll take a look at it, thanks for
> bringing it up.
>
Yeah. The plugin mechanism is indeed foreseen for that - and the minimal
FAB implementation should indeed implement plugin interface to add the
current Admin menu - that was what I had in mind by the current "Security"
tab - it can **just** be added using the current plugin mechanism.
>
> > What is the need for “get_user_name”? Are we going to invent our own
> Framework? Otherwise Flask might work?
>
> get_user_name is needed for the UI. On the top right corner of Airflow
> console, you can see your name. This is why it is needed. The only goal of
> this API is to retrieve your user name from the user manager. That's it.
> Flask will then use this API to get the user name and display it in the UI.
> My experience in Flask is pretty limited so I might miss something but the
> way I see it is, you need a way to retrieve your user name from whatever
> user manager you are using. This is the way.
>
> I hope that helps,
> Vincent
>
> On 2023-05-15, 5:30 PM, "Bolke de Bruin" <bdbruin@gmail.com <mailto:
> bdbruin@gmail.com>> wrote:
>
>
>
> I think think the name “User Manager” is a misnomer and it should just be
> called “Security Manager”. It was confusing the hell out of me when reading
> the AIP and I was convinced you wanted to fully manage e.g. keycloak
> provided users inside Airflow (CRUD). What you are doing but aren’t exactly
> saying:
>
>
> 1 - Define API that a Security Plugin should adhere to (AuthZ/AuthN)
> 2 - Have a default Security Plugin based on FAB
>
>
> Points of improvement:
>
>
> * is_authorized should include context so that it becomes possible the
> determine whether a user has access to a resource based on for example the
> location the user is at. Or in which environment. It is also useful for
> auditing. Non implementing security managers can ignore it.
> * I’d prefer some kind of ’standards’ based API. Like using Flask’s way of
> registering endpoints or something along those lines. See also questions
> below.
>
>
> Questions:
>
>
> * What is the need of post_login? Why would you want to store the access
> token? The security manager should, imho, do this on its own: it executes
> the login flow.
> * Why is get_tab_configuration special? We have this possibility in
> standard “plugins”. I suggest using that framework
> * What is the need for “get_user_name”? Are we going to invent our own
> Framework? Otherwise Flask might work?
>
>
>
>
> Cheers
> Bolke
>
>
> On 9 May 2023 at 19:55:32, Beck, Vincent (vincbeck@amazon.com.inva
> <ma...@amazon.com.inva>lid)
> wrote:
>
>
> Hello all,
>
>
> I would like to start voting on this one so please add your comments on the
> API or by replying here if you're interested, you still have time to do it
> :)
>
>
> AIP:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
>
>
> Vincent
>
>
> On 2023-05-03, 8:03 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> jarek@potiuk.com> <mailto:
> jarek@potiuk.com <ma...@potiuk.com>>> wrote:
>
>
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
>
>
>
>
>
>
> Alsoo to add (and make more concrete) to what it could mean at the
> implementation level.
>
>
>
>
> It means the for our API endpoints and views is that instead of
> @auth.hasaccess (for example here
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680 <
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <
> https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>
> ;>)
> where it check for permissions assigned to current role, airflow would
> entirely delegate the decision to the 'user management plugin'.
>
>
>
>
> We would simply delegate the decision to whatever plugin we have
> configured. In case FAB pługin is used, it would do exactly the same checks
> as today (and FAB pługin would contribute the 'Security' menu the same way
> other plugins can contribute their own (as well as CLI commands and
> user/role APIs - we would have to add a mechanism where plugins could
> contribute CLIs and APIs on their own, but that should be easy). So if you
> have FAB pługin chosen as default, nothing changes for the users comparing
> to today. It would not require bumping Airflow Major version because it
> would be backwards compatible. However similarly as providers it would move
> the code of FAB plugins and roles out of the core of Airflow.
>
>
>
>
> We could have keycloak 'reference' implementation where we would come up
> with very opinionated approach where we have simple admin/user roles
> (matching roughly default set of roles and permissions we currently have
> with FAB). The big difference here will be that the user/role APIs, views,
> CLIs will be gone entirely. And we could add there the concept of
> 'tenants' or rather 'teams' giving access to 'dag groups' and implement
> decisions ob authorisation based on 'group' assignment for dag and user.
> This is the main goal of this change to make it possible and easy without
> having to mess with FAB permission and Role model which is completely not
> suitable to manage 'groups of resources of the same type' with separate set
> of permissions (think having few separate teams or tenants - each of them
> wanting access to separate set of DAGs).
>
>
>
>
> Currently if an SSO/enterprise user wants to plug in their authn/authz with
> group access - they have to deal with all that including cumbersome syncing
> of permissions for individual DAGs matching them to users separately - just
> to satisfy FAB permission model lacking this feature and flexibility.
>
>
>
>
> No more need for that if we go the direction described here.
>
>
>
>
> And that's basically it for what we would implement in Airflow
> out-of-the-box: FAB minimal implementation + be keycloak opinionated
> reference implementation with simple teams/tenant support
>
>
>
>
> This also makes it super flexible for those who want more flexibility and
> do not want to use our opinionated keycloak plugin. They will be entirely
> free to implement their own ways - as flexible or as opinionated they want
> - without having to messwith permission syncing and the role model of FAB.
> Nor keycloak integration. They could go with their own.
>
>
>
>
> And with that - sky is the limit. The decision to access particular dag
> could be made based on folder where the dag is from, id of dag, or dag
> label or whatever other parameter (as long as we pass all that information
> to the plugin). You could even - if you really want - make a decision to
> allow certain users (and only them) to be able to clear individual tasks,
> or tasks where specific operator types are used. For example you could
> decide if you really want to only allow th google cloud admin users to
> clear tasks that are operators coming from Google Provider (why not?). Or
> only allow to that in certain time windows, or .... whatever.
>
>
>
>
> It's extremely flexible and powerful - but we will not have to deal with
> all the complexity in the community - we could delegate it to those who
> mange the deployment or manage deployment of Airflow as a service and
> decide they want to integrate it with the service authorisation that is
> already there - think direct integratio with IAM ON AWS or GCP. And if we
> just provide a reference, opinionated implementation for keycloak with
> tenant support and legacy/minimal FAB as optional plugins, that should also
> 'good enough' for those who do not want to implement their own plugin
> implementation.
>
>
>
>
>
>
>
>
> J.
>
>
>
>
>
>
>
>
> śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <jarek@potiuk.com <mailto:
> jarek@potiuk.com> <mailto:
> jarek@potiuk.com <ma...@potiuk.com>>> napisał:
>
>
>
>
> > > I don't think we can do that: we still need to consider users when they
> > are just getting started out with Airflow -- if we only cater to giant
> > enterprises with SSO and a central directory we remove our ability for
> > individual or small teams/companies to get started with Airflow.
> >
> > Let me clarify this because I think we are talking about the same thing.
> >
> > I **think** having the "minimal FAB user manager implementation" as the
> > "default" option when you install Airflow serves the purpose you mention
> > Ash - if we do it the way that User/Role management is coming together
> with
> > FAB implementation (enabled by default for the initial installation).
> What
> > I am saying is that it should not be part of the "core airflow" in the
> > sense that one could live without it (specifically in those big
> SSO/Central
> > directory case).
> > In my vision those two cases are separated, but the "minimal
> > implementation" when enabled should act and behave "as if" nothing
> changed
> > compared to the current way it looks and behaves.
> >
> > J.
> >
> >
> > On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <ash@apache.org
> <ma...@apache.org> <mailto:
> ash@apache.org <ma...@apache.org>>> wrote:
> >
> >> > - I am fine with (and actually I like it better) removing User and
> Role
> >> related Rest APIs and CLI commands from Airflow. That will indeed make
> the
> >> AIP way simpler and shorter
> >>
> >> I don't think we can do that: we still need to consider users when they
> >> are just getting started out with Airflow -- if we only cater to giant
> >> enterprises with SSO and a central directory we remove our ability for
> >> individual or small teams/companies to get started with Airflow.
> >> On May 2 2023, at 8:23 pm, Beck, Vincent <vincbeck@amazon.com.INVA
> <ma...@amazon.com.INVA>
> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>LID>
> >> wrote:
> >> > Thank you Jarek and Bolke for taking time to read and provide feedback
> >> to the AIP, much appreciated! Multiple points here:
> >> >
> >> > - I am fine with (and actually I like it better) removing User and
> Role
> >> related Rest APIs and CLI commands from Airflow. That will indeed make
> the
> >> AIP way simpler and shorter. As I explained in the AIP comments, I am
> >> suggesting though to leave an option to user managers to define
> additional
> >> REST APIs and CLI commands if they need. By default no additional REST
> API
> >> and CLI command would be defined but user managers would have the option
> to
> >> override it. FAB user manager would be an example of user manager
> >> overriding such methods. In any case, if additional Rest API or CLI
> command
> >> is defined, this would be done in user managers (as opposed to core
> >> Airflow).
> >> > - "the whole "Security" menu should be GONE completely".
> >> > > I agree with this when the user manager is not the FAB one, however
> >> we need to introduce a new one so Admins can easily go the external user
> >> manager console (e.g. KeyCloak, ...). This new tab would only be a
> >> link/redirect to the external console.
> >> >
> >> > - "All those components (and code related to those) should be moved to
> >> FAB minimalistic user manager (and removed from Airflow Core)".
> >> > > 100%. This is the main principle of this AIP, anything related to
> >> user management will be moved to user managers.
> >> >
> >> > - "AuthN and AuthZ should be completely removed from Airflow".
> >> > > 100%.
> >> >
> >> > - "So for a nice out-of-the-box experience (pip install
> apache-airflow)
> >> a lightweight User Manager could exist as a side project and have some
> >> menus via the plugin system until something better comes along.".
> >> > > I agree with you on the long term. Though, as a first iteration I
> >> think it is better to define a user manager using FAB as default user
> >> manager. The benefit of doing so is to have a backward compatible
> >> transition between before AIP and after AIP. This also simplifies the
> AIP
> >> since it is easier to move files from core Airflow to user managers than
> >> creating something new. Creating a new security model would be a
> challenge
> >> too. However, once the AIP implemented and merged, we can easily replace
> >> this out-of-the-box FAB user manager by something "better" if the
> communty
> >> thinks this is the right way to go.
> >> >
> >> > As always, feel free to give me your thoughts.
> >> > Vincent
> >> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <mailto:
> bdbruin@gmail.com> <mailto:
> bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> >> bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com
> <ma...@gmail.com>>>> wrote:
> >> > I'm not sure if I read the AIP correctly. I think I concur with Jarek
> >> here.
> >> >
> >> > AuthN and AuthZ should be completely removed from Airflow. Access
> >> > management should be outsourced to something like Open Policy Agent.
> In
> >> > other words there should be no user management at all within Airflow.
> >> This
> >> > removes the need to "sync" users (and thus become outdated) and
> reduces
> >> the
> >> > attack surface drastically. So for a nice out-of-the-box experience
> (pip
> >> > install apache-airflow) a lightweight User Manager could exist as a
> side
> >> > project and have some menus via the plugin system until something
> better
> >> > comes along. Keycloak is great, a bit heavy for us to package
> >> > unfortunately.
> >> >
> >> >
> >> > FAB's security model is very non-standard and requires 'unnatural'
> >> mappings
> >> > in an enterprise context. It has served us well, but I would say good
> >> > riddance.
> >> >
> >> >
> >> > Bolke
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>
> <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:
> jarek@potiuk.com <ma...@potiuk.com>>>>:
> >> >
> >> > > I had a look and I liked what I saw for how we should integrate the
> >> new
> >> > > user manager for getting and checking the access :).
> >> > >
> >> > > But I think the AIP proposal is not bold enough when it comes to the
> >> user
> >> > > management part. This is one big comment - we should simplify it
> even
> >> > > further and cut more deeply.
> >> > >
> >> > > One of the big changes I think is super important with extensible
> user
> >> > > management is that we should delegate even more to the external
> >> systems
> >> > > managing users. The AIP attempts to map the current API/CLI/UI for
> >> managing
> >> > > the users and roles to the new external user management services -
> >> but it
> >> > > is IMHO absolutely not needed.
> >> > >
> >> > > As I see the extensible user management is that when you choose
> >> something
> >> > > else than the legacy FAB minimalist user management. Airflow should
> >> become
> >> > > just "user" of that user/role information managed elsewhere. It
> >> should not
> >> > > provide ANY option to see and manage the users or roles, it should
> >> merely
> >> > > read the information and give access to the users for appropriate
> >> actions,
> >> > > but then all the role/user management should happen outside of
> >> Airflow - in
> >> > > the system that is dedicated to manage those.
> >> > >
> >> > > IMHO - when other than the "non FAB minimalistic user manager"
> system
> >> is
> >> > > used. Airflow should just not have a number of functionalities at
> all:
> >> > >
> >> > > * not have "users" or "roles" CLI groups at all - they should be
> >> missing
> >> > > * the
> >> > >
> >> > >
> >>
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> >
> <
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> ;>
>
>
> >> <
> >>
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> >
> <
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> ;>
>
>
> >> >
> >> > > and
> >> > >
> >> > >
> >>
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> >
> <
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> ;>
>
>
> >> <
> >>
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> >
> <
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> ;>
>
>
> >> >
> >> > > should return 404 NOT FOUND
> >> > > * the whole "Security" menu should be GONE completely.
> >> > >
> >> > > All those components (and code related to those) should be moved to
> >> FAB
> >> > > minimalistic user manager (and removed from Airflow Core).
> Eventually
> >> maybe
> >> > > even moved out of the Airflow repo altogether and moved to a FAB
> user
> >> > > manager add-on to airflow.
> >> > >
> >> > > This might seem a drastic change, but in fact - it's not drastic at
> >> all. It
> >> > > reflects what many of the Airflow users are doing now - they never
> >> access
> >> > > Security/Admin/Roles. Once they integrate with their LDAP or other
> >> > > corporate directory, the Security UI, CLI and REST API are
> essentially
> >> > > useless (and not used).
> >> > >
> >> > > And it is precisely what we want to achieve - we want to delegate
> the
> >> > > user/role management completely out of Airflow. In fact - getting
> rid
> >> of
> >> > > those is the only reason we want to implement the change. If we
> >> **just**
> >> > > replace the current FAB and keep all the complexity of managing the
> >> users
> >> > > /roles in Airflow, I'd say we have not achieved anything but adding
> >> > > complexity.
> >> > >
> >> > > And - eventually - it makes the AIP-56 way *smaller*. We just need
> to
> >> make
> >> > > sure that all the stuff for user management (including REST API, CLI
> >> and
> >> > > the UI) is moved to the "FAB minimalistic user manager" (and add
> ways
> >> to
> >> > > plug-it in the core - possibly using existing plugins mechanisms
> where
> >> > > needed). We do not need to define new API for user/role management.
> >> We only
> >> > > need to describe the way how to check if the user has permission to
> do
> >> > > stuff.
> >> > >
> >> > > J.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>
> <mailto:jarek@potiuk.com <ma...@potiuk.com>>
> >> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:
> jarek@potiuk.com <ma...@potiuk.com>>>> wrote:
> >> > >
> >> > > > I promise to provide my feedback by the end of the week, sorry for
> >> > > > delaying it, but we had some 2.6 preparation, branching, feature
> >> flags
> >> > > etc.
> >> > > > also for AIP-44 and I am getting back on track with that one soon.
> >> > > >
> >> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
> >> <shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:
> shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:
> shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:
> shubhx@amazon.com.inva <ma...@amazon.com.inva>>>lid
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > >> Bumping this up for feedback!
> >> > > >>
> >> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite busy,
> but
> >> we
> >> > > >> would greatly appreciate your feedback here. Your inputs could
> >> help us
> >> > > >> improve the proposal and move forward with multi-tenancy work.
> >> > > >>
> >> > > >> Cheers,
> >> > > >> Shubham
> >> > > >>
> >> > > >>
> >> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA
> <ma...@amazon.com.INVA>
> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
> >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
> >> > > >> <mailto:vincbeck@amazon.com.INVA <mailto:
> vincbeck@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <mailto:
> vincbeck@amazon.com.INVA>>
> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>>LID>
> >> wrote:
> >> > > >>
> >> > > >>
> >> > > >> CAUTION: This email originated from outside of the organization.
> >> Do not
> >> > > >> click links or open attachments unless you can confirm the sender
> >> and
> >> > > know
> >> > > >> the content is safe.
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >> Hi all,
> >> > > >>
> >> > > >>
> >> > > >> I would like to get your feedback on an AIP I wrote: "Extensible
> >> user
> >> > > >> management". This AIP is a follow-up on a discussion we had in
> this
> >> > > email
> >> > > >> list on the multi tenancy topic. I decided to create a new email
> >> thread
> >> > > >> because I feel the topic had diverged a bit from the original
> topic
> >> > > (multi
> >> > > >> tenancy).
> >> > > >>
> >> > > >>
> >> > > >> As a summary, the purpose of this AIP is to extract the user
> >> management
> >> > > >> from core Airflow by introducing a user manager interface in the
> >> core
> >> > > >> Airflow that can be extended by any provider package who want to
> >> support
> >> > > >> user management natively. As a result, if this AIP gets approved
> >> and go
> >> > > >> through, the multi tenancy feature will be implemented as a
> second
> >> step
> >> > > in
> >> > > >> a new user manager and not in Airflow directly.
> >> > > >>
> >> > > >>
> >> > > >> As always, feel very free to give your opinion on this email
> >> thread or
> >> > > on
> >> > > >> the AIP by adding comments.
> >> > > >>
> >> > > >>
> >> > > >> References:
> >> > > >> - AIP:
> >> > > >>
> >> > >
> >>
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> <
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> ;>
>
>
> >> <
> >>
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> <
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> ;>
>
>
> >> >
> >> > > >> <
> >> > > >>
> >> > >
> >>
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> <
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> ;>
>
>
> >> <
> >>
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> <
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> ;>
>
>
> >> >
> >> > > >> >
> >> > > >> - Initial email list discussion:
> >> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61
> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>>
> <
> >> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>
> ;>
> >> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>>
> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>>
> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt;>
> ;>
>
>
> >> > > >>
> >> > > >>
> >> > > >> Vincent
> >> > > >>
> >> > > >>
> >> > > >>
> >> > > >>
> >> > >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > --
> >> > Bolke de Bruin
> >> > bdbruin@gmail.com <ma...@gmail.com> <mailto:
> bdbruin@gmail.com <ma...@gmail.com>> <mailto:bdbruin@gmail.com
> <ma...@gmail.com>
> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>
> >> >
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org> <mailto:
> dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>>
> >> > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org> <mailto:
> dev-help@airflow.apache.org <ma...@airflow.apache.org>>
> >> >
> >>
> >>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>
> For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org>
>
>
>
>
Re: AIP-56 Extensible user management
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
Hi Bolke,
Thanks for your feedbacks! Here are my points:
> I think the name “User Manager” is a misnomer and it should just be called “Security Manager”.
I am very open to change the name of "User Manager" but I do not think we should call it "Security Manager". "Security Manager" is the current name and I think this is a very bad name because: it is very generic and does not say much about what it is actually doing. By reading "Security Manager" I would not think this is where users and roles are managed. The main goal of AIP-56 is removing the whole user management part from core Airflow to providers. Keeping the name "Security Manager" would, I think, leave a legacy for all future user managers (or whatever name we use) on Flask-AppBuilder (FAB) and that's exactly what I want to avoid. I chose the name “User Manager” but I think it is simple, the main goal of this new module is to manage users and access for them.
> is_authorized should include context so that it becomes possible the determine whether a user has access to a resource based on for example the location the user is at. Or in which environment. It is also useful for auditing. Non implementing security managers can ignore it.
I agree, that can be indeed a good improvement. Although I have to say I am pretty confident the AIPs I defined in the API will change while implementing them and over time. Thus, another option would be to add this context when needed. I do not think (I might be wrong) features such as " determine whether a user has access to a resource based on for example the location the user is at. Or in which environment" currently exist in Airflow. The goal of the AIP is not to implement new features but to have the same features as we have today with this separation from core Airflow to providers. Again, I am not saying this is a wrong idea (and I actually think this is a good one), I just think if this feature does not exist in Airflow today, we should add that later.
> What is the need of post_login? Why would you want to store the access token? The security manager should, imho, do this on its own: it executes the login flow.
The access token is needed to call any API from the user manager to the third party backend (KeyCloak, ...). How would you call is_authorized() without it?
> Why is get_tab_configuration special? We have this possibility in standard “plugins”. I suggest using that framework
I am not familiar with plugins. I'll take a look at it, thanks for bringing it up.
> What is the need for “get_user_name”? Are we going to invent our own Framework? Otherwise Flask might work?
get_user_name is needed for the UI. On the top right corner of Airflow console, you can see your name. This is why it is needed. The only goal of this API is to retrieve your user name from the user manager. That's it. Flask will then use this API to get the user name and display it in the UI. My experience in Flask is pretty limited so I might miss something but the way I see it is, you need a way to retrieve your user name from whatever user manager you are using. This is the way.
I hope that helps,
Vincent
On 2023-05-15, 5:30 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com>> wrote:
I think think the name “User Manager” is a misnomer and it should just be
called “Security Manager”. It was confusing the hell out of me when reading
the AIP and I was convinced you wanted to fully manage e.g. keycloak
provided users inside Airflow (CRUD). What you are doing but aren’t exactly
saying:
1 - Define API that a Security Plugin should adhere to (AuthZ/AuthN)
2 - Have a default Security Plugin based on FAB
Points of improvement:
* is_authorized should include context so that it becomes possible the
determine whether a user has access to a resource based on for example the
location the user is at. Or in which environment. It is also useful for
auditing. Non implementing security managers can ignore it.
* I’d prefer some kind of ’standards’ based API. Like using Flask’s way of
registering endpoints or something along those lines. See also questions
below.
Questions:
* What is the need of post_login? Why would you want to store the access
token? The security manager should, imho, do this on its own: it executes
the login flow.
* Why is get_tab_configuration special? We have this possibility in
standard “plugins”. I suggest using that framework
* What is the need for “get_user_name”? Are we going to invent our own
Framework? Otherwise Flask might work?
Cheers
Bolke
On 9 May 2023 at 19:55:32, Beck, Vincent (vincbeck@amazon.com.inva <ma...@amazon.com.inva>lid)
wrote:
Hello all,
I would like to start voting on this one so please add your comments on the
API or by replying here if you're interested, you still have time to do it
:)
AIP:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
Vincent
On 2023-05-03, 8:03 AM, "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com> <mailto:
jarek@potiuk.com <ma...@potiuk.com>>> wrote:
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you can confirm the sender and know
the content is safe.
Alsoo to add (and make more concrete) to what it could mean at the
implementation level.
It means the for our API endpoints and views is that instead of
@auth.hasaccess (for example here
https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680 <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <
https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680> <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>>)
where it check for permissions assigned to current role, airflow would
entirely delegate the decision to the 'user management plugin'.
We would simply delegate the decision to whatever plugin we have
configured. In case FAB pługin is used, it would do exactly the same checks
as today (and FAB pługin would contribute the 'Security' menu the same way
other plugins can contribute their own (as well as CLI commands and
user/role APIs - we would have to add a mechanism where plugins could
contribute CLIs and APIs on their own, but that should be easy). So if you
have FAB pługin chosen as default, nothing changes for the users comparing
to today. It would not require bumping Airflow Major version because it
would be backwards compatible. However similarly as providers it would move
the code of FAB plugins and roles out of the core of Airflow.
We could have keycloak 'reference' implementation where we would come up
with very opinionated approach where we have simple admin/user roles
(matching roughly default set of roles and permissions we currently have
with FAB). The big difference here will be that the user/role APIs, views,
CLIs will be gone entirely. And we could add there the concept of
'tenants' or rather 'teams' giving access to 'dag groups' and implement
decisions ob authorisation based on 'group' assignment for dag and user.
This is the main goal of this change to make it possible and easy without
having to mess with FAB permission and Role model which is completely not
suitable to manage 'groups of resources of the same type' with separate set
of permissions (think having few separate teams or tenants - each of them
wanting access to separate set of DAGs).
Currently if an SSO/enterprise user wants to plug in their authn/authz with
group access - they have to deal with all that including cumbersome syncing
of permissions for individual DAGs matching them to users separately - just
to satisfy FAB permission model lacking this feature and flexibility.
No more need for that if we go the direction described here.
And that's basically it for what we would implement in Airflow
out-of-the-box: FAB minimal implementation + be keycloak opinionated
reference implementation with simple teams/tenant support
This also makes it super flexible for those who want more flexibility and
do not want to use our opinionated keycloak plugin. They will be entirely
free to implement their own ways - as flexible or as opinionated they want
- without having to messwith permission syncing and the role model of FAB.
Nor keycloak integration. They could go with their own.
And with that - sky is the limit. The decision to access particular dag
could be made based on folder where the dag is from, id of dag, or dag
label or whatever other parameter (as long as we pass all that information
to the plugin). You could even - if you really want - make a decision to
allow certain users (and only them) to be able to clear individual tasks,
or tasks where specific operator types are used. For example you could
decide if you really want to only allow th google cloud admin users to
clear tasks that are operators coming from Google Provider (why not?). Or
only allow to that in certain time windows, or .... whatever.
It's extremely flexible and powerful - but we will not have to deal with
all the complexity in the community - we could delegate it to those who
mange the deployment or manage deployment of Airflow as a service and
decide they want to integrate it with the service authorisation that is
already there - think direct integratio with IAM ON AWS or GCP. And if we
just provide a reference, opinionated implementation for keycloak with
tenant support and legacy/minimal FAB as optional plugins, that should also
'good enough' for those who do not want to implement their own plugin
implementation.
J.
śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com> <mailto:
jarek@potiuk.com <ma...@potiuk.com>>> napisał:
> > I don't think we can do that: we still need to consider users when they
> are just getting started out with Airflow -- if we only cater to giant
> enterprises with SSO and a central directory we remove our ability for
> individual or small teams/companies to get started with Airflow.
>
> Let me clarify this because I think we are talking about the same thing.
>
> I **think** having the "minimal FAB user manager implementation" as the
> "default" option when you install Airflow serves the purpose you mention
> Ash - if we do it the way that User/Role management is coming together
with
> FAB implementation (enabled by default for the initial installation).
What
> I am saying is that it should not be part of the "core airflow" in the
> sense that one could live without it (specifically in those big
SSO/Central
> directory case).
> In my vision those two cases are separated, but the "minimal
> implementation" when enabled should act and behave "as if" nothing
changed
> compared to the current way it looks and behaves.
>
> J.
>
>
> On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org> <mailto:
ash@apache.org <ma...@apache.org>>> wrote:
>
>> > - I am fine with (and actually I like it better) removing User and
Role
>> related Rest APIs and CLI commands from Airflow. That will indeed make
the
>> AIP way simpler and shorter
>>
>> I don't think we can do that: we still need to consider users when they
>> are just getting started out with Airflow -- if we only cater to giant
>> enterprises with SSO and a central directory we remove our ability for
>> individual or small teams/companies to get started with Airflow.
>> On May 2 2023, at 8:23 pm, Beck, Vincent <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
<mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>LID>
>> wrote:
>> > Thank you Jarek and Bolke for taking time to read and provide feedback
>> to the AIP, much appreciated! Multiple points here:
>> >
>> > - I am fine with (and actually I like it better) removing User and
Role
>> related Rest APIs and CLI commands from Airflow. That will indeed make
the
>> AIP way simpler and shorter. As I explained in the AIP comments, I am
>> suggesting though to leave an option to user managers to define
additional
>> REST APIs and CLI commands if they need. By default no additional REST
API
>> and CLI command would be defined but user managers would have the option
to
>> override it. FAB user manager would be an example of user manager
>> overriding such methods. In any case, if additional Rest API or CLI
command
>> is defined, this would be done in user managers (as opposed to core
>> Airflow).
>> > - "the whole "Security" menu should be GONE completely".
>> > > I agree with this when the user manager is not the FAB one, however
>> we need to introduce a new one so Admins can easily go the external user
>> manager console (e.g. KeyCloak, ...). This new tab would only be a
>> link/redirect to the external console.
>> >
>> > - "All those components (and code related to those) should be moved to
>> FAB minimalistic user manager (and removed from Airflow Core)".
>> > > 100%. This is the main principle of this AIP, anything related to
>> user management will be moved to user managers.
>> >
>> > - "AuthN and AuthZ should be completely removed from Airflow".
>> > > 100%.
>> >
>> > - "So for a nice out-of-the-box experience (pip install
apache-airflow)
>> a lightweight User Manager could exist as a side project and have some
>> menus via the plugin system until something better comes along.".
>> > > I agree with you on the long term. Though, as a first iteration I
>> think it is better to define a user manager using FAB as default user
>> manager. The benefit of doing so is to have a backward compatible
>> transition between before AIP and after AIP. This also simplifies the
AIP
>> since it is easier to move files from core Airflow to user managers than
>> creating something new. Creating a new security model would be a
challenge
>> too. However, once the AIP implemented and merged, we can easily replace
>> this out-of-the-box FAB user manager by something "better" if the
communty
>> thinks this is the right way to go.
>> >
>> > As always, feel free to give me your thoughts.
>> > Vincent
>> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com> <mailto:
bdbruin@gmail.com <ma...@gmail.com>> <mailto:
>> bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>> wrote:
>> > I'm not sure if I read the AIP correctly. I think I concur with Jarek
>> here.
>> >
>> > AuthN and AuthZ should be completely removed from Airflow. Access
>> > management should be outsourced to something like Open Policy Agent.
In
>> > other words there should be no user management at all within Airflow.
>> This
>> > removes the need to "sync" users (and thus become outdated) and
reduces
>> the
>> > attack surface drastically. So for a nice out-of-the-box experience
(pip
>> > install apache-airflow) a lightweight User Manager could exist as a
side
>> > project and have some menus via the plugin system until something
better
>> > comes along. Keycloak is great, a bit heavy for us to package
>> > unfortunately.
>> >
>> >
>> > FAB's security model is very non-standard and requires 'unnatural'
>> mappings
>> > in an enterprise context. It has served us well, but I would say good
>> > riddance.
>> >
>> >
>> > Bolke
>> >
>> >
>> >
>> >
>> >
>> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>
<mailto:jarek@potiuk.com <ma...@potiuk.com>>
>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>>:
>> >
>> > > I had a look and I liked what I saw for how we should integrate the
>> new
>> > > user manager for getting and checking the access :).
>> > >
>> > > But I think the AIP proposal is not bold enough when it comes to the
>> user
>> > > management part. This is one big comment - we should simplify it
even
>> > > further and cut more deeply.
>> > >
>> > > One of the big changes I think is super important with extensible
user
>> > > management is that we should delegate even more to the external
>> systems
>> > > managing users. The AIP attempts to map the current API/CLI/UI for
>> managing
>> > > the users and roles to the new external user management services -
>> but it
>> > > is IMHO absolutely not needed.
>> > >
>> > > As I see the extensible user management is that when you choose
>> something
>> > > else than the legacy FAB minimalist user management. Airflow should
>> become
>> > > just "user" of that user/role information managed elsewhere. It
>> should not
>> > > provide ANY option to see and manage the users or roles, it should
>> merely
>> > > read the information and give access to the users for appropriate
>> actions,
>> > > but then all the role/user management should happen outside of
>> Airflow - in
>> > > the system that is dedicated to manage those.
>> > >
>> > > IMHO - when other than the "non FAB minimalistic user manager"
system
>> is
>> > > used. Airflow should just not have a number of functionalities at
all:
>> > >
>> > > * not have "users" or "roles" CLI groups at all - they should be
>> missing
>> > > * the
>> > >
>> > >
>>
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
<
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>>
>> <
>>
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
<
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>>
>> >
>> > > and
>> > >
>> > >
>>
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
<
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>>
>> <
>>
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
<
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role> <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>>
>> >
>> > > should return 404 NOT FOUND
>> > > * the whole "Security" menu should be GONE completely.
>> > >
>> > > All those components (and code related to those) should be moved to
>> FAB
>> > > minimalistic user manager (and removed from Airflow Core).
Eventually
>> maybe
>> > > even moved out of the Airflow repo altogether and moved to a FAB
user
>> > > manager add-on to airflow.
>> > >
>> > > This might seem a drastic change, but in fact - it's not drastic at
>> all. It
>> > > reflects what many of the Airflow users are doing now - they never
>> access
>> > > Security/Admin/Roles. Once they integrate with their LDAP or other
>> > > corporate directory, the Security UI, CLI and REST API are
essentially
>> > > useless (and not used).
>> > >
>> > > And it is precisely what we want to achieve - we want to delegate
the
>> > > user/role management completely out of Airflow. In fact - getting
rid
>> of
>> > > those is the only reason we want to implement the change. If we
>> **just**
>> > > replace the current FAB and keep all the complexity of managing the
>> users
>> > > /roles in Airflow, I'd say we have not achieved anything but adding
>> > > complexity.
>> > >
>> > > And - eventually - it makes the AIP-56 way *smaller*. We just need
to
>> make
>> > > sure that all the stuff for user management (including REST API, CLI
>> and
>> > > the UI) is moved to the "FAB minimalistic user manager" (and add
ways
>> to
>> > > plug-it in the core - possibly using existing plugins mechanisms
where
>> > > needed). We do not need to define new API for user/role management.
>> We only
>> > > need to describe the way how to check if the user has permission to
do
>> > > stuff.
>> > >
>> > > J.
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>
<mailto:jarek@potiuk.com <ma...@potiuk.com>>
>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>> wrote:
>> > >
>> > > > I promise to provide my feedback by the end of the week, sorry for
>> > > > delaying it, but we had some 2.6 preparation, branching, feature
>> flags
>> > > etc.
>> > > > also for AIP-44 and I am getting back on track with that one soon.
>> > > >
>> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
>> <shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>> <mailto:
shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>>>lid
>> > > >
>> > > > wrote:
>> > > >
>> > > >> Bumping this up for feedback!
>> > > >>
>> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite busy,
but
>> we
>> > > >> would greatly appreciate your feedback here. Your inputs could
>> help us
>> > > >> improve the proposal and move forward with multi-tenancy work.
>> > > >>
>> > > >> Cheers,
>> > > >> Shubham
>> > > >>
>> > > >>
>> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
<mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>
>> > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
<mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>>LID>
>> wrote:
>> > > >>
>> > > >>
>> > > >> CAUTION: This email originated from outside of the organization.
>> Do not
>> > > >> click links or open attachments unless you can confirm the sender
>> and
>> > > know
>> > > >> the content is safe.
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> Hi all,
>> > > >>
>> > > >>
>> > > >> I would like to get your feedback on an AIP I wrote: "Extensible
>> user
>> > > >> management". This AIP is a follow-up on a discussion we had in
this
>> > > email
>> > > >> list on the multi tenancy topic. I decided to create a new email
>> thread
>> > > >> because I feel the topic had diverged a bit from the original
topic
>> > > (multi
>> > > >> tenancy).
>> > > >>
>> > > >>
>> > > >> As a summary, the purpose of this AIP is to extract the user
>> management
>> > > >> from core Airflow by introducing a user manager interface in the
>> core
>> > > >> Airflow that can be extended by any provider package who want to
>> support
>> > > >> user management natively. As a result, if this AIP gets approved
>> and go
>> > > >> through, the multi tenancy feature will be implemented as a
second
>> step
>> > > in
>> > > >> a new user manager and not in Airflow directly.
>> > > >>
>> > > >>
>> > > >> As always, feel very free to give your opinion on this email
>> thread or
>> > > on
>> > > >> the AIP by adding comments.
>> > > >>
>> > > >>
>> > > >> References:
>> > > >> - AIP:
>> > > >>
>> > >
>>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
>> <
>>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
>> >
>> > > >> <
>> > > >>
>> > >
>>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
>> <
>>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>>
>> >
>> > > >> >
>> > > >> - Initial email list discussion:
>> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
>> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>> <
>> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
<https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>>
>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>> <
https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&amp;gt;&gt;>>
>> > > >>
>> > > >>
>> > > >> Vincent
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>> >
>> >
>> >
>> > --
>> >
>> > --
>> > Bolke de Bruin
>> > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:bdbruin@gmail.com <ma...@gmail.com>
<mailto:bdbruin@gmail.com <ma...@gmail.com>>>
>> >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:
dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>>
>> > For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org> <mailto:
dev-help@airflow.apache.org <ma...@airflow.apache.org>>
>> >
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>
Re: AIP-56 Extensible user management
Posted by Bolke de Bruin <bd...@gmail.com>.
I think think the name “User Manager” is a misnomer and it should just be
called “Security Manager”. It was confusing the hell out of me when reading
the AIP and I was convinced you wanted to fully manage e.g. keycloak
provided users inside Airflow (CRUD). What you are doing but aren’t exactly
saying:
1 - Define API that a Security Plugin should adhere to (AuthZ/AuthN)
2 - Have a default Security Plugin based on FAB
Points of improvement:
* is_authorized should include context so that it becomes possible the
determine whether a user has access to a resource based on for example the
location the user is at. Or in which environment. It is also useful for
auditing. Non implementing security managers can ignore it.
* I’d prefer some kind of ’standards’ based API. Like using Flask’s way of
registering endpoints or something along those lines. See also questions
below.
Questions:
* What is the need of post_login? Why would you want to store the access
token? The security manager should, imho, do this on its own: it executes
the login flow.
* Why is get_tab_configuration special? We have this possibility in
standard “plugins”. I suggest using that framework
* What is the need for “get_user_name”? Are we going to invent our own
Framework? Otherwise Flask might work?
Cheers
Bolke
On 9 May 2023 at 19:55:32, Beck, Vincent (vincbeck@amazon.com.invalid)
wrote:
Hello all,
I would like to start voting on this one so please add your comments on the
API or by replying here if you're interested, you still have time to do it
:)
AIP:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
Vincent
On 2023-05-03, 8:03 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
jarek@potiuk.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you can confirm the sender and know
the content is safe.
Alsoo to add (and make more concrete) to what it could mean at the
implementation level.
It means the for our API endpoints and views is that instead of
@auth.hasaccess (for example here
https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680 <
https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>)
where it check for permissions assigned to current role, airflow would
entirely delegate the decision to the 'user management plugin'.
We would simply delegate the decision to whatever plugin we have
configured. In case FAB pługin is used, it would do exactly the same checks
as today (and FAB pługin would contribute the 'Security' menu the same way
other plugins can contribute their own (as well as CLI commands and
user/role APIs - we would have to add a mechanism where plugins could
contribute CLIs and APIs on their own, but that should be easy). So if you
have FAB pługin chosen as default, nothing changes for the users comparing
to today. It would not require bumping Airflow Major version because it
would be backwards compatible. However similarly as providers it would move
the code of FAB plugins and roles out of the core of Airflow.
We could have keycloak 'reference' implementation where we would come up
with very opinionated approach where we have simple admin/user roles
(matching roughly default set of roles and permissions we currently have
with FAB). The big difference here will be that the user/role APIs, views,
CLIs will be gone entirely. And we could add there the concept of
'tenants' or rather 'teams' giving access to 'dag groups' and implement
decisions ob authorisation based on 'group' assignment for dag and user.
This is the main goal of this change to make it possible and easy without
having to mess with FAB permission and Role model which is completely not
suitable to manage 'groups of resources of the same type' with separate set
of permissions (think having few separate teams or tenants - each of them
wanting access to separate set of DAGs).
Currently if an SSO/enterprise user wants to plug in their authn/authz with
group access - they have to deal with all that including cumbersome syncing
of permissions for individual DAGs matching them to users separately - just
to satisfy FAB permission model lacking this feature and flexibility.
No more need for that if we go the direction described here.
And that's basically it for what we would implement in Airflow
out-of-the-box: FAB minimal implementation + be keycloak opinionated
reference implementation with simple teams/tenant support
This also makes it super flexible for those who want more flexibility and
do not want to use our opinionated keycloak plugin. They will be entirely
free to implement their own ways - as flexible or as opinionated they want
- without having to messwith permission syncing and the role model of FAB.
Nor keycloak integration. They could go with their own.
And with that - sky is the limit. The decision to access particular dag
could be made based on folder where the dag is from, id of dag, or dag
label or whatever other parameter (as long as we pass all that information
to the plugin). You could even - if you really want - make a decision to
allow certain users (and only them) to be able to clear individual tasks,
or tasks where specific operator types are used. For example you could
decide if you really want to only allow th google cloud admin users to
clear tasks that are operators coming from Google Provider (why not?). Or
only allow to that in certain time windows, or .... whatever.
It's extremely flexible and powerful - but we will not have to deal with
all the complexity in the community - we could delegate it to those who
mange the deployment or manage deployment of Airflow as a service and
decide they want to integrate it with the service authorisation that is
already there - think direct integratio with IAM ON AWS or GCP. And if we
just provide a reference, opinionated implementation for keycloak with
tenant support and legacy/minimal FAB as optional plugins, that should also
'good enough' for those who do not want to implement their own plugin
implementation.
J.
śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <jarek@potiuk.com <mailto:
jarek@potiuk.com>> napisał:
> > I don't think we can do that: we still need to consider users when they
> are just getting started out with Airflow -- if we only cater to giant
> enterprises with SSO and a central directory we remove our ability for
> individual or small teams/companies to get started with Airflow.
>
> Let me clarify this because I think we are talking about the same thing.
>
> I **think** having the "minimal FAB user manager implementation" as the
> "default" option when you install Airflow serves the purpose you mention
> Ash - if we do it the way that User/Role management is coming together
with
> FAB implementation (enabled by default for the initial installation).
What
> I am saying is that it should not be part of the "core airflow" in the
> sense that one could live without it (specifically in those big
SSO/Central
> directory case).
> In my vision those two cases are separated, but the "minimal
> implementation" when enabled should act and behave "as if" nothing
changed
> compared to the current way it looks and behaves.
>
> J.
>
>
> On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <ash@apache.org <mailto:
ash@apache.org>> wrote:
>
>> > - I am fine with (and actually I like it better) removing User and
Role
>> related Rest APIs and CLI commands from Airflow. That will indeed make
the
>> AIP way simpler and shorter
>>
>> I don't think we can do that: we still need to consider users when they
>> are just getting started out with Airflow -- if we only cater to giant
>> enterprises with SSO and a central directory we remove our ability for
>> individual or small teams/companies to get started with Airflow.
>> On May 2 2023, at 8:23 pm, Beck, Vincent <vincbeck@amazon.com.INVA
<ma...@amazon.com.INVA>LID>
>> wrote:
>> > Thank you Jarek and Bolke for taking time to read and provide feedback
>> to the AIP, much appreciated! Multiple points here:
>> >
>> > - I am fine with (and actually I like it better) removing User and
Role
>> related Rest APIs and CLI commands from Airflow. That will indeed make
the
>> AIP way simpler and shorter. As I explained in the AIP comments, I am
>> suggesting though to leave an option to user managers to define
additional
>> REST APIs and CLI commands if they need. By default no additional REST
API
>> and CLI command would be defined but user managers would have the option
to
>> override it. FAB user manager would be an example of user manager
>> overriding such methods. In any case, if additional Rest API or CLI
command
>> is defined, this would be done in user managers (as opposed to core
>> Airflow).
>> > - "the whole "Security" menu should be GONE completely".
>> > > I agree with this when the user manager is not the FAB one, however
>> we need to introduce a new one so Admins can easily go the external user
>> manager console (e.g. KeyCloak, ...). This new tab would only be a
>> link/redirect to the external console.
>> >
>> > - "All those components (and code related to those) should be moved to
>> FAB minimalistic user manager (and removed from Airflow Core)".
>> > > 100%. This is the main principle of this AIP, anything related to
>> user management will be moved to user managers.
>> >
>> > - "AuthN and AuthZ should be completely removed from Airflow".
>> > > 100%.
>> >
>> > - "So for a nice out-of-the-box experience (pip install
apache-airflow)
>> a lightweight User Manager could exist as a side project and have some
>> menus via the plugin system until something better comes along.".
>> > > I agree with you on the long term. Though, as a first iteration I
>> think it is better to define a user manager using FAB as default user
>> manager. The benefit of doing so is to have a backward compatible
>> transition between before AIP and after AIP. This also simplifies the
AIP
>> since it is easier to move files from core Airflow to user managers than
>> creating something new. Creating a new security model would be a
challenge
>> too. However, once the AIP implemented and merged, we can easily replace
>> this out-of-the-box FAB user manager by something "better" if the
communty
>> thinks this is the right way to go.
>> >
>> > As always, feel free to give me your thoughts.
>> > Vincent
>> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <mailto:
bdbruin@gmail.com> <mailto:
>> bdbruin@gmail.com <ma...@gmail.com>>> wrote:
>> > I'm not sure if I read the AIP correctly. I think I concur with Jarek
>> here.
>> >
>> > AuthN and AuthZ should be completely removed from Airflow. Access
>> > management should be outsourced to something like Open Policy Agent.
In
>> > other words there should be no user management at all within Airflow.
>> This
>> > removes the need to "sync" users (and thus become outdated) and
reduces
>> the
>> > attack surface drastically. So for a nice out-of-the-box experience
(pip
>> > install apache-airflow) a lightweight User Manager could exist as a
side
>> > project and have some menus via the plugin system until something
better
>> > comes along. Keycloak is great, a bit heavy for us to package
>> > unfortunately.
>> >
>> >
>> > FAB's security model is very non-standard and requires 'unnatural'
>> mappings
>> > in an enterprise context. It has served us well, but I would say good
>> > riddance.
>> >
>> >
>> > Bolke
>> >
>> >
>> >
>> >
>> >
>> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com
<ma...@potiuk.com>
>> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>:
>> >
>> > > I had a look and I liked what I saw for how we should integrate the
>> new
>> > > user manager for getting and checking the access :).
>> > >
>> > > But I think the AIP proposal is not bold enough when it comes to the
>> user
>> > > management part. This is one big comment - we should simplify it
even
>> > > further and cut more deeply.
>> > >
>> > > One of the big changes I think is super important with extensible
user
>> > > management is that we should delegate even more to the external
>> systems
>> > > managing users. The AIP attempts to map the current API/CLI/UI for
>> managing
>> > > the users and roles to the new external user management services -
>> but it
>> > > is IMHO absolutely not needed.
>> > >
>> > > As I see the extensible user management is that when you choose
>> something
>> > > else than the legacy FAB minimalist user management. Airflow should
>> become
>> > > just "user" of that user/role information managed elsewhere. It
>> should not
>> > > provide ANY option to see and manage the users or roles, it should
>> merely
>> > > read the information and give access to the users for appropriate
>> actions,
>> > > but then all the role/user management should happen outside of
>> Airflow - in
>> > > the system that is dedicated to manage those.
>> > >
>> > > IMHO - when other than the "non FAB minimalistic user manager"
system
>> is
>> > > used. Airflow should just not have a number of functionalities at
all:
>> > >
>> > > * not have "users" or "roles" CLI groups at all - they should be
>> missing
>> > > * the
>> > >
>> > >
>>
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
<
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
>> <
>>
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
<
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
>> >
>> > > and
>> > >
>> > >
>>
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
<
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
>> <
>>
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
<
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
>> >
>> > > should return 404 NOT FOUND
>> > > * the whole "Security" menu should be GONE completely.
>> > >
>> > > All those components (and code related to those) should be moved to
>> FAB
>> > > minimalistic user manager (and removed from Airflow Core).
Eventually
>> maybe
>> > > even moved out of the Airflow repo altogether and moved to a FAB
user
>> > > manager add-on to airflow.
>> > >
>> > > This might seem a drastic change, but in fact - it's not drastic at
>> all. It
>> > > reflects what many of the Airflow users are doing now - they never
>> access
>> > > Security/Admin/Roles. Once they integrate with their LDAP or other
>> > > corporate directory, the Security UI, CLI and REST API are
essentially
>> > > useless (and not used).
>> > >
>> > > And it is precisely what we want to achieve - we want to delegate
the
>> > > user/role management completely out of Airflow. In fact - getting
rid
>> of
>> > > those is the only reason we want to implement the change. If we
>> **just**
>> > > replace the current FAB and keep all the complexity of managing the
>> users
>> > > /roles in Airflow, I'd say we have not achieved anything but adding
>> > > complexity.
>> > >
>> > > And - eventually - it makes the AIP-56 way *smaller*. We just need
to
>> make
>> > > sure that all the stuff for user management (including REST API, CLI
>> and
>> > > the UI) is moved to the "FAB minimalistic user manager" (and add
ways
>> to
>> > > plug-it in the core - possibly using existing plugins mechanisms
where
>> > > needed). We do not need to define new API for user/role management.
>> We only
>> > > need to describe the way how to check if the user has permission to
do
>> > > stuff.
>> > >
>> > > J.
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com
<ma...@potiuk.com>
>> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> wrote:
>> > >
>> > > > I promise to provide my feedback by the end of the week, sorry for
>> > > > delaying it, but we had some 2.6 preparation, branching, feature
>> flags
>> > > etc.
>> > > > also for AIP-44 and I am getting back on track with that one soon.
>> > > >
>> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
>> <shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:
shubhx@amazon.com.inva <ma...@amazon.com.inva>>lid
>> > > >
>> > > > wrote:
>> > > >
>> > > >> Bumping this up for feedback!
>> > > >>
>> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite busy,
but
>> we
>> > > >> would greatly appreciate your feedback here. Your inputs could
>> help us
>> > > >> improve the proposal and move forward with multi-tenancy work.
>> > > >>
>> > > >> Cheers,
>> > > >> Shubham
>> > > >>
>> > > >>
>> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA
<ma...@amazon.com.INVA>
>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
>> > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
<mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>LID>
>> wrote:
>> > > >>
>> > > >>
>> > > >> CAUTION: This email originated from outside of the organization.
>> Do not
>> > > >> click links or open attachments unless you can confirm the sender
>> and
>> > > know
>> > > >> the content is safe.
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> Hi all,
>> > > >>
>> > > >>
>> > > >> I would like to get your feedback on an AIP I wrote: "Extensible
>> user
>> > > >> management". This AIP is a follow-up on a discussion we had in
this
>> > > email
>> > > >> list on the multi tenancy topic. I decided to create a new email
>> thread
>> > > >> because I feel the topic had diverged a bit from the original
topic
>> > > (multi
>> > > >> tenancy).
>> > > >>
>> > > >>
>> > > >> As a summary, the purpose of this AIP is to extract the user
>> management
>> > > >> from core Airflow by introducing a user manager interface in the
>> core
>> > > >> Airflow that can be extended by any provider package who want to
>> support
>> > > >> user management natively. As a result, if this AIP gets approved
>> and go
>> > > >> through, the multi tenancy feature will be implemented as a
second
>> step
>> > > in
>> > > >> a new user manager and not in Airflow directly.
>> > > >>
>> > > >>
>> > > >> As always, feel very free to give your opinion on this email
>> thread or
>> > > on
>> > > >> the AIP by adding comments.
>> > > >>
>> > > >>
>> > > >> References:
>> > > >> - AIP:
>> > > >>
>> > >
>>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
>> <
>>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
>> >
>> > > >> <
>> > > >>
>> > >
>>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
>> <
>>
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
<
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
>> >
>> > > >> >
>> > > >> - Initial email list discussion:
>> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <
https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
>> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
>> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
<https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>>
>> > > >>
>> > > >>
>> > > >> Vincent
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>> >
>> >
>> >
>> > --
>> >
>> > --
>> > Bolke de Bruin
>> > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com
<ma...@gmail.com>>
>> >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
dev-unsubscribe@airflow.apache.org>
>> > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
dev-help@airflow.apache.org>
>> >
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: AIP-56 Extensible user management
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
Hello all,
I would like to start voting on this one so please add your comments on the API or by replying here if you're interested, you still have time to do it :)
AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
Vincent
On 2023-05-03, 8:03 AM, "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Alsoo to add (and make more concrete) to what it could mean at the
implementation level.
It means the for our API endpoints and views is that instead of
@auth.hasaccess (for example here
https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680 <https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680>)
where it check for permissions assigned to current role, airflow would
entirely delegate the decision to the 'user management plugin'.
We would simply delegate the decision to whatever plugin we have
configured. In case FAB pługin is used, it would do exactly the same checks
as today (and FAB pługin would contribute the 'Security' menu the same way
other plugins can contribute their own (as well as CLI commands and
user/role APIs - we would have to add a mechanism where plugins could
contribute CLIs and APIs on their own, but that should be easy). So if you
have FAB pługin chosen as default, nothing changes for the users comparing
to today. It would not require bumping Airflow Major version because it
would be backwards compatible. However similarly as providers it would move
the code of FAB plugins and roles out of the core of Airflow.
We could have keycloak 'reference' implementation where we would come up
with very opinionated approach where we have simple admin/user roles
(matching roughly default set of roles and permissions we currently have
with FAB). The big difference here will be that the user/role APIs, views,
CLIs will be gone entirely. And we could add there the concept of
'tenants' or rather 'teams' giving access to 'dag groups' and implement
decisions ob authorisation based on 'group' assignment for dag and user.
This is the main goal of this change to make it possible and easy without
having to mess with FAB permission and Role model which is completely not
suitable to manage 'groups of resources of the same type' with separate set
of permissions (think having few separate teams or tenants - each of them
wanting access to separate set of DAGs).
Currently if an SSO/enterprise user wants to plug in their authn/authz with
group access - they have to deal with all that including cumbersome syncing
of permissions for individual DAGs matching them to users separately - just
to satisfy FAB permission model lacking this feature and flexibility.
No more need for that if we go the direction described here.
And that's basically it for what we would implement in Airflow
out-of-the-box: FAB minimal implementation + be keycloak opinionated
reference implementation with simple teams/tenant support
This also makes it super flexible for those who want more flexibility and
do not want to use our opinionated keycloak plugin. They will be entirely
free to implement their own ways - as flexible or as opinionated they want
- without having to messwith permission syncing and the role model of FAB.
Nor keycloak integration. They could go with their own.
And with that - sky is the limit. The decision to access particular dag
could be made based on folder where the dag is from, id of dag, or dag
label or whatever other parameter (as long as we pass all that information
to the plugin). You could even - if you really want - make a decision to
allow certain users (and only them) to be able to clear individual tasks,
or tasks where specific operator types are used. For example you could
decide if you really want to only allow th google cloud admin users to
clear tasks that are operators coming from Google Provider (why not?). Or
only allow to that in certain time windows, or .... whatever.
It's extremely flexible and powerful - but we will not have to deal with
all the complexity in the community - we could delegate it to those who
mange the deployment or manage deployment of Airflow as a service and
decide they want to integrate it with the service authorisation that is
already there - think direct integratio with IAM ON AWS or GCP. And if we
just provide a reference, opinionated implementation for keycloak with
tenant support and legacy/minimal FAB as optional plugins, that should also
'good enough' for those who do not want to implement their own plugin
implementation.
J.
śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> napisał:
> > I don't think we can do that: we still need to consider users when they
> are just getting started out with Airflow -- if we only cater to giant
> enterprises with SSO and a central directory we remove our ability for
> individual or small teams/companies to get started with Airflow.
>
> Let me clarify this because I think we are talking about the same thing.
>
> I **think** having the "minimal FAB user manager implementation" as the
> "default" option when you install Airflow serves the purpose you mention
> Ash - if we do it the way that User/Role management is coming together with
> FAB implementation (enabled by default for the initial installation). What
> I am saying is that it should not be part of the "core airflow" in the
> sense that one could live without it (specifically in those big SSO/Central
> directory case).
> In my vision those two cases are separated, but the "minimal
> implementation" when enabled should act and behave "as if" nothing changed
> compared to the current way it looks and behaves.
>
> J.
>
>
> On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> wrote:
>
>> > - I am fine with (and actually I like it better) removing User and Role
>> related Rest APIs and CLI commands from Airflow. That will indeed make the
>> AIP way simpler and shorter
>>
>> I don't think we can do that: we still need to consider users when they
>> are just getting started out with Airflow -- if we only cater to giant
>> enterprises with SSO and a central directory we remove our ability for
>> individual or small teams/companies to get started with Airflow.
>> On May 2 2023, at 8:23 pm, Beck, Vincent <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>LID>
>> wrote:
>> > Thank you Jarek and Bolke for taking time to read and provide feedback
>> to the AIP, much appreciated! Multiple points here:
>> >
>> > - I am fine with (and actually I like it better) removing User and Role
>> related Rest APIs and CLI commands from Airflow. That will indeed make the
>> AIP way simpler and shorter. As I explained in the AIP comments, I am
>> suggesting though to leave an option to user managers to define additional
>> REST APIs and CLI commands if they need. By default no additional REST API
>> and CLI command would be defined but user managers would have the option to
>> override it. FAB user manager would be an example of user manager
>> overriding such methods. In any case, if additional Rest API or CLI command
>> is defined, this would be done in user managers (as opposed to core
>> Airflow).
>> > - "the whole "Security" menu should be GONE completely".
>> > > I agree with this when the user manager is not the FAB one, however
>> we need to introduce a new one so Admins can easily go the external user
>> manager console (e.g. KeyCloak, ...). This new tab would only be a
>> link/redirect to the external console.
>> >
>> > - "All those components (and code related to those) should be moved to
>> FAB minimalistic user manager (and removed from Airflow Core)".
>> > > 100%. This is the main principle of this AIP, anything related to
>> user management will be moved to user managers.
>> >
>> > - "AuthN and AuthZ should be completely removed from Airflow".
>> > > 100%.
>> >
>> > - "So for a nice out-of-the-box experience (pip install apache-airflow)
>> a lightweight User Manager could exist as a side project and have some
>> menus via the plugin system until something better comes along.".
>> > > I agree with you on the long term. Though, as a first iteration I
>> think it is better to define a user manager using FAB as default user
>> manager. The benefit of doing so is to have a backward compatible
>> transition between before AIP and after AIP. This also simplifies the AIP
>> since it is easier to move files from core Airflow to user managers than
>> creating something new. Creating a new security model would be a challenge
>> too. However, once the AIP implemented and merged, we can easily replace
>> this out-of-the-box FAB user manager by something "better" if the communty
>> thinks this is the right way to go.
>> >
>> > As always, feel free to give me your thoughts.
>> > Vincent
>> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com> <mailto:
>> bdbruin@gmail.com <ma...@gmail.com>>> wrote:
>> > I'm not sure if I read the AIP correctly. I think I concur with Jarek
>> here.
>> >
>> > AuthN and AuthZ should be completely removed from Airflow. Access
>> > management should be outsourced to something like Open Policy Agent. In
>> > other words there should be no user management at all within Airflow.
>> This
>> > removes the need to "sync" users (and thus become outdated) and reduces
>> the
>> > attack surface drastically. So for a nice out-of-the-box experience (pip
>> > install apache-airflow) a lightweight User Manager could exist as a side
>> > project and have some menus via the plugin system until something better
>> > comes along. Keycloak is great, a bit heavy for us to package
>> > unfortunately.
>> >
>> >
>> > FAB's security model is very non-standard and requires 'unnatural'
>> mappings
>> > in an enterprise context. It has served us well, but I would say good
>> > riddance.
>> >
>> >
>> > Bolke
>> >
>> >
>> >
>> >
>> >
>> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>
>> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>:
>> >
>> > > I had a look and I liked what I saw for how we should integrate the
>> new
>> > > user manager for getting and checking the access :).
>> > >
>> > > But I think the AIP proposal is not bold enough when it comes to the
>> user
>> > > management part. This is one big comment - we should simplify it even
>> > > further and cut more deeply.
>> > >
>> > > One of the big changes I think is super important with extensible user
>> > > management is that we should delegate even more to the external
>> systems
>> > > managing users. The AIP attempts to map the current API/CLI/UI for
>> managing
>> > > the users and roles to the new external user management services -
>> but it
>> > > is IMHO absolutely not needed.
>> > >
>> > > As I see the extensible user management is that when you choose
>> something
>> > > else than the legacy FAB minimalist user management. Airflow should
>> become
>> > > just "user" of that user/role information managed elsewhere. It
>> should not
>> > > provide ANY option to see and manage the users or roles, it should
>> merely
>> > > read the information and give access to the users for appropriate
>> actions,
>> > > but then all the role/user management should happen outside of
>> Airflow - in
>> > > the system that is dedicated to manage those.
>> > >
>> > > IMHO - when other than the "non FAB minimalistic user manager" system
>> is
>> > > used. Airflow should just not have a number of functionalities at all:
>> > >
>> > > * not have "users" or "roles" CLI groups at all - they should be
>> missing
>> > > * the
>> > >
>> > >
>> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
>> <
>> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
>> >
>> > > and
>> > >
>> > >
>> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
>> <
>> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
>> >
>> > > should return 404 NOT FOUND
>> > > * the whole "Security" menu should be GONE completely.
>> > >
>> > > All those components (and code related to those) should be moved to
>> FAB
>> > > minimalistic user manager (and removed from Airflow Core). Eventually
>> maybe
>> > > even moved out of the Airflow repo altogether and moved to a FAB user
>> > > manager add-on to airflow.
>> > >
>> > > This might seem a drastic change, but in fact - it's not drastic at
>> all. It
>> > > reflects what many of the Airflow users are doing now - they never
>> access
>> > > Security/Admin/Roles. Once they integrate with their LDAP or other
>> > > corporate directory, the Security UI, CLI and REST API are essentially
>> > > useless (and not used).
>> > >
>> > > And it is precisely what we want to achieve - we want to delegate the
>> > > user/role management completely out of Airflow. In fact - getting rid
>> of
>> > > those is the only reason we want to implement the change. If we
>> **just**
>> > > replace the current FAB and keep all the complexity of managing the
>> users
>> > > /roles in Airflow, I'd say we have not achieved anything but adding
>> > > complexity.
>> > >
>> > > And - eventually - it makes the AIP-56 way *smaller*. We just need to
>> make
>> > > sure that all the stuff for user management (including REST API, CLI
>> and
>> > > the UI) is moved to the "FAB minimalistic user manager" (and add ways
>> to
>> > > plug-it in the core - possibly using existing plugins mechanisms where
>> > > needed). We do not need to define new API for user/role management.
>> We only
>> > > need to describe the way how to check if the user has permission to do
>> > > stuff.
>> > >
>> > > J.
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>
>> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> wrote:
>> > >
>> > > > I promise to provide my feedback by the end of the week, sorry for
>> > > > delaying it, but we had some 2.6 preparation, branching, feature
>> flags
>> > > etc.
>> > > > also for AIP-44 and I am getting back on track with that one soon.
>> > > >
>> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
>> <shubhx@amazon.com.inva <ma...@amazon.com.inva> <mailto:shubhx@amazon.com.inva <ma...@amazon.com.inva>>lid
>> > > >
>> > > > wrote:
>> > > >
>> > > >> Bumping this up for feedback!
>> > > >>
>> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite busy, but
>> we
>> > > >> would greatly appreciate your feedback here. Your inputs could
>> help us
>> > > >> improve the proposal and move forward with multi-tenancy work.
>> > > >>
>> > > >> Cheers,
>> > > >> Shubham
>> > > >>
>> > > >>
>> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
>> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>
>> > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>>LID>
>> wrote:
>> > > >>
>> > > >>
>> > > >> CAUTION: This email originated from outside of the organization.
>> Do not
>> > > >> click links or open attachments unless you can confirm the sender
>> and
>> > > know
>> > > >> the content is safe.
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> Hi all,
>> > > >>
>> > > >>
>> > > >> I would like to get your feedback on an AIP I wrote: "Extensible
>> user
>> > > >> management". This AIP is a follow-up on a discussion we had in this
>> > > email
>> > > >> list on the multi tenancy topic. I decided to create a new email
>> thread
>> > > >> because I feel the topic had diverged a bit from the original topic
>> > > (multi
>> > > >> tenancy).
>> > > >>
>> > > >>
>> > > >> As a summary, the purpose of this AIP is to extract the user
>> management
>> > > >> from core Airflow by introducing a user manager interface in the
>> core
>> > > >> Airflow that can be extended by any provider package who want to
>> support
>> > > >> user management natively. As a result, if this AIP gets approved
>> and go
>> > > >> through, the multi tenancy feature will be implemented as a second
>> step
>> > > in
>> > > >> a new user manager and not in Airflow directly.
>> > > >>
>> > > >>
>> > > >> As always, feel very free to give your opinion on this email
>> thread or
>> > > on
>> > > >> the AIP by adding comments.
>> > > >>
>> > > >>
>> > > >> References:
>> > > >> - AIP:
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
>> >
>> > > >> <
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
>> >
>> > > >> >
>> > > >> - Initial email list discussion:
>> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
>> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <
>> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61&gt;>>
>> > > >>
>> > > >>
>> > > >> Vincent
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>> >
>> >
>> >
>> > --
>> >
>> > --
>> > Bolke de Bruin
>> > bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>
>> >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
>> > For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org>
>> >
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: AIP-56 Extensible user management
Posted by Jarek Potiuk <ja...@potiuk.com>.
Alsoo to add (and make more concrete) to what it could mean at the
implementation level.
It means the for our API endpoints and views is that instead of
@auth.hasaccess (for example here
https://github.com/apache/airflow/blob/main/airflow/www/views.py#L680)
where it check for permissions assigned to current role, airflow would
entirely delegate the decision to the 'user management plugin'.
We would simply delegate the decision to whatever plugin we have
configured. In case FAB pługin is used, it would do exactly the same checks
as today (and FAB pługin would contribute the 'Security' menu the same way
other plugins can contribute their own (as well as CLI commands and
user/role APIs - we would have to add a mechanism where plugins could
contribute CLIs and APIs on their own, but that should be easy). So if you
have FAB pługin chosen as default, nothing changes for the users comparing
to today. It would not require bumping Airflow Major version because it
would be backwards compatible. However similarly as providers it would move
the code of FAB plugins and roles out of the core of Airflow.
We could have keycloak 'reference' implementation where we would come up
with very opinionated approach where we have simple admin/user roles
(matching roughly default set of roles and permissions we currently have
with FAB). The big difference here will be that the user/role APIs, views,
CLIs will be gone entirely. And we could add there the concept of
'tenants' or rather 'teams' giving access to 'dag groups' and implement
decisions ob authorisation based on 'group' assignment for dag and user.
This is the main goal of this change to make it possible and easy without
having to mess with FAB permission and Role model which is completely not
suitable to manage 'groups of resources of the same type' with separate set
of permissions (think having few separate teams or tenants - each of them
wanting access to separate set of DAGs).
Currently if an SSO/enterprise user wants to plug in their authn/authz with
group access - they have to deal with all that including cumbersome syncing
of permissions for individual DAGs matching them to users separately - just
to satisfy FAB permission model lacking this feature and flexibility.
No more need for that if we go the direction described here.
And that's basically it for what we would implement in Airflow
out-of-the-box: FAB minimal implementation + be keycloak opinionated
reference implementation with simple teams/tenant support
This also makes it super flexible for those who want more flexibility and
do not want to use our opinionated keycloak plugin. They will be entirely
free to implement their own ways - as flexible or as opinionated they want
- without having to messwith permission syncing and the role model of FAB.
Nor keycloak integration. They could go with their own.
And with that - sky is the limit. The decision to access particular dag
could be made based on folder where the dag is from, id of dag, or dag
label or whatever other parameter (as long as we pass all that information
to the plugin). You could even - if you really want - make a decision to
allow certain users (and only them) to be able to clear individual tasks,
or tasks where specific operator types are used. For example you could
decide if you really want to only allow th google cloud admin users to
clear tasks that are operators coming from Google Provider (why not?). Or
only allow to that in certain time windows, or .... whatever.
It's extremely flexible and powerful - but we will not have to deal with
all the complexity in the community - we could delegate it to those who
mange the deployment or manage deployment of Airflow as a service and
decide they want to integrate it with the service authorisation that is
already there - think direct integratio with IAM ON AWS or GCP. And if we
just provide a reference, opinionated implementation for keycloak with
tenant support and legacy/minimal FAB as optional plugins, that should also
'good enough' for those who do not want to implement their own plugin
implementation.
J.
śr., 3 maj 2023, 12:34 użytkownik Jarek Potiuk <ja...@potiuk.com> napisał:
> > I don't think we can do that: we still need to consider users when they
> are just getting started out with Airflow -- if we only cater to giant
> enterprises with SSO and a central directory we remove our ability for
> individual or small teams/companies to get started with Airflow.
>
> Let me clarify this because I think we are talking about the same thing.
>
> I **think** having the "minimal FAB user manager implementation" as the
> "default" option when you install Airflow serves the purpose you mention
> Ash - if we do it the way that User/Role management is coming together with
> FAB implementation (enabled by default for the initial installation). What
> I am saying is that it should not be part of the "core airflow" in the
> sense that one could live without it (specifically in those big SSO/Central
> directory case).
> In my vision those two cases are separated, but the "minimal
> implementation" when enabled should act and behave "as if" nothing changed
> compared to the current way it looks and behaves.
>
> J.
>
>
> On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> > - I am fine with (and actually I like it better) removing User and Role
>> related Rest APIs and CLI commands from Airflow. That will indeed make the
>> AIP way simpler and shorter
>>
>> I don't think we can do that: we still need to consider users when they
>> are just getting started out with Airflow -- if we only cater to giant
>> enterprises with SSO and a central directory we remove our ability for
>> individual or small teams/companies to get started with Airflow.
>> On May 2 2023, at 8:23 pm, Beck, Vincent <vi...@amazon.com.INVALID>
>> wrote:
>> > Thank you Jarek and Bolke for taking time to read and provide feedback
>> to the AIP, much appreciated! Multiple points here:
>> >
>> > - I am fine with (and actually I like it better) removing User and Role
>> related Rest APIs and CLI commands from Airflow. That will indeed make the
>> AIP way simpler and shorter. As I explained in the AIP comments, I am
>> suggesting though to leave an option to user managers to define additional
>> REST APIs and CLI commands if they need. By default no additional REST API
>> and CLI command would be defined but user managers would have the option to
>> override it. FAB user manager would be an example of user manager
>> overriding such methods. In any case, if additional Rest API or CLI command
>> is defined, this would be done in user managers (as opposed to core
>> Airflow).
>> > - "the whole "Security" menu should be GONE completely".
>> > > I agree with this when the user manager is not the FAB one, however
>> we need to introduce a new one so Admins can easily go the external user
>> manager console (e.g. KeyCloak, ...). This new tab would only be a
>> link/redirect to the external console.
>> >
>> > - "All those components (and code related to those) should be moved to
>> FAB minimalistic user manager (and removed from Airflow Core)".
>> > > 100%. This is the main principle of this AIP, anything related to
>> user management will be moved to user managers.
>> >
>> > - "AuthN and AuthZ should be completely removed from Airflow".
>> > > 100%.
>> >
>> > - "So for a nice out-of-the-box experience (pip install apache-airflow)
>> a lightweight User Manager could exist as a side project and have some
>> menus via the plugin system until something better comes along.".
>> > > I agree with you on the long term. Though, as a first iteration I
>> think it is better to define a user manager using FAB as default user
>> manager. The benefit of doing so is to have a backward compatible
>> transition between before AIP and after AIP. This also simplifies the AIP
>> since it is easier to move files from core Airflow to user managers than
>> creating something new. Creating a new security model would be a challenge
>> too. However, once the AIP implemented and merged, we can easily replace
>> this out-of-the-box FAB user manager by something "better" if the communty
>> thinks this is the right way to go.
>> >
>> > As always, feel free to give me your thoughts.
>> > Vincent
>> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <mailto:
>> bdbruin@gmail.com>> wrote:
>> > I'm not sure if I read the AIP correctly. I think I concur with Jarek
>> here.
>> >
>> > AuthN and AuthZ should be completely removed from Airflow. Access
>> > management should be outsourced to something like Open Policy Agent. In
>> > other words there should be no user management at all within Airflow.
>> This
>> > removes the need to "sync" users (and thus become outdated) and reduces
>> the
>> > attack surface drastically. So for a nice out-of-the-box experience (pip
>> > install apache-airflow) a lightweight User Manager could exist as a side
>> > project and have some menus via the plugin system until something better
>> > comes along. Keycloak is great, a bit heavy for us to package
>> > unfortunately.
>> >
>> >
>> > FAB's security model is very non-standard and requires 'unnatural'
>> mappings
>> > in an enterprise context. It has served us well, but I would say good
>> > riddance.
>> >
>> >
>> > Bolke
>> >
>> >
>> >
>> >
>> >
>> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com
>> <ma...@potiuk.com>>:
>> >
>> > > I had a look and I liked what I saw for how we should integrate the
>> new
>> > > user manager for getting and checking the access :).
>> > >
>> > > But I think the AIP proposal is not bold enough when it comes to the
>> user
>> > > management part. This is one big comment - we should simplify it even
>> > > further and cut more deeply.
>> > >
>> > > One of the big changes I think is super important with extensible user
>> > > management is that we should delegate even more to the external
>> systems
>> > > managing users. The AIP attempts to map the current API/CLI/UI for
>> managing
>> > > the users and roles to the new external user management services -
>> but it
>> > > is IMHO absolutely not needed.
>> > >
>> > > As I see the extensible user management is that when you choose
>> something
>> > > else than the legacy FAB minimalist user management. Airflow should
>> become
>> > > just "user" of that user/role information managed elsewhere. It
>> should not
>> > > provide ANY option to see and manage the users or roles, it should
>> merely
>> > > read the information and give access to the users for appropriate
>> actions,
>> > > but then all the role/user management should happen outside of
>> Airflow - in
>> > > the system that is dedicated to manage those.
>> > >
>> > > IMHO - when other than the "non FAB minimalistic user manager" system
>> is
>> > > used. Airflow should just not have a number of functionalities at all:
>> > >
>> > > * not have "users" or "roles" CLI groups at all - they should be
>> missing
>> > > * the
>> > >
>> > >
>> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
>> <
>> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
>> >
>> > > and
>> > >
>> > >
>> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
>> <
>> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
>> >
>> > > should return 404 NOT FOUND
>> > > * the whole "Security" menu should be GONE completely.
>> > >
>> > > All those components (and code related to those) should be moved to
>> FAB
>> > > minimalistic user manager (and removed from Airflow Core). Eventually
>> maybe
>> > > even moved out of the Airflow repo altogether and moved to a FAB user
>> > > manager add-on to airflow.
>> > >
>> > > This might seem a drastic change, but in fact - it's not drastic at
>> all. It
>> > > reflects what many of the Airflow users are doing now - they never
>> access
>> > > Security/Admin/Roles. Once they integrate with their LDAP or other
>> > > corporate directory, the Security UI, CLI and REST API are essentially
>> > > useless (and not used).
>> > >
>> > > And it is precisely what we want to achieve - we want to delegate the
>> > > user/role management completely out of Airflow. In fact - getting rid
>> of
>> > > those is the only reason we want to implement the change. If we
>> **just**
>> > > replace the current FAB and keep all the complexity of managing the
>> users
>> > > /roles in Airflow, I'd say we have not achieved anything but adding
>> > > complexity.
>> > >
>> > > And - eventually - it makes the AIP-56 way *smaller*. We just need to
>> make
>> > > sure that all the stuff for user management (including REST API, CLI
>> and
>> > > the UI) is moved to the "FAB minimalistic user manager" (and add ways
>> to
>> > > plug-it in the core - possibly using existing plugins mechanisms where
>> > > needed). We do not need to define new API for user/role management.
>> We only
>> > > need to describe the way how to check if the user has permission to do
>> > > stuff.
>> > >
>> > > J.
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com
>> <ma...@potiuk.com>> wrote:
>> > >
>> > > > I promise to provide my feedback by the end of the week, sorry for
>> > > > delaying it, but we had some 2.6 preparation, branching, feature
>> flags
>> > > etc.
>> > > > also for AIP-44 and I am getting back on track with that one soon.
>> > > >
>> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
>> <shubhx@amazon.com.inva <ma...@amazon.com.inva>lid
>> > > >
>> > > > wrote:
>> > > >
>> > > >> Bumping this up for feedback!
>> > > >>
>> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite busy, but
>> we
>> > > >> would greatly appreciate your feedback here. Your inputs could
>> help us
>> > > >> improve the proposal and move forward with multi-tenancy work.
>> > > >>
>> > > >> Cheers,
>> > > >> Shubham
>> > > >>
>> > > >>
>> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA
>> <ma...@amazon.com.INVA>
>> > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>LID>
>> wrote:
>> > > >>
>> > > >>
>> > > >> CAUTION: This email originated from outside of the organization.
>> Do not
>> > > >> click links or open attachments unless you can confirm the sender
>> and
>> > > know
>> > > >> the content is safe.
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> Hi all,
>> > > >>
>> > > >>
>> > > >> I would like to get your feedback on an AIP I wrote: "Extensible
>> user
>> > > >> management". This AIP is a follow-up on a discussion we had in this
>> > > email
>> > > >> list on the multi tenancy topic. I decided to create a new email
>> thread
>> > > >> because I feel the topic had diverged a bit from the original topic
>> > > (multi
>> > > >> tenancy).
>> > > >>
>> > > >>
>> > > >> As a summary, the purpose of this AIP is to extract the user
>> management
>> > > >> from core Airflow by introducing a user manager interface in the
>> core
>> > > >> Airflow that can be extended by any provider package who want to
>> support
>> > > >> user management natively. As a result, if this AIP gets approved
>> and go
>> > > >> through, the multi tenancy feature will be implemented as a second
>> step
>> > > in
>> > > >> a new user manager and not in Airflow directly.
>> > > >>
>> > > >>
>> > > >> As always, feel very free to give your opinion on this email
>> thread or
>> > > on
>> > > >> the AIP by adding comments.
>> > > >>
>> > > >>
>> > > >> References:
>> > > >> - AIP:
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
>> >
>> > > >> <
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
>> >
>> > > >> >
>> > > >> - Initial email list discussion:
>> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <
>> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
>> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
>> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
>> > > >>
>> > > >>
>> > > >> Vincent
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > >
>> >
>> >
>> >
>> >
>> > --
>> >
>> > --
>> > Bolke de Bruin
>> > bdbruin@gmail.com <ma...@gmail.com>
>> >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > For additional commands, e-mail: dev-help@airflow.apache.org
>> >
>>
>>
Re: AIP-56 Extensible user management
Posted by Jarek Potiuk <ja...@potiuk.com>.
> I don't think we can do that: we still need to consider users when they
are just getting started out with Airflow -- if we only cater to giant
enterprises with SSO and a central directory we remove our ability for
individual or small teams/companies to get started with Airflow.
Let me clarify this because I think we are talking about the same thing.
I **think** having the "minimal FAB user manager implementation" as the
"default" option when you install Airflow serves the purpose you mention
Ash - if we do it the way that User/Role management is coming together with
FAB implementation (enabled by default for the initial installation). What
I am saying is that it should not be part of the "core airflow" in the
sense that one could live without it (specifically in those big SSO/Central
directory case).
In my vision those two cases are separated, but the "minimal
implementation" when enabled should act and behave "as if" nothing changed
compared to the current way it looks and behaves.
J.
On Wed, May 3, 2023 at 12:06 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> > - I am fine with (and actually I like it better) removing User and Role
> related Rest APIs and CLI commands from Airflow. That will indeed make the
> AIP way simpler and shorter
>
> I don't think we can do that: we still need to consider users when they
> are just getting started out with Airflow -- if we only cater to giant
> enterprises with SSO and a central directory we remove our ability for
> individual or small teams/companies to get started with Airflow.
> On May 2 2023, at 8:23 pm, Beck, Vincent <vi...@amazon.com.INVALID>
> wrote:
> > Thank you Jarek and Bolke for taking time to read and provide feedback
> to the AIP, much appreciated! Multiple points here:
> >
> > - I am fine with (and actually I like it better) removing User and Role
> related Rest APIs and CLI commands from Airflow. That will indeed make the
> AIP way simpler and shorter. As I explained in the AIP comments, I am
> suggesting though to leave an option to user managers to define additional
> REST APIs and CLI commands if they need. By default no additional REST API
> and CLI command would be defined but user managers would have the option to
> override it. FAB user manager would be an example of user manager
> overriding such methods. In any case, if additional Rest API or CLI command
> is defined, this would be done in user managers (as opposed to core
> Airflow).
> > - "the whole "Security" menu should be GONE completely".
> > > I agree with this when the user manager is not the FAB one, however we
> need to introduce a new one so Admins can easily go the external user
> manager console (e.g. KeyCloak, ...). This new tab would only be a
> link/redirect to the external console.
> >
> > - "All those components (and code related to those) should be moved to
> FAB minimalistic user manager (and removed from Airflow Core)".
> > > 100%. This is the main principle of this AIP, anything related to user
> management will be moved to user managers.
> >
> > - "AuthN and AuthZ should be completely removed from Airflow".
> > > 100%.
> >
> > - "So for a nice out-of-the-box experience (pip install apache-airflow)
> a lightweight User Manager could exist as a side project and have some
> menus via the plugin system until something better comes along.".
> > > I agree with you on the long term. Though, as a first iteration I
> think it is better to define a user manager using FAB as default user
> manager. The benefit of doing so is to have a backward compatible
> transition between before AIP and after AIP. This also simplifies the AIP
> since it is easier to move files from core Airflow to user managers than
> creating something new. Creating a new security model would be a challenge
> too. However, once the AIP implemented and merged, we can easily replace
> this out-of-the-box FAB user manager by something "better" if the communty
> thinks this is the right way to go.
> >
> > As always, feel free to give me your thoughts.
> > Vincent
> > On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <mailto:
> bdbruin@gmail.com>> wrote:
> > I'm not sure if I read the AIP correctly. I think I concur with Jarek
> here.
> >
> > AuthN and AuthZ should be completely removed from Airflow. Access
> > management should be outsourced to something like Open Policy Agent. In
> > other words there should be no user management at all within Airflow.
> This
> > removes the need to "sync" users (and thus become outdated) and reduces
> the
> > attack surface drastically. So for a nice out-of-the-box experience (pip
> > install apache-airflow) a lightweight User Manager could exist as a side
> > project and have some menus via the plugin system until something better
> > comes along. Keycloak is great, a bit heavy for us to package
> > unfortunately.
> >
> >
> > FAB's security model is very non-standard and requires 'unnatural'
> mappings
> > in an enterprise context. It has served us well, but I would say good
> > riddance.
> >
> >
> > Bolke
> >
> >
> >
> >
> >
> > Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>>:
> >
> > > I had a look and I liked what I saw for how we should integrate the new
> > > user manager for getting and checking the access :).
> > >
> > > But I think the AIP proposal is not bold enough when it comes to the
> user
> > > management part. This is one big comment - we should simplify it even
> > > further and cut more deeply.
> > >
> > > One of the big changes I think is super important with extensible user
> > > management is that we should delegate even more to the external systems
> > > managing users. The AIP attempts to map the current API/CLI/UI for
> managing
> > > the users and roles to the new external user management services - but
> it
> > > is IMHO absolutely not needed.
> > >
> > > As I see the extensible user management is that when you choose
> something
> > > else than the legacy FAB minimalist user management. Airflow should
> become
> > > just "user" of that user/role information managed elsewhere. It should
> not
> > > provide ANY option to see and manage the users or roles, it should
> merely
> > > read the information and give access to the users for appropriate
> actions,
> > > but then all the role/user management should happen outside of Airflow
> - in
> > > the system that is dedicated to manage those.
> > >
> > > IMHO - when other than the "non FAB minimalistic user manager" system
> is
> > > used. Airflow should just not have a number of functionalities at all:
> > >
> > > * not have "users" or "roles" CLI groups at all - they should be
> missing
> > > * the
> > >
> > >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> >
> > > and
> > >
> > >
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> <
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> >
> > > should return 404 NOT FOUND
> > > * the whole "Security" menu should be GONE completely.
> > >
> > > All those components (and code related to those) should be moved to FAB
> > > minimalistic user manager (and removed from Airflow Core). Eventually
> maybe
> > > even moved out of the Airflow repo altogether and moved to a FAB user
> > > manager add-on to airflow.
> > >
> > > This might seem a drastic change, but in fact - it's not drastic at
> all. It
> > > reflects what many of the Airflow users are doing now - they never
> access
> > > Security/Admin/Roles. Once they integrate with their LDAP or other
> > > corporate directory, the Security UI, CLI and REST API are essentially
> > > useless (and not used).
> > >
> > > And it is precisely what we want to achieve - we want to delegate the
> > > user/role management completely out of Airflow. In fact - getting rid
> of
> > > those is the only reason we want to implement the change. If we
> **just**
> > > replace the current FAB and keep all the complexity of managing the
> users
> > > /roles in Airflow, I'd say we have not achieved anything but adding
> > > complexity.
> > >
> > > And - eventually - it makes the AIP-56 way *smaller*. We just need to
> make
> > > sure that all the stuff for user management (including REST API, CLI
> and
> > > the UI) is moved to the "FAB minimalistic user manager" (and add ways
> to
> > > plug-it in the core - possibly using existing plugins mechanisms where
> > > needed). We do not need to define new API for user/role management. We
> only
> > > need to describe the way how to check if the user has permission to do
> > > stuff.
> > >
> > > J.
> > >
> > >
> > >
> > >
> > > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>> wrote:
> > >
> > > > I promise to provide my feedback by the end of the week, sorry for
> > > > delaying it, but we had some 2.6 preparation, branching, feature
> flags
> > > etc.
> > > > also for AIP-44 and I am getting back on track with that one soon.
> > > >
> > > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham
> <shubhx@amazon.com.inva <ma...@amazon.com.inva>lid
> > > >
> > > > wrote:
> > > >
> > > >> Bumping this up for feedback!
> > > >>
> > > >> @Vikram, @Kaxil, et al. - I understand that you're quite busy, but
> we
> > > >> would greatly appreciate your feedback here. Your inputs could help
> us
> > > >> improve the proposal and move forward with multi-tenancy work.
> > > >>
> > > >> Cheers,
> > > >> Shubham
> > > >>
> > > >>
> > > >> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA
> <ma...@amazon.com.INVA>
> > > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>LID>
> wrote:
> > > >>
> > > >>
> > > >> CAUTION: This email originated from outside of the organization. Do
> not
> > > >> click links or open attachments unless you can confirm the sender
> and
> > > know
> > > >> the content is safe.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> Hi all,
> > > >>
> > > >>
> > > >> I would like to get your feedback on an AIP I wrote: "Extensible
> user
> > > >> management". This AIP is a follow-up on a discussion we had in this
> > > email
> > > >> list on the multi tenancy topic. I decided to create a new email
> thread
> > > >> because I feel the topic had diverged a bit from the original topic
> > > (multi
> > > >> tenancy).
> > > >>
> > > >>
> > > >> As a summary, the purpose of this AIP is to extract the user
> management
> > > >> from core Airflow by introducing a user manager interface in the
> core
> > > >> Airflow that can be extended by any provider package who want to
> support
> > > >> user management natively. As a result, if this AIP gets approved
> and go
> > > >> through, the multi tenancy feature will be implemented as a second
> step
> > > in
> > > >> a new user manager and not in Airflow directly.
> > > >>
> > > >>
> > > >> As always, feel very free to give your opinion on this email thread
> or
> > > on
> > > >> the AIP by adding comments.
> > > >>
> > > >>
> > > >> References:
> > > >> - AIP:
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> > > >> <
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> > > >> >
> > > >> - Initial email list discussion:
> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> > > >>
> > > >>
> > > >> Vincent
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> >
> >
> >
> >
> > --
> >
> > --
> > Bolke de Bruin
> > bdbruin@gmail.com <ma...@gmail.com>
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
>
Re: AIP-56 Extensible user management
Posted by Ash Berlin-Taylor <as...@apache.org>.
> - I am fine with (and actually I like it better) removing User and Role related Rest APIs and CLI commands from Airflow. That will indeed make the AIP way simpler and shorter
I don't think we can do that: we still need to consider users when they are just getting started out with Airflow -- if we only cater to giant enterprises with SSO and a central directory we remove our ability for individual or small teams/companies to get started with Airflow.
On May 2 2023, at 8:23 pm, Beck, Vincent <vi...@amazon.com.INVALID> wrote:
> Thank you Jarek and Bolke for taking time to read and provide feedback to the AIP, much appreciated! Multiple points here:
>
> - I am fine with (and actually I like it better) removing User and Role related Rest APIs and CLI commands from Airflow. That will indeed make the AIP way simpler and shorter. As I explained in the AIP comments, I am suggesting though to leave an option to user managers to define additional REST APIs and CLI commands if they need. By default no additional REST API and CLI command would be defined but user managers would have the option to override it. FAB user manager would be an example of user manager overriding such methods. In any case, if additional Rest API or CLI command is defined, this would be done in user managers (as opposed to core Airflow).
> - "the whole "Security" menu should be GONE completely".
> > I agree with this when the user manager is not the FAB one, however we need to introduce a new one so Admins can easily go the external user manager console (e.g. KeyCloak, ...). This new tab would only be a link/redirect to the external console.
>
> - "All those components (and code related to those) should be moved to FAB minimalistic user manager (and removed from Airflow Core)".
> > 100%. This is the main principle of this AIP, anything related to user management will be moved to user managers.
>
> - "AuthN and AuthZ should be completely removed from Airflow".
> > 100%.
>
> - "So for a nice out-of-the-box experience (pip install apache-airflow) a lightweight User Manager could exist as a side project and have some menus via the plugin system until something better comes along.".
> > I agree with you on the long term. Though, as a first iteration I think it is better to define a user manager using FAB as default user manager. The benefit of doing so is to have a backward compatible transition between before AIP and after AIP. This also simplifies the AIP since it is easier to move files from core Airflow to user managers than creating something new. Creating a new security model would be a challenge too. However, once the AIP implemented and merged, we can easily replace this out-of-the-box FAB user manager by something "better" if the communty thinks this is the right way to go.
>
> As always, feel free to give me your thoughts.
> Vincent
> On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com>> wrote:
> I'm not sure if I read the AIP correctly. I think I concur with Jarek here.
>
> AuthN and AuthZ should be completely removed from Airflow. Access
> management should be outsourced to something like Open Policy Agent. In
> other words there should be no user management at all within Airflow. This
> removes the need to "sync" users (and thus become outdated) and reduces the
> attack surface drastically. So for a nice out-of-the-box experience (pip
> install apache-airflow) a lightweight User Manager could exist as a side
> project and have some menus via the plugin system until something better
> comes along. Keycloak is great, a bit heavy for us to package
> unfortunately.
>
>
> FAB's security model is very non-standard and requires 'unnatural' mappings
> in an enterprise context. It has served us well, but I would say good
> riddance.
>
>
> Bolke
>
>
>
>
>
> Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>>:
>
> > I had a look and I liked what I saw for how we should integrate the new
> > user manager for getting and checking the access :).
> >
> > But I think the AIP proposal is not bold enough when it comes to the user
> > management part. This is one big comment - we should simplify it even
> > further and cut more deeply.
> >
> > One of the big changes I think is super important with extensible user
> > management is that we should delegate even more to the external systems
> > managing users. The AIP attempts to map the current API/CLI/UI for managing
> > the users and roles to the new external user management services - but it
> > is IMHO absolutely not needed.
> >
> > As I see the extensible user management is that when you choose something
> > else than the legacy FAB minimalist user management. Airflow should become
> > just "user" of that user/role information managed elsewhere. It should not
> > provide ANY option to see and manage the users or roles, it should merely
> > read the information and give access to the users for appropriate actions,
> > but then all the role/user management should happen outside of Airflow - in
> > the system that is dedicated to manage those.
> >
> > IMHO - when other than the "non FAB minimalistic user manager" system is
> > used. Airflow should just not have a number of functionalities at all:
> >
> > * not have "users" or "roles" CLI groups at all - they should be missing
> > * the
> >
> > https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> > and
> >
> > https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> > should return 404 NOT FOUND
> > * the whole "Security" menu should be GONE completely.
> >
> > All those components (and code related to those) should be moved to FAB
> > minimalistic user manager (and removed from Airflow Core). Eventually maybe
> > even moved out of the Airflow repo altogether and moved to a FAB user
> > manager add-on to airflow.
> >
> > This might seem a drastic change, but in fact - it's not drastic at all. It
> > reflects what many of the Airflow users are doing now - they never access
> > Security/Admin/Roles. Once they integrate with their LDAP or other
> > corporate directory, the Security UI, CLI and REST API are essentially
> > useless (and not used).
> >
> > And it is precisely what we want to achieve - we want to delegate the
> > user/role management completely out of Airflow. In fact - getting rid of
> > those is the only reason we want to implement the change. If we **just**
> > replace the current FAB and keep all the complexity of managing the users
> > /roles in Airflow, I'd say we have not achieved anything but adding
> > complexity.
> >
> > And - eventually - it makes the AIP-56 way *smaller*. We just need to make
> > sure that all the stuff for user management (including REST API, CLI and
> > the UI) is moved to the "FAB minimalistic user manager" (and add ways to
> > plug-it in the core - possibly using existing plugins mechanisms where
> > needed). We do not need to define new API for user/role management. We only
> > need to describe the way how to check if the user has permission to do
> > stuff.
> >
> > J.
> >
> >
> >
> >
> > On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
> >
> > > I promise to provide my feedback by the end of the week, sorry for
> > > delaying it, but we had some 2.6 preparation, branching, feature flags
> > etc.
> > > also for AIP-44 and I am getting back on track with that one soon.
> > >
> > > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham <shubhx@amazon.com.inva <ma...@amazon.com.inva>lid
> > >
> > > wrote:
> > >
> > >> Bumping this up for feedback!
> > >>
> > >> @Vikram, @Kaxil, et al. - I understand that you're quite busy, but we
> > >> would greatly appreciate your feedback here. Your inputs could help us
> > >> improve the proposal and move forward with multi-tenancy work.
> > >>
> > >> Cheers,
> > >> Shubham
> > >>
> > >>
> > >> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> > >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>LID> wrote:
> > >>
> > >>
> > >> CAUTION: This email originated from outside of the organization. Do not
> > >> click links or open attachments unless you can confirm the sender and
> > know
> > >> the content is safe.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> Hi all,
> > >>
> > >>
> > >> I would like to get your feedback on an AIP I wrote: "Extensible user
> > >> management". This AIP is a follow-up on a discussion we had in this
> > email
> > >> list on the multi tenancy topic. I decided to create a new email thread
> > >> because I feel the topic had diverged a bit from the original topic
> > (multi
> > >> tenancy).
> > >>
> > >>
> > >> As a summary, the purpose of this AIP is to extract the user management
> > >> from core Airflow by introducing a user manager interface in the core
> > >> Airflow that can be extended by any provider package who want to support
> > >> user management natively. As a result, if this AIP gets approved and go
> > >> through, the multi tenancy feature will be implemented as a second step
> > in
> > >> a new user manager and not in Airflow directly.
> > >>
> > >>
> > >> As always, feel very free to give your opinion on this email thread or
> > on
> > >> the AIP by adding comments.
> > >>
> > >>
> > >> References:
> > >> - AIP:
> > >>
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > >> <
> > >>
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> > >> >
> > >> - Initial email list discussion:
> > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> > >>
> > >>
> > >> Vincent
> > >>
> > >>
> > >>
> > >>
> >
>
>
>
>
> --
>
> --
> Bolke de Bruin
> bdbruin@gmail.com <ma...@gmail.com>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
Re: AIP-56 Extensible user management
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
Thank you Jarek and Bolke for taking time to read and provide feedback to the AIP, much appreciated! Multiple points here:
- I am fine with (and actually I like it better) removing User and Role related Rest APIs and CLI commands from Airflow. That will indeed make the AIP way simpler and shorter. As I explained in the AIP comments, I am suggesting though to leave an option to user managers to define additional REST APIs and CLI commands if they need. By default no additional REST API and CLI command would be defined but user managers would have the option to override it. FAB user manager would be an example of user manager overriding such methods. In any case, if additional Rest API or CLI command is defined, this would be done in user managers (as opposed to core Airflow).
- "the whole "Security" menu should be GONE completely".
> I agree with this when the user manager is not the FAB one, however we need to introduce a new one so Admins can easily go the external user manager console (e.g. KeyCloak, ...). This new tab would only be a link/redirect to the external console.
- "All those components (and code related to those) should be moved to FAB minimalistic user manager (and removed from Airflow Core)".
> 100%. This is the main principle of this AIP, anything related to user management will be moved to user managers.
- "AuthN and AuthZ should be completely removed from Airflow".
> 100%.
- "So for a nice out-of-the-box experience (pip install apache-airflow) a lightweight User Manager could exist as a side project and have some menus via the plugin system until something better comes along.".
> I agree with you on the long term. Though, as a first iteration I think it is better to define a user manager using FAB as default user manager. The benefit of doing so is to have a backward compatible transition between before AIP and after AIP. This also simplifies the AIP since it is easier to move files from core Airflow to user managers than creating something new. Creating a new security model would be a challenge too. However, once the AIP implemented and merged, we can easily replace this out-of-the-box FAB user manager by something "better" if the communty thinks this is the right way to go.
As always, feel free to give me your thoughts.
Vincent
On 2023-04-25, 2:00 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com>> wrote:
I'm not sure if I read the AIP correctly. I think I concur with Jarek here.
AuthN and AuthZ should be completely removed from Airflow. Access
management should be outsourced to something like Open Policy Agent. In
other words there should be no user management at all within Airflow. This
removes the need to "sync" users (and thus become outdated) and reduces the
attack surface drastically. So for a nice out-of-the-box experience (pip
install apache-airflow) a lightweight User Manager could exist as a side
project and have some menus via the plugin system until something better
comes along. Keycloak is great, a bit heavy for us to package
unfortunately.
FAB's security model is very non-standard and requires 'unnatural' mappings
in an enterprise context. It has served us well, but I would say good
riddance.
Bolke
Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>>:
> I had a look and I liked what I saw for how we should integrate the new
> user manager for getting and checking the access :).
>
> But I think the AIP proposal is not bold enough when it comes to the user
> management part. This is one big comment - we should simplify it even
> further and cut more deeply.
>
> One of the big changes I think is super important with extensible user
> management is that we should delegate even more to the external systems
> managing users. The AIP attempts to map the current API/CLI/UI for managing
> the users and roles to the new external user management services - but it
> is IMHO absolutely not needed.
>
> As I see the extensible user management is that when you choose something
> else than the legacy FAB minimalist user management. Airflow should become
> just "user" of that user/role information managed elsewhere. It should not
> provide ANY option to see and manage the users or roles, it should merely
> read the information and give access to the users for appropriate actions,
> but then all the role/user management should happen outside of Airflow - in
> the system that is dedicated to manage those.
>
> IMHO - when other than the "non FAB minimalistic user manager" system is
> used. Airflow should just not have a number of functionalities at all:
>
> * not have "users" or "roles" CLI groups at all - they should be missing
> * the
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User>
> and
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role <https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role>
> should return 404 NOT FOUND
> * the whole "Security" menu should be GONE completely.
>
> All those components (and code related to those) should be moved to FAB
> minimalistic user manager (and removed from Airflow Core). Eventually maybe
> even moved out of the Airflow repo altogether and moved to a FAB user
> manager add-on to airflow.
>
> This might seem a drastic change, but in fact - it's not drastic at all. It
> reflects what many of the Airflow users are doing now - they never access
> Security/Admin/Roles. Once they integrate with their LDAP or other
> corporate directory, the Security UI, CLI and REST API are essentially
> useless (and not used).
>
> And it is precisely what we want to achieve - we want to delegate the
> user/role management completely out of Airflow. In fact - getting rid of
> those is the only reason we want to implement the change. If we **just**
> replace the current FAB and keep all the complexity of managing the users
> /roles in Airflow, I'd say we have not achieved anything but adding
> complexity.
>
> And - eventually - it makes the AIP-56 way *smaller*. We just need to make
> sure that all the stuff for user management (including REST API, CLI and
> the UI) is moved to the "FAB minimalistic user manager" (and add ways to
> plug-it in the core - possibly using existing plugins mechanisms where
> needed). We do not need to define new API for user/role management. We only
> need to describe the way how to check if the user has permission to do
> stuff.
>
> J.
>
>
>
>
> On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
>
> > I promise to provide my feedback by the end of the week, sorry for
> > delaying it, but we had some 2.6 preparation, branching, feature flags
> etc.
> > also for AIP-44 and I am getting back on track with that one soon.
> >
> > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham <shubhx@amazon.com.inva <ma...@amazon.com.inva>lid
> >
> > wrote:
> >
> >> Bumping this up for feedback!
> >>
> >> @Vikram, @Kaxil, et al. - I understand that you're quite busy, but we
> >> would greatly appreciate your feedback here. Your inputs could help us
> >> improve the proposal and move forward with multi-tenancy work.
> >>
> >> Cheers,
> >> Shubham
> >>
> >>
> >> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>
> >> <mailto:vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>>LID> wrote:
> >>
> >>
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you can confirm the sender and
> know
> >> the content is safe.
> >>
> >>
> >>
> >>
> >>
> >>
> >> Hi all,
> >>
> >>
> >> I would like to get your feedback on an AIP I wrote: "Extensible user
> >> management". This AIP is a follow-up on a discussion we had in this
> email
> >> list on the multi tenancy topic. I decided to create a new email thread
> >> because I feel the topic had diverged a bit from the original topic
> (multi
> >> tenancy).
> >>
> >>
> >> As a summary, the purpose of this AIP is to extract the user management
> >> from core Airflow by introducing a user manager interface in the core
> >> Airflow that can be extended by any provider package who want to support
> >> user management natively. As a result, if this AIP gets approved and go
> >> through, the multi tenancy feature will be implemented as a second step
> in
> >> a new user manager and not in Airflow directly.
> >>
> >>
> >> As always, feel very free to give your opinion on this email thread or
> on
> >> the AIP by adding comments.
> >>
> >>
> >> References:
> >> - AIP:
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
> >> >
> >> - Initial email list discussion:
> >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <
> >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>>
> >>
> >>
> >> Vincent
> >>
> >>
> >>
> >>
>
--
--
Bolke de Bruin
bdbruin@gmail.com <ma...@gmail.com>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: AIP-56 Extensible user management
Posted by Bolke de Bruin <bd...@gmail.com>.
I'm not sure if I read the AIP correctly. I think I concur with Jarek here.
AuthN and AuthZ should be completely removed from Airflow. Access
management should be outsourced to something like Open Policy Agent. In
other words there should be no user management at all within Airflow. This
removes the need to "sync" users (and thus become outdated) and reduces the
attack surface drastically. So for a nice out-of-the-box experience (pip
install apache-airflow) a lightweight User Manager could exist as a side
project and have some menus via the plugin system until something better
comes along. Keycloak is great, a bit heavy for us to package
unfortunately.
FAB's security model is very non-standard and requires 'unnatural' mappings
in an enterprise context. It has served us well, but I would say good
riddance.
Bolke
Op ma 24 apr 2023 om 08:39 schreef Jarek Potiuk <ja...@potiuk.com>:
> I had a look and I liked what I saw for how we should integrate the new
> user manager for getting and checking the access :).
>
> But I think the AIP proposal is not bold enough when it comes to the user
> management part. This is one big comment - we should simplify it even
> further and cut more deeply.
>
> One of the big changes I think is super important with extensible user
> management is that we should delegate even more to the external systems
> managing users. The AIP attempts to map the current API/CLI/UI for managing
> the users and roles to the new external user management services - but it
> is IMHO absolutely not needed.
>
> As I see the extensible user management is that when you choose something
> else than the legacy FAB minimalist user management. Airflow should become
> just "user" of that user/role information managed elsewhere. It should not
> provide ANY option to see and manage the users or roles, it should merely
> read the information and give access to the users for appropriate actions,
> but then all the role/user management should happen outside of Airflow - in
> the system that is dedicated to manage those.
>
> IMHO - when other than the "non FAB minimalistic user manager" system is
> used. Airflow should just not have a number of functionalities at all:
>
> * not have "users" or "roles" CLI groups at all - they should be missing
> * the
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
> and
>
> https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
> should return 404 NOT FOUND
> * the whole "Security" menu should be GONE completely.
>
> All those components (and code related to those) should be moved to FAB
> minimalistic user manager (and removed from Airflow Core). Eventually maybe
> even moved out of the Airflow repo altogether and moved to a FAB user
> manager add-on to airflow.
>
> This might seem a drastic change, but in fact - it's not drastic at all. It
> reflects what many of the Airflow users are doing now - they never access
> Security/Admin/Roles. Once they integrate with their LDAP or other
> corporate directory, the Security UI, CLI and REST API are essentially
> useless (and not used).
>
> And it is precisely what we want to achieve - we want to delegate the
> user/role management completely out of Airflow. In fact - getting rid of
> those is the only reason we want to implement the change. If we **just**
> replace the current FAB and keep all the complexity of managing the users
> /roles in Airflow, I'd say we have not achieved anything but adding
> complexity.
>
> And - eventually - it makes the AIP-56 way *smaller*. We just need to make
> sure that all the stuff for user management (including REST API, CLI and
> the UI) is moved to the "FAB minimalistic user manager" (and add ways to
> plug-it in the core - possibly using existing plugins mechanisms where
> needed). We do not need to define new API for user/role management. We only
> need to describe the way how to check if the user has permission to do
> stuff.
>
> J.
>
>
>
>
> On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I promise to provide my feedback by the end of the week, sorry for
> > delaying it, but we had some 2.6 preparation, branching, feature flags
> etc.
> > also for AIP-44 and I am getting back on track with that one soon.
> >
> > On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham <shubhx@amazon.com.invalid
> >
> > wrote:
> >
> >> Bumping this up for feedback!
> >>
> >> @Vikram, @Kaxil, et al. - I understand that you're quite busy, but we
> >> would greatly appreciate your feedback here. Your inputs could help us
> >> improve the proposal and move forward with multi-tenancy work.
> >>
> >> Cheers,
> >> Shubham
> >>
> >>
> >> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA
> >> <ma...@amazon.com.INVA>LID> wrote:
> >>
> >>
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you can confirm the sender and
> know
> >> the content is safe.
> >>
> >>
> >>
> >>
> >>
> >>
> >> Hi all,
> >>
> >>
> >> I would like to get your feedback on an AIP I wrote: "Extensible user
> >> management". This AIP is a follow-up on a discussion we had in this
> email
> >> list on the multi tenancy topic. I decided to create a new email thread
> >> because I feel the topic had diverged a bit from the original topic
> (multi
> >> tenancy).
> >>
> >>
> >> As a summary, the purpose of this AIP is to extract the user management
> >> from core Airflow by introducing a user manager interface in the core
> >> Airflow that can be extended by any provider package who want to support
> >> user management natively. As a result, if this AIP gets approved and go
> >> through, the multi tenancy feature will be implemented as a second step
> in
> >> a new user manager and not in Airflow directly.
> >>
> >>
> >> As always, feel very free to give your opinion on this email thread or
> on
> >> the AIP by adding comments.
> >>
> >>
> >> References:
> >> - AIP:
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >> <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >> >
> >> - Initial email list discussion:
> >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <
> >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
> >>
> >>
> >> Vincent
> >>
> >>
> >>
> >>
>
--
--
Bolke de Bruin
bdbruin@gmail.com
Re: AIP-56 Extensible user management
Posted by Jarek Potiuk <ja...@potiuk.com>.
I had a look and I liked what I saw for how we should integrate the new
user manager for getting and checking the access :).
But I think the AIP proposal is not bold enough when it comes to the user
management part. This is one big comment - we should simplify it even
further and cut more deeply.
One of the big changes I think is super important with extensible user
management is that we should delegate even more to the external systems
managing users. The AIP attempts to map the current API/CLI/UI for managing
the users and roles to the new external user management services - but it
is IMHO absolutely not needed.
As I see the extensible user management is that when you choose something
else than the legacy FAB minimalist user management. Airflow should become
just "user" of that user/role information managed elsewhere. It should not
provide ANY option to see and manage the users or roles, it should merely
read the information and give access to the users for appropriate actions,
but then all the role/user management should happen outside of Airflow - in
the system that is dedicated to manage those.
IMHO - when other than the "non FAB minimalistic user manager" system is
used. Airflow should just not have a number of functionalities at all:
* not have "users" or "roles" CLI groups at all - they should be missing
* the
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/User
and
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Role
should return 404 NOT FOUND
* the whole "Security" menu should be GONE completely.
All those components (and code related to those) should be moved to FAB
minimalistic user manager (and removed from Airflow Core). Eventually maybe
even moved out of the Airflow repo altogether and moved to a FAB user
manager add-on to airflow.
This might seem a drastic change, but in fact - it's not drastic at all. It
reflects what many of the Airflow users are doing now - they never access
Security/Admin/Roles. Once they integrate with their LDAP or other
corporate directory, the Security UI, CLI and REST API are essentially
useless (and not used).
And it is precisely what we want to achieve - we want to delegate the
user/role management completely out of Airflow. In fact - getting rid of
those is the only reason we want to implement the change. If we **just**
replace the current FAB and keep all the complexity of managing the users
/roles in Airflow, I'd say we have not achieved anything but adding
complexity.
And - eventually - it makes the AIP-56 way *smaller*. We just need to make
sure that all the stuff for user management (including REST API, CLI and
the UI) is moved to the "FAB minimalistic user manager" (and add ways to
plug-it in the core - possibly using existing plugins mechanisms where
needed). We do not need to define new API for user/role management. We only
need to describe the way how to check if the user has permission to do
stuff.
J.
On Mon, Apr 17, 2023 at 12:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> I promise to provide my feedback by the end of the week, sorry for
> delaying it, but we had some 2.6 preparation, branching, feature flags etc.
> also for AIP-44 and I am getting back on track with that one soon.
>
> On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham <sh...@amazon.com.invalid>
> wrote:
>
>> Bumping this up for feedback!
>>
>> @Vikram, @Kaxil, et al. - I understand that you're quite busy, but we
>> would greatly appreciate your feedback here. Your inputs could help us
>> improve the proposal and move forward with multi-tenancy work.
>>
>> Cheers,
>> Shubham
>>
>>
>> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA
>> <ma...@amazon.com.INVA>LID> wrote:
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>>
>>
>>
>> Hi all,
>>
>>
>> I would like to get your feedback on an AIP I wrote: "Extensible user
>> management". This AIP is a follow-up on a discussion we had in this email
>> list on the multi tenancy topic. I decided to create a new email thread
>> because I feel the topic had diverged a bit from the original topic (multi
>> tenancy).
>>
>>
>> As a summary, the purpose of this AIP is to extract the user management
>> from core Airflow by introducing a user manager interface in the core
>> Airflow that can be extended by any provider package who want to support
>> user management natively. As a result, if this AIP gets approved and go
>> through, the multi tenancy feature will be implemented as a second step in
>> a new user manager and not in Airflow directly.
>>
>>
>> As always, feel very free to give your opinion on this email thread or on
>> the AIP by adding comments.
>>
>>
>> References:
>> - AIP:
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
>> <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
>> >
>> - Initial email list discussion:
>> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <
>> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
>>
>>
>> Vincent
>>
>>
>>
>>
Re: AIP-56 Extensible user management
Posted by Jarek Potiuk <ja...@potiuk.com>.
I promise to provide my feedback by the end of the week, sorry for delaying
it, but we had some 2.6 preparation, branching, feature flags etc. also for
AIP-44 and I am getting back on track with that one soon.
On Fri, Mar 31, 2023 at 9:35 AM Mehta, Shubham <sh...@amazon.com.invalid>
wrote:
> Bumping this up for feedback!
>
> @Vikram, @Kaxil, et al. - I understand that you're quite busy, but we
> would greatly appreciate your feedback here. Your inputs could help us
> improve the proposal and move forward with multi-tenancy work.
>
> Cheers,
> Shubham
>
>
> On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA
> <ma...@amazon.com.INVA>LID> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> Hi all,
>
>
> I would like to get your feedback on an AIP I wrote: "Extensible user
> management". This AIP is a follow-up on a discussion we had in this email
> list on the multi tenancy topic. I decided to create a new email thread
> because I feel the topic had diverged a bit from the original topic (multi
> tenancy).
>
>
> As a summary, the purpose of this AIP is to extract the user management
> from core Airflow by introducing a user manager interface in the core
> Airflow that can be extended by any provider package who want to support
> user management natively. As a result, if this AIP gets approved and go
> through, the multi tenancy feature will be implemented as a second step in
> a new user manager and not in Airflow directly.
>
>
> As always, feel very free to give your opinion on this email thread or on
> the AIP by adding comments.
>
>
> References:
> - AIP:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management
> >
> - Initial email list discussion:
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <
> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
>
>
> Vincent
>
>
>
>
Re: AIP-56 Extensible user management
Posted by "Mehta, Shubham" <sh...@amazon.com.INVALID>.
Bumping this up for feedback!
@Vikram, @Kaxil, et al. - I understand that you're quite busy, but we would greatly appreciate your feedback here. Your inputs could help us improve the proposal and move forward with multi-tenancy work.
Cheers,
Shubham
On 2023-03-28, 2:05 PM, "Beck, Vincent" <vincbeck@amazon.com.INVA <ma...@amazon.com.INVA>LID> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi all,
I would like to get your feedback on an AIP I wrote: "Extensible user management". This AIP is a follow-up on a discussion we had in this email list on the multi tenancy topic. I decided to create a new email thread because I feel the topic had diverged a bit from the original topic (multi tenancy).
As a summary, the purpose of this AIP is to extract the user management from core Airflow by introducing a user manager interface in the core Airflow that can be extended by any provider package who want to support user management natively. As a result, if this AIP gets approved and go through, the multi tenancy feature will be implemented as a second step in a new user manager and not in Airflow directly.
As always, feel very free to give your opinion on this email thread or on the AIP by adding comments.
References:
- AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management>
- Initial email list discussion: https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 <https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61>
Vincent