You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Song Liu <so...@outlook.com> on 2018/02/02 04:12:09 UTC
答复: About multi user support in airflow
Hi Trent,
One way is to deploy one standalone airflow components (scheduler, worker, broker, mysql etc.) in docker for every single user, but there needs to only one web service for user accessing, that would mean the web service should be interact with multiple other worker / scheduler for individual user, do I understand correctly ?
For Flask, how could the multi-user feature be supported ? could you help clarify ?
Also, could you help share what's the reason/design behind why it doesn't support multi-user feature in airflow ?
Thanks,
Song
________________________________
发件人: Trent Robbins <ro...@gmail.com>
发送时间: 2018年2月2日 3:50
收件人: dev@airflow.incubator.apache.org
主题: Re: About multi user support in airflow
Hi Song,
I would recommend using airflow in docker with individual databases for
individual contributors. Does this make sense for your needs? There are
other similar concepts that could also be used.
I don't think there is another way to split DAGs by user permission.
However, this feature doesn't seem particularly challenging to implement
since it uses Flask. But who knows!
Trent
On Thu, Feb 1, 2018 at 19:24 Song Liu <so...@outlook.com> wrote:
> Hi,
>
> In production, multi-user support is needed, which means that every user
> could login the airflow platform, and could manage their own dags
> separately.
>
> But currently it seem that all the dags are managed into a single dags
> folder, so for multi-user support what I need to do ?
>
> Many thanks for helping out.
>
> Thanks,
> Song
>
--
(Sent from cellphone)
答复: About multi user support in airflow
Posted by Song Liu <so...@outlook.com>.
Hi Trent,
Thanks for your sharing.
Multi-tenancy
You can filter the list of dags in webserver by owner name when authentication is turned on by setting webserver:filter_by_owner in your config. With this, a user will see only the dags which it is owner of, unless it is a superuser.
[webserver]
filter_by_owner = True
It seems that it could be achieved by the filter_by_owner, I would try is this usable already or not.
But here I still have another question about the dags management:
1- for the cluster deployment, what's the better way to sync among the scheduler and other workers ? and why not build-in this sync mechanism ?
2- one real case that when I add a new dag into dags folder, I need refresh many times via. web UI to see it and it will appear the notice about "scheduler seems doesn't see this dag locally", I want to know when a new dag added, how does the scheduler will do ? will it parse the new added dag only ?
Thanks again.
Thanks,
Song
________________________________
发件人: Trent Robbins <ro...@gmail.com>
发送时间: 2018年2月2日 4:19
收件人: Song Liu
抄送: dev@airflow.incubator.apache.org
主题: Re: About multi user support in airflow
Hi Song,
It looks like there is a basic "owner" DAG filter feature already available but I do not know how detailed the implementation is: https://airflow.apache.org/security.html
I can't speak for the developers but I assume that a fully built out feature hasn't been an extremely high priority for anyone or perhaps there is more code support than I know about and it isn't fully documented.
From a quick scan of the project I see that flask-login is installed during setup although that doesn't necessarily mean that it plays a role in these features.
Best,
Trent
On Thu, Feb 1, 2018 at 20:12 Song Liu <so...@outlook.com>> wrote:
Hi Trent,
One way is to deploy one standalone airflow components (scheduler, worker, broker, mysql etc.) in docker for every single user, but there needs to only one web service for user accessing, that would mean the web service should be interact with multiple other worker / scheduler for individual user, do I understand correctly ?
For Flask, how could the multi-user feature be supported ? could you help clarify ?
Also, could you help share what's the reason/design behind why it doesn't support multi-user feature in airflow ?
Thanks,
Song
________________________________
发件人: Trent Robbins <ro...@gmail.com>>
发送时间: 2018年2月2日 3:50
收件人: dev@airflow.incubator.apache.org<ma...@airflow.incubator.apache.org>
主题: Re: About multi user support in airflow
Hi Song,
I would recommend using airflow in docker with individual databases for
individual contributors. Does this make sense for your needs? There are
other similar concepts that could also be used.
I don't think there is another way to split DAGs by user permission.
However, this feature doesn't seem particularly challenging to implement
since it uses Flask. But who knows!
Trent
On Thu, Feb 1, 2018 at 19:24 Song Liu <so...@outlook.com>> wrote:
> Hi,
>
> In production, multi-user support is needed, which means that every user
> could login the airflow platform, and could manage their own dags
> separately.
>
> But currently it seem that all the dags are managed into a single dags
> folder, so for multi-user support what I need to do ?
>
> Many thanks for helping out.
>
> Thanks,
> Song
>
--
(Sent from cellphone)
--
(Sent from cellphone)
Re: About multi user support in airflow
Posted by Trent Robbins <ro...@gmail.com>.
Hi Song,
It looks like there is a basic "owner" DAG filter feature already available
but I do not know how detailed the implementation is:
https://airflow.apache.org/security.html
I can't speak for the developers but I assume that a fully built out
feature hasn't been an extremely high priority for anyone or perhaps there
is more code support than I know about and it isn't fully documented.
From a quick scan of the project I see that flask-login is installed during
setup although that doesn't necessarily mean that it plays a role in these
features.
Best,
Trent
On Thu, Feb 1, 2018 at 20:12 Song Liu <so...@outlook.com> wrote:
> Hi Trent,
>
> One way is to deploy one standalone airflow components (scheduler, worker,
> broker, mysql etc.) in docker for every single user, but there needs to
> only one web service for user accessing, that would mean the web service
> should be interact with multiple other worker / scheduler for individual
> user, do I understand correctly ?
>
> For Flask, how could the multi-user feature be supported ? could you help
> clarify ?
>
> Also, could you help share what's the reason/design behind why it doesn't
> support multi-user feature in airflow ?
>
> Thanks,
> Song
> ------------------------------
> *发件人:* Trent Robbins <ro...@gmail.com>
> *发送时间:* 2018年2月2日 3:50
> *收件人:* dev@airflow.incubator.apache.org
> *主题:* Re: About multi user support in airflow
>
> Hi Song,
>
> I would recommend using airflow in docker with individual databases for
> individual contributors. Does this make sense for your needs? There are
> other similar concepts that could also be used.
>
> I don't think there is another way to split DAGs by user permission.
> However, this feature doesn't seem particularly challenging to implement
> since it uses Flask. But who knows!
>
> Trent
>
> On Thu, Feb 1, 2018 at 19:24 Song Liu <so...@outlook.com> wrote:
>
> > Hi,
> >
> > In production, multi-user support is needed, which means that every user
> > could login the airflow platform, and could manage their own dags
> > separately.
> >
> > But currently it seem that all the dags are managed into a single dags
> > folder, so for multi-user support what I need to do ?
> >
> > Many thanks for helping out.
> >
> > Thanks,
> > Song
> >
> --
> (Sent from cellphone)
>
--
(Sent from cellphone)
答复: 答复: About multi user support in airflow
Posted by Song Liu <so...@outlook.com>.
Hi Chris & Joy,
About the access control feature in new web-server, is it kind of a filter mechanism based on the rights ? will the dag storage be separated per user also ?
Also for the multi-user support, how to isolate the execution space for different user ? for example, if one user execute one bash to delete other user's data.
Or from the design perspective, is the airflow aiming to serve multi-user ?
Thanks,
Song
________________________________
发件人: Chris Riccomini <cr...@apache.org>
发送时间: 2018年2月2日 15:28
收件人: dev@airflow.incubator.apache.org
抄送: Trent Robbins
主题: Re: 答复: About multi user support in airflow
@song, you might also be interested in the work that Joy Gao is doing to
add more access controls to the UI:
https://github.com/wepay/airflow-webserver
On Thu, Feb 1, 2018 at 8:12 PM, Song Liu <so...@outlook.com> wrote:
> Hi Trent,
>
> One way is to deploy one standalone airflow components (scheduler, worker,
> broker, mysql etc.) in docker for every single user, but there needs to
> only one web service for user accessing, that would mean the web service
> should be interact with multiple other worker / scheduler for individual
> user, do I understand correctly ?
>
> For Flask, how could the multi-user feature be supported ? could you help
> clarify ?
>
> Also, could you help share what's the reason/design behind why it doesn't
> support multi-user feature in airflow ?
>
> Thanks,
> Song
> ________________________________
> 发件人: Trent Robbins <ro...@gmail.com>
> 发送时间: 2018年2月2日 3:50
> 收件人: dev@airflow.incubator.apache.org
> 主题: Re: About multi user support in airflow
>
> Hi Song,
>
> I would recommend using airflow in docker with individual databases for
> individual contributors. Does this make sense for your needs? There are
> other similar concepts that could also be used.
>
> I don't think there is another way to split DAGs by user permission.
> However, this feature doesn't seem particularly challenging to implement
> since it uses Flask. But who knows!
>
> Trent
>
> On Thu, Feb 1, 2018 at 19:24 Song Liu <so...@outlook.com> wrote:
>
> > Hi,
> >
> > In production, multi-user support is needed, which means that every user
> > could login the airflow platform, and could manage their own dags
> > separately.
> >
> > But currently it seem that all the dags are managed into a single dags
> > folder, so for multi-user support what I need to do ?
> >
> > Many thanks for helping out.
> >
> > Thanks,
> > Song
> >
> --
> (Sent from cellphone)
>
Re: 答复: About multi user support in airflow
Posted by Chris Riccomini <cr...@apache.org>.
@song, you might also be interested in the work that Joy Gao is doing to
add more access controls to the UI:
https://github.com/wepay/airflow-webserver
On Thu, Feb 1, 2018 at 8:12 PM, Song Liu <so...@outlook.com> wrote:
> Hi Trent,
>
> One way is to deploy one standalone airflow components (scheduler, worker,
> broker, mysql etc.) in docker for every single user, but there needs to
> only one web service for user accessing, that would mean the web service
> should be interact with multiple other worker / scheduler for individual
> user, do I understand correctly ?
>
> For Flask, how could the multi-user feature be supported ? could you help
> clarify ?
>
> Also, could you help share what's the reason/design behind why it doesn't
> support multi-user feature in airflow ?
>
> Thanks,
> Song
> ________________________________
> 发件人: Trent Robbins <ro...@gmail.com>
> 发送时间: 2018年2月2日 3:50
> 收件人: dev@airflow.incubator.apache.org
> 主题: Re: About multi user support in airflow
>
> Hi Song,
>
> I would recommend using airflow in docker with individual databases for
> individual contributors. Does this make sense for your needs? There are
> other similar concepts that could also be used.
>
> I don't think there is another way to split DAGs by user permission.
> However, this feature doesn't seem particularly challenging to implement
> since it uses Flask. But who knows!
>
> Trent
>
> On Thu, Feb 1, 2018 at 19:24 Song Liu <so...@outlook.com> wrote:
>
> > Hi,
> >
> > In production, multi-user support is needed, which means that every user
> > could login the airflow platform, and could manage their own dags
> > separately.
> >
> > But currently it seem that all the dags are managed into a single dags
> > folder, so for multi-user support what I need to do ?
> >
> > Many thanks for helping out.
> >
> > Thanks,
> > Song
> >
> --
> (Sent from cellphone)
>