Upload DAG / owner / multi-tenancy

[Thomas Weise <th...@apache.org>]

Hi,

Dropping files into dags_folder is a convenient way to deploy new DAGs, but
this way they are owned by the respective OS user. I'm looking for a
multi-tenant setup with multiple "owners", where each can add new DAGs and
see only their DAGs in the UI.

Is this supported?

Thanks,
Thomas

Re: Upload DAG / owner / multi-tenancy

[Thomas Weise <th...@apache.org>]

Thanks for the reply. I got till the point where I separate owners by
subfolder and the DAGs are also picked up from there. The problem is that
owner and DAG name are embedded in the script. The user can set owner to
anything and the DAG names have to be globally unique to avoid collision
between users (since there isn't a namespace per owner).

I assume this is hard to solve w/o support in Airflow. Is there any option
other than parsing the scripts to verify owner and uniqueness of DAG name
prior to admitting them into dags_folder?

Thanks!


On Thu, Mar 30, 2017 at 6:02 PM, Bolke de Bruin <bd...@gmail.com> wrote:

> No not yet. It is planned for maybe 1.9 or 1.10. You can do this with git
> or any other artifact manager though.
>
> B.
>
> Sent from my iPhone
>
> > On 30 Mar 2017, at 16:36, Thomas Weise <th...@apache.org> wrote:
> >
> > Hi,
> >
> > Dropping files into dags_folder is a convenient way to deploy new DAGs,
> but
> > this way they are owned by the respective OS user. I'm looking for a
> > multi-tenant setup with multiple "owners", where each can add new DAGs
> and
> > see only their DAGs in the UI.
> >
> > Is this supported?
> >
> > Thanks,
> > Thomas
>

Re: Upload DAG / owner / multi-tenancy

[Ruslan Dautkhanov <da...@gmail.com>]

It would be interesting to see how Kerberos setup will be implemented in
multi-tenant Airflow. Would it be just one keytab for a "service account"
and then authentication will impersonate to actual users? Similar approach
is done by some other projects, I believe, for example, Livy, Hue and some
others.


On Thu, Mar 30, 2017 at 7:03 PM Bolke de Bruin <bd...@gmail.com> wrote:

> No not yet. It is planned for maybe 1.9 or 1.10. You can do this with git
> or any other artifact manager though.
>
> B.
>
> Sent from my iPhone
>
> > On 30 Mar 2017, at 16:36, Thomas Weise <th...@apache.org> wrote:
> >
> > Hi,
> >
> > Dropping files into dags_folder is a convenient way to deploy new DAGs,
> but
> > this way they are owned by the respective OS user. I'm looking for a
> > multi-tenant setup with multiple "owners", where each can add new DAGs
> and
> > see only their DAGs in the UI.
> >
> > Is this supported?
> >
> > Thanks,
> > Thomas
>

Re: Upload DAG / owner / multi-tenancy

[Bolke de Bruin <bd...@gmail.com>]

No not yet. It is planned for maybe 1.9 or 1.10. You can do this with git or any other artifact manager though. 

B. 

Sent from my iPhone

> On 30 Mar 2017, at 16:36, Thomas Weise <th...@apache.org> wrote:
> 
> Hi,
> 
> Dropping files into dags_folder is a convenient way to deploy new DAGs, but
> this way they are owned by the respective OS user. I'm looking for a
> multi-tenant setup with multiple "owners", where each can add new DAGs and
> see only their DAGs in the UI.
> 
> Is this supported?
> 
> Thanks,
> Thomas