You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Joy Gao <jo...@wepay.com> on 2017/07/20 02:33:46 UTC

Re: Role Based Access Control for Airflow UI

Hey everyone,

I recently transferred to Data Infra team here at WePay to focus on
Airflow-related initiatives.

Given the RBAC design is mostly hashed out, I'm happy to get this feature
off the ground for Q3, starting with converting Airflow to Fab, if there
are no objections.

Cheers,
Joy

On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
gurer.kiratli@airbnb.com.invalid> wrote:

> Hey all,
>
> We talked about this internally. We would like to work on this feature but
> given the immediate priorities we are not going to be working on it in Q3.
> Comes end of Q3 we will reevaluate. Likely scenario is we can work on it
> late Q4 or Q12018.
>
> Cheers,
>
> Gurer
>
> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > I think FAB sounds like the right approach. Waiting to hear back with
> notes
> > on AirBNB H2 discussion to see if they want to take this up.
> >
> > @Gurer, any idea when this will happen?
> >
> > On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> > > One downside I see from FAB is that is does not do Business Role
> mapping
> > > to FAB role. I would prefer to create groups in IPA/LDAP/AD and have
> > those
> > > map to FAB roles instead of needing to manage that in FAB.
> > >
> > > B.
> > >
> > > > On 22 Jun 2017, at 09:36, Bolke de Bruin <bd...@gmail.com> wrote:
> > > >
> > > > Hi Guys,
> > > >
> > > > Thanks for putting the thinking in! It is about time that we get this
> > > moving.
> > > >
> > > > The design looks pretty sound. One can argue about the different
> roles
> > > that are required, but that will be situation dependent I guess.
> > > >
> > > > Implementation wise I would argue together with Max that FAB is a
> > better
> > > or best fit. The ER model that is being described is pretty much a copy
> > of
> > > a normal security model. So a reimplementation of that is 1)
> significant
> > > duplication of effort and 2) bound to have bugs that have been solved
> in
> > > the other framework. Moreover, FAB does have integration out of the box
> > > with some enterprisey systems like IPA, ActiveDirectory, and LDAP.
> > > >
> > > > So while you argue that using FAB would increase the scope of the
> > > proposal significantly, but I think that is not true. Using FAB would
> > allow
> > > you to focus on what kind of out-of-the-box permission sets and roles
> we
> > > would need and maybe address some issues that FAB lacks (maybe how to
> > deal
> > > with non web access - ie. in DAGs, maybe Kerberos, probably how to deal
> > > with API calls that are not CRUD). Implementation wise it probably
> > > simplifies what we need to do. Maybe - using Max’s early POC as an
> > example
> > > - we can slowly move over?
> > > >
> > > > On a side note: Im planning to hire 2-3 ppl to work on Airflow coming
> > > year. Improvement of Security, Enterprise Integration, Revamp UI are on
> > the
> > > todo list. However, this is not confirmed yet as business priorities
> > might
> > > change.
> > > >
> > > > Bolke.
> > > >
> > > >
> > > >> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> > > kalpeshdharwadkar@gmail.com> wrote:
> > > >>
> > > >> @Dan:
> > > >>
> > > >> Thanks for your feedback. I will remove the REFRESH_DAG permission.
> > > >>
> > > >> @Max:
> > > >>
> > > >> Thanks for your response.
> > > >>
> > > >> The scope of my proposal was just to add RBAC security feature to
> > > Airflow
> > > >> without replacing any existing frameworks.
> > > >>
> > > >> I understand that adopting FAB would serve Airflow better moving
> > > forward,
> > > >> however porting Airflow to using FAB significantly increases the
> scope
> > > of
> > > >> the proposal and I don't have the time and expertise to carry out
> the
> > > tasks
> > > >> in the extended scope.
> > > >>
> > > >> Hence, I'm curious to know if there's a plan for Airflow to migrate
> to
> > > FAB
> > > >> this year?
> > > >>
> > > >> - Kalpesh
> > > >>
> > > >> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
> > > >> maximebeauchemin@gmail.com> wrote:
> > > >>
> > > >>> It would be nice to go with a framework for this. I did some
> > > >>> experimentation using FlaskAppBuilder to go in this direction. It
> > > provides
> > > >>> auth on different authentication backends out of the box (oauth,
> > > openid,
> > > >>> ldap, registration, ...), generates perms for each view that has an
> > > >>> @has_access decorator, generates at set of perms for each ORM model
> > > (show,
> > > >>> edit, delete, add, ...) and enforces it in the CRUD views as well
> as
> > > in the
> > > >>> generated REST api that you get for free as a byprdoduct of
> deriving
> > > FAB's
> > > >>> models (essentially it's SqlAlchemy with a layer on top).
> > > >>>
> > > >>> I started a POC on FAB here a while ago:
> > > >>> https://github.com/mistercrunch/airflow_webserver at the time my
> > main
> > > >>> motivation was the free/instantaneous REST api.
> > > >>>
> > > >>> I think FAB is a decent fit as the porting should be fairly
> > > straightforward
> > > >>> (moving the flask views over and deprecating Flask-Admin in favor
> of
> > > FAB's
> > > >>> crud) though there was a few blockers. From memory I think FAB
> didn't
> > > like
> > > >>> the compound PKs we use in some of the Airflow models. We'd have to
> > > either
> > > >>> write a db migration script on the Airflow side, or add support for
> > > >>> compound keys to FAB (I recently became a maintainer of the
> project,
> > > so I
> > > >>> could help with that)
> > > >>>
> > > >>> The only downside of FAB is that it's not as mature as something
> like
> > > >>> Django, but porting to Django would surely be much more work.
> > > >>>
> > > >>> Then there's the flask-security suite, but that looks like a bit
> of a
> > > >>> patchwork to me, I guess we can pick and choose which we want to
> use.
> > > >>>
> > > >>> Max
> > > >>>
> > > >>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> > > >>> dan.davydov@airbnb.com.invalid> wrote:
> > > >>>
> > > >>>> Looks good to me in general, thanks for putting this together!
> > > >>>>
> > > >>>> I think the ability to integrate with external RBAC systems like
> > LDAP
> > > is
> > > >>>> important (i.e. the Airflow DB should not be decoupled with the
> RBAC
> > > >>>> database wherever possible).
> > > >>>>
> > > >>>> I wouldn't be too worried about the permissions about refreshing
> > > DAGs, as
> > > >>>> far as I know this functionality is no longer required with the
> new
> > > >>>> webservers which reload state periodically, and will certainly be
> > > removed
> > > >>>> when we have a better DAG consistency story.
> > > >>>>
> > > >>>> I think it would also be good to think about this
> > > proposal/implementation
> > > >>>> and how it applied in the API-driven world (e.g. when webserver
> hits
> > > APIs
> > > >>>> like /clear on behalf of users instead of running commands against
> > the
> > > >>>> database directly).
> > > >>>>
> > > >>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
> bdbruin@gmail.com
> > >
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Will respond but im traveling at the moment. Give me a few days.
> > > >>>>>
> > > >>>>> Sent from my iPhone
> > > >>>>>
> > > >>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> criccomini@apache.org>
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>> Hey all,
> > > >>>>>>
> > > >>>>>> Checking in on this. We spent a good chunk of time thinking
> about
> > > >>> this,
> > > >>>>> and
> > > >>>>>> want to move forward with it, but want to make sure we're all on
> > the
> > > >>>> same
> > > >>>>>> page.
> > > >>>>>>
> > > >>>>>> Max? Bolke? Dan? Jeremiah?
> > > >>>>>>
> > > >>>>>> Cheers,
> > > >>>>>> Chris
> > > >>>>>>
> > > >>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
> > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>>> Hello everyone,
> > > >>>>>>>
> > > >>>>>>> As you all know, currently Airflow doesn’t have a built-in Role
> > > >>> Based
> > > >>>>>>> Access Control(RBAC) capability.  It does provide very limited
> > > >>>>>>> authorization capability by providing admin, data_profiler, and
> > > user
> > > >>>>> roles.
> > > >>>>>>> However, associating these roles to authenticated identities is
> > not
> > > >>> a
> > > >>>>>>> simple effort.
> > > >>>>>>>
> > > >>>>>>> To address this issue, I have created a design proposal for
> > > building
> > > >>>>> RBAC
> > > >>>>>>> into Airflow and simplifying user access management via the
> > Airflow
> > > >>>> UI.
> > > >>>>>>>
> > > >>>>>>> The design proposal is located at https://cwiki.apache.org/
> > > >>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
> > > >>>>>>>
> > > >>>>>>> Any comments/questions/feedback are much appreciated.
> > > >>>>>>>
> > > >>>>>>> Thanks
> > > >>>>>>> Kalpesh
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >
> > >
> > >
> >
>



-- 

Joy Gao
Software Engineer
350 Convention Way, Suite 200
Redwood City, CA 94063
Mobile:  669-224-9305

Payments partner to the platform economy

Re: Role Based Access Control for Airflow UI

Posted by Joy Gao <jo...@wepay.com>.

Hi all,

We had a good discussion about next steps on implementing RBAC this
morning, here is the current agreed-upon approach:

   1. I will be working on prototyping the FAB-based webserver on a forked
   Apache-Airflow repo. This will reside in the directory
   apache-airflow/airflow/www2/.
   2. I will also create a fork of Flask-AppBuilder to fix any potential
   limitations in current version of FAB
   3. This FAB-based prototype will have its own models.py, views.py, and
   config.py
      1. The prototype will cover all ModelViews + the home page view only
      2. The goal is feature parity:  there will be no redesign of UI
      3. The new models.py will only have the ORM mapping of each existing
      model; while the new views will use the old models' methods to
prevent code
      duplication
   4. Once the prototype is ready I’ll create a PR against Apache-Airflow’s
   master branch. This will be dependent on the FAB-related fixes getting
   merged and released to FAB
   5. Going forward, the remaining views will be done on the master branch,
   this encourages all new features on the views to be compatible with both
   versions of webservers going forward.
   6. Eventually, once we reach complete feature parity on all the views
   and add in the RBAC support, we can deprecate www/ in favor of www2/, and
   point `airflow webserver` to the FAB-based webserver.

Some additional notes:

   - The reason we want to develop the FAB webserver on the master branch
   as soon as possible instead of creating a separate pypi package outside of
   Apache is to avoid the situation where the new fork deviates so much from
   the master branch that it becomes risky to merge it back in
   - One of the current major blockers is compound PKs support in FAB.
   Without it, we’d have to do create new Primary Keys and force a DB
   migration, and that is not ideal so we want to avoid that.
   - FAB currently does not do Business Role (group) to FAB role mapping,
   that’s something we’ll need to discuss later to see how to implement it.


We attempted to record a video of this discussion, but apparently it came
out sounding like a skrillex dubstep mix. So we'll skip that :P

Let me know if you have any questions/concerns regarding this approach.

Regards,
Joy


On Fri, Aug 4, 2017 at 1:30 PM, Chris Riccomini <cr...@apache.org>
wrote:
>
> OK! I booked another meeting for Monday at 10am Pacific. Zoom link is in
> the invite but just in case:
>
> Join from PC, Mac, Linux, iOS or Android: https://wepay.zoom.us/j/
620652517
>
> Or iPhone one-tap (US Toll):  +16465588656 <(646)%20558-8656>
<(646)%20558-8656>,,620652517#
> or +14086380968 <(408)%20638-0968> <(408)%20638-0968>,,620652517#
>
> Or Telephone:
>     Dial: +1 646 558 8656 <(646)%20558-8656> <(646)%20558-8656> (US Toll)
or +1 408 638 0968 <(408)%20638-0968>
> <(408)%20638-0968> (US Toll)
>     +1 855 880 1246 <(855)%20880-1246> <(855)%20880-1246> (US Toll Free)
>     +1 877 369 0926 <(877)%20369-0926> <(877)%20369-0926> (US Toll Free)
>     Meeting ID: 620 652 517
>     International numbers available: https://wepay.zoom.us/
zoomconference?m=
> oRQvXKiEAD4tQlraBOLujkKBPZ_220mB
>
> On Wed, Aug 2, 2017 at 10:46 AM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Hey all,
> >
> > Unfortunately, we're going to have to reschedule this meeting. A few
> > critical folks ended up having to back out at the last minute. I'll
send an
> > update when I've coordinated a new time. Ideally, I'd like to do it
later
> > this week, but we'll see.
> >
> > Cheers,
> > Chris
> >
> > On Tue, Jul 25, 2017 at 3:48 PM, Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> >> Invite sent for 8/2 from 3pm-5pm Pacific time! Here's the dial-in info:
> >>
> >> Join from PC, Mac, Linux, iOS or Android:
https://wepay.zoom.us/j/994482
> >> 435
> >>
> >> Or iPhone one-tap (US Toll):  +16465588656 <(646)%20558-8656>
<(646)%20558-8656>,,994482435#
> >> or +14086380968 <(408)%20638-0968> <(408)%20638-0968>,,994482435#
> >>
> >> Or Telephone:
> >>     Dial: +1 646 558 8656 <(646)%20558-8656> <(646)%20558-8656> (US
Toll) or +1 408 638 0968 <(408)%20638-0968>
> >> <(408)%20638-0968> (US Toll)
> >>     +1 855 880 1246 <(855)%20880-1246> <(855)%20880-1246> (US Toll
Free)
> >>     +1 877 369 0926 <(877)%20369-0926> <(877)%20369-0926> (US Toll
Free)
> >>     Meeting ID: 994 482 435
> >>     International numbers available: https://wepay.zoom.us/zoomconf
> >> erence?m=wHL1YsNLTvoAf8xdE8ki9yNEPVcus_eF
> >>
> >>
> >> On Tue, Jul 25, 2017 at 2:53 PM, Chris Riccomini <criccomini@apache.org
>
> >> wrote:
> >>
> >>> Great! I'll send an invite.
> >>>
> >>> On Tue, Jul 25, 2017 at 1:43 PM, Dan Davydov <
> >>> dan.davydov@airbnb.com.invalid> wrote:
> >>>
> >>>> Same as Alex, would be great to be able to remote in though I'm very
> >>>> interested.
> >>>>
> >>>> On Tue, Jul 25, 2017 at 1:37 PM, Alex Guziel <alex.guziel@airbnb.com
> >>>> .invalid
> >>>> > wrote:
> >>>>
> >>>> > Yeah, I could call in but I probably won't be able to come down
that
> >>>> day.
> >>>> >
> >>>> > On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
> >>>> > maximebeauchemin@gmail.com> wrote:
> >>>> >
> >>>> > > Works for me! Dan said he might confcall in. Alex?
> >>>> > >
> >>>> > > Max
> >>>> > >
> >>>> > > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <
> >>>> criccomini@apache.org
> >>>> > >
> >>>> > > wrote:
> >>>> > >
> >>>> > > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a
room
> >>>> for
> >>>> > 12
> >>>> > > > people (and video conferencing) at WePay in this time slot.
Should
> >>>> > allow
> >>>> > > > you to head home easily afterwards. :) That work for you guys?
> >>>> > > >
> >>>> > > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
> >>>> > > > maximebeauchemin@gmail.com> wrote:
> >>>> > > >
> >>>> > > > > The week of the 31st sound good. Wednesday?
> >>>> > > > >
> >>>> > > > > About React we may not need a frontend lib like it (or at
least
> >>>> not
> >>>> > > just
> >>>> > > > > yet). We can talk about it at the meeting.
> >>>> > > > >
> >>>> > > > > Max
> >>>> > > > >
> >>>> > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <
> >>>> bdbruin@gmail.com>
> >>>> > > > > wrote:
> >>>> > > > >
> >>>> > > > > > We avoid React for the same reasons as the ASF and use
> >>>> Polymer 2
> >>>> > > > instead.
> >>>> > > > > > Would that work?
> >>>> > > > > >
> >>>> > > > > > Bolke.
> >>>> > > > > >
> >>>> > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <
> >>>> criccomini@apache.org
> >>>> > >
> >>>> > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > Hey Max,
> >>>> > > > > > >
> >>>> > > > > > > Want to come down to WePay? We can set up a zoom for
those
> >>>> that
> >>>> > > want
> >>>> > > > to
> >>>> > > > > > > join online, and record it as well to post for the
> >>>> community.
> >>>> > > > > > >
> >>>> > > > > > > Since Joy is just getting started, and it looks like
there's
> >>>> > going
> >>>> > > to
> >>>> > > > > be
> >>>> > > > > > a
> >>>> > > > > > > K8s discussion next week, maybe we can shoot for the week
> >>>> after
> >>>> > > (the
> >>>> > > > > week
> >>>> > > > > > > of the 31st of July)? Care to float a few times that
week?
> >>>> > > > > > >
> >>>> > > > > > > Cheers,
> >>>> > > > > > > Chris
> >>>> > > > > > >
> >>>> > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
> >>>> > > > > > > maximebeauchemin@gmail.com> wrote:
> >>>> > > > > > >
> >>>> > > > > > >> Sounds awesome, count me in!
> >>>> > > > > > >>
> >>>> > > > > > >> * check out the prototype in my fork, I went far enough
to
> >>>> hit
> >>>> > > some
> >>>> > > > > > >> hurdles, try different workarounds. I hooked up the
Airflow
> >>>> > > > Bootstrap
> >>>> > > > > > >> template too so that we feel at home in this new UI
> >>>> > > > > > >> * using a single `id` field is a requirement for FAB
that
> >>>> > airflow
> >>>> > > > > > doesn't
> >>>> > > > > > >> respect (composite pks), either we add the feature to
> >>>> support
> >>>> > that
> >>>> > > > in
> >>>> > > > > > FAB,
> >>>> > > > > > >> or we align on the Airflow side and modify the models
and
> >>>> add a
> >>>> > > > > > migration
> >>>> > > > > > >> script. This upgrade would require downtime and might be
> >>>> > annoying
> >>>> > > to
> >>>> > > > > the
> >>>> > > > > > >> Airflow community, but could help with db performance a
bit
> >>>> > > (smaller
> >>>> > > > > > >> index)... I probably could be convinced either way but
I'm
> >>>> > leaning
> >>>> > > > on
> >>>> > > > > > >> improving FAB
> >>>> > > > > > >> * I'm a maintainer for FAB so I can help get stuff
through
> >>>> there
> >>>> > > > > > >> * React is in limbo at the ASF for licensing reasons,
so no
> >>>> > React
> >>>> > > at
> >>>> > > > > > least
> >>>> > > > > > >> for now
> >>>> > > > > > >> * npm/webpack/ES6, javascript only in `.js` files
> >>>> > > > > > >> * I vote for eslint + eslint-config-airbnb as a set of
> >>>> linting
> >>>> > > rules
> >>>> > > > > > for JS
> >>>> > > > > > >> * Keep out of apache (for now), this new app ships as
its
> >>>> own
> >>>> > pypi
> >>>> > > > > > package
> >>>> > > > > > >> `airflow-webserver`, have a period of overlap
(maintaining
> >>>> 2 web
> >>>> > > > apps)
> >>>> > > > > > >> before ripping out `airflow/www` from the core package
> >>>> > > > > > >> * You need to get in touch with Marty Kausas, an intern
at
> >>>> > Airbnb
> >>>> > > > > who's
> >>>> > > > > > >> been working on a Flask blueprint for improved, more
> >>>> > personalized
> >>>> > > > > views
> >>>> > > > > > on
> >>>> > > > > > >> DAGs that we were planning on merging into the main
branch
> >>>> > > > eventually.
> >>>> > > > > > Some
> >>>> > > > > > >> of Marty's idea and code could be merged into this
effort.
> >>>> > > > > > >>
> >>>> > > > > > >> These are ideas on how I would proceed personally on
this
> >>>> but
> >>>> > > > > definitely
> >>>> > > > > > >> everything here is up for discussion.
> >>>> > > > > > >>
> >>>> > > > > > >> Let's meet physically at either WePay or Airbnb. Folks
> >>>> from the
> >>>> > > > > > community,
> >>>> > > > > > >> let us know on this thread if you want to be part of
this
> >>>> > effort,
> >>>> > > > > we'll
> >>>> > > > > > be
> >>>> > > > > > >> happy to include you.
> >>>> > > > > > >>
> >>>> > > > > > >> Thanks,
> >>>> > > > > > >>
> >>>> > > > > > >> Max
> >>>> > > > > > >>
> >>>> > > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <joyg@wepay.com
>
> >>>> > wrote:
> >>>> > > > > > >>
> >>>> > > > > > >>> Hey everyone,
> >>>> > > > > > >>>
> >>>> > > > > > >>> I recently transferred to Data Infra team here at
WePay to
> >>>> > focus
> >>>> > > on
> >>>> > > > > > >>> Airflow-related initiatives.
> >>>> > > > > > >>>
> >>>> > > > > > >>> Given the RBAC design is mostly hashed out, I'm happy
to
> >>>> get
> >>>> > this
> >>>> > > > > > feature
> >>>> > > > > > >>> off the ground for Q3, starting with converting Airflow
> >>>> to Fab,
> >>>> > > if
> >>>> > > > > > there
> >>>> > > > > > >>> are no objections.
> >>>> > > > > > >>>
> >>>> > > > > > >>> Cheers,
> >>>> > > > > > >>> Joy
> >>>> > > > > > >>>
> >>>> > > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> >>>> > > > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
> >>>> > > > > > >>>
> >>>> > > > > > >>>> Hey all,
> >>>> > > > > > >>>>
> >>>> > > > > > >>>> We talked about this internally. We would like to work
> >>>> on this
> >>>> > > > > feature
> >>>> > > > > > >>> but
> >>>> > > > > > >>>> given the immediate priorities we are not going to be
> >>>> working
> >>>> > on
> >>>> > > > it
> >>>> > > > > in
> >>>> > > > > > >>> Q3.
> >>>> > > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is
> >>>> we can
> >>>> > > work
> >>>> > > > > on
> >>>> > > > > > >> it
> >>>> > > > > > >>>> late Q4 or Q12018.
> >>>> > > > > > >>>>
> >>>> > > > > > >>>> Cheers,
> >>>> > > > > > >>>>
> >>>> > > > > > >>>> Gurer
> >>>> > > > > > >>>>
> >>>> > > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
> >>>> > > > > > >> criccomini@apache.org>
> >>>> > > > > > >>>> wrote:
> >>>> > > > > > >>>>
> >>>> > > > > > >>>>> I think FAB sounds like the right approach. Waiting
to
> >>>> hear
> >>>> > > back
> >>>> > > > > with
> >>>> > > > > > >>>> notes
> >>>> > > > > > >>>>> on AirBNB H2 discussion to see if they want to take
> >>>> this up.
> >>>> > > > > > >>>>>
> >>>> > > > > > >>>>> @Gurer, any idea when this will happen?
> >>>> > > > > > >>>>>
> >>>> > > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
> >>>> > > > bdbruin@gmail.com
> >>>> > > > > >
> >>>> > > > > > >>>> wrote:
> >>>> > > > > > >>>>>
> >>>> > > > > > >>>>>> One downside I see from FAB is that is does not do
> >>>> Business
> >>>> > > Role
> >>>> > > > > > >>>> mapping
> >>>> > > > > > >>>>>> to FAB role. I would prefer to create groups in
> >>>> IPA/LDAP/AD
> >>>> > > and
> >>>> > > > > > >> have
> >>>> > > > > > >>>>> those
> >>>> > > > > > >>>>>> map to FAB roles instead of needing to manage that
in
> >>>> FAB.
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>> B.
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <
> >>>> > bdbruin@gmail.com>
> >>>> > > > > > >>> wrote:
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> Hi Guys,
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> Thanks for putting the thinking in! It is about
time
> >>>> that
> >>>> > we
> >>>> > > > get
> >>>> > > > > > >>> this
> >>>> > > > > > >>>>>> moving.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> The design looks pretty sound. One can argue about
the
> >>>> > > > different
> >>>> > > > > > >>>> roles
> >>>> > > > > > >>>>>> that are required, but that will be situation
> >>>> dependent I
> >>>> > > guess.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> Implementation wise I would argue together with Max
> >>>> that
> >>>> > FAB
> >>>> > > > is a
> >>>> > > > > > >>>>> better
> >>>> > > > > > >>>>>> or best fit. The ER model that is being described is
> >>>> pretty
> >>>> > > > much a
> >>>> > > > > > >>> copy
> >>>> > > > > > >>>>> of
> >>>> > > > > > >>>>>> a normal security model. So a reimplementation of
that
> >>>> is 1)
> >>>> > > > > > >>>> significant
> >>>> > > > > > >>>>>> duplication of effort and 2) bound to have bugs that
> >>>> have
> >>>> > been
> >>>> > > > > > >> solved
> >>>> > > > > > >>>> in
> >>>> > > > > > >>>>>> the other framework. Moreover, FAB does have
> >>>> integration out
> >>>> > > of
> >>>> > > > > the
> >>>> > > > > > >>> box
> >>>> > > > > > >>>>>> with some enterprisey systems like IPA,
> >>>> ActiveDirectory, and
> >>>> > > > LDAP.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> So while you argue that using FAB would increase
the
> >>>> scope
> >>>> > of
> >>>> > > > the
> >>>> > > > > > >>>>>> proposal significantly, but I think that is not
true.
> >>>> Using
> >>>> > > FAB
> >>>> > > > > > >> would
> >>>> > > > > > >>>>> allow
> >>>> > > > > > >>>>>> you to focus on what kind of out-of-the-box
permission
> >>>> sets
> >>>> > > and
> >>>> > > > > > >> roles
> >>>> > > > > > >>>> we
> >>>> > > > > > >>>>>> would need and maybe address some issues that FAB
lacks
> >>>> > (maybe
> >>>> > > > how
> >>>> > > > > > >> to
> >>>> > > > > > >>>>> deal
> >>>> > > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos,
> >>>> probably
> >>>> > > how
> >>>> > > > to
> >>>> > > > > > >>> deal
> >>>> > > > > > >>>>>> with API calls that are not CRUD). Implementation
wise
> >>>> it
> >>>> > > > probably
> >>>> > > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s
> >>>> early POC
> >>>> > > as
> >>>> > > > an
> >>>> > > > > > >>>>> example
> >>>> > > > > > >>>>>> - we can slowly move over?
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to
work on
> >>>> > > Airflow
> >>>> > > > > > >>> coming
> >>>> > > > > > >>>>>> year. Improvement of Security, Enterprise
Integration,
> >>>> > Revamp
> >>>> > > UI
> >>>> > > > > > >> are
> >>>> > > > > > >>> on
> >>>> > > > > > >>>>> the
> >>>> > > > > > >>>>>> todo list. However, this is not confirmed yet as
> >>>> business
> >>>> > > > > > >> priorities
> >>>> > > > > > >>>>> might
> >>>> > > > > > >>>>>> change.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> Bolke.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> >>>> > > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> @Dan:
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> Thanks for your feedback. I will remove the
> >>>> REFRESH_DAG
> >>>> > > > > > >>> permission.
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> @Max:
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> Thanks for your response.
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> The scope of my proposal was just to add RBAC
> >>>> security
> >>>> > > feature
> >>>> > > > > > >> to
> >>>> > > > > > >>>>>> Airflow
> >>>> > > > > > >>>>>>>> without replacing any existing frameworks.
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> I understand that adopting FAB would serve Airflow
> >>>> better
> >>>> > > > moving
> >>>> > > > > > >>>>>> forward,
> >>>> > > > > > >>>>>>>> however porting Airflow to using FAB significantly
> >>>> > increases
> >>>> > > > the
> >>>> > > > > > >>>> scope
> >>>> > > > > > >>>>>> of
> >>>> > > > > > >>>>>>>> the proposal and I don't have the time and
expertise
> >>>> to
> >>>> > > carry
> >>>> > > > > > >> out
> >>>> > > > > > >>>> the
> >>>> > > > > > >>>>>> tasks
> >>>> > > > > > >>>>>>>> in the extended scope.
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for
> >>>> Airflow
> >>>> > to
> >>>> > > > > > >>> migrate
> >>>> > > > > > >>>> to
> >>>> > > > > > >>>>>> FAB
> >>>> > > > > > >>>>>>>> this year?
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> - Kalpesh
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime
Beauchemin <
> >>>> > > > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>>> It would be nice to go with a framework for
this. I
> >>>> did
> >>>> > > some
> >>>> > > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in
this
> >>>> > > > direction.
> >>>> > > > > > >> It
> >>>> > > > > > >>>>>> provides
> >>>> > > > > > >>>>>>>>> auth on different authentication backends out of
> >>>> the box
> >>>> > > > > > >> (oauth,
> >>>> > > > > > >>>>>> openid,
> >>>> > > > > > >>>>>>>>> ldap, registration, ...), generates perms for
each
> >>>> view
> >>>> > > that
> >>>> > > > > > >> has
> >>>> > > > > > >>> an
> >>>> > > > > > >>>>>>>>> @has_access decorator, generates at set of perms
> >>>> for each
> >>>> > > ORM
> >>>> > > > > > >>> model
> >>>> > > > > > >>>>>> (show,
> >>>> > > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the
CRUD
> >>>> views
> >>>> > > as
> >>>> > > > > > >> well
> >>>> > > > > > >>>> as
> >>>> > > > > > >>>>>> in the
> >>>> > > > > > >>>>>>>>> generated REST api that you get for free as a
> >>>> byprdoduct
> >>>> > of
> >>>> > > > > > >>>> deriving
> >>>> > > > > > >>>>>> FAB's
> >>>> > > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer
on
> >>>> top).
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> I started a POC on FAB here a while ago:
> >>>> > > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver
> >>>> at the
> >>>> > > > time
> >>>> > > > > > >> my
> >>>> > > > > > >>>>> main
> >>>> > > > > > >>>>>>>>> motivation was the free/instantaneous REST api.
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> I think FAB is a decent fit as the porting
should be
> >>>> > fairly
> >>>> > > > > > >>>>>> straightforward
> >>>> > > > > > >>>>>>>>> (moving the flask views over and deprecating
> >>>> Flask-Admin
> >>>> > in
> >>>> > > > > > >> favor
> >>>> > > > > > >>>> of
> >>>> > > > > > >>>>>> FAB's
> >>>> > > > > > >>>>>>>>> crud) though there was a few blockers. From
memory I
> >>>> > think
> >>>> > > > FAB
> >>>> > > > > > >>>> didn't
> >>>> > > > > > >>>>>> like
> >>>> > > > > > >>>>>>>>> the compound PKs we use in some of the Airflow
> >>>> models.
> >>>> > We'd
> >>>> > > > > > >> have
> >>>> > > > > > >>> to
> >>>> > > > > > >>>>>> either
> >>>> > > > > > >>>>>>>>> write a db migration script on the Airflow side,
or
> >>>> add
> >>>> > > > support
> >>>> > > > > > >>> for
> >>>> > > > > > >>>>>>>>> compound keys to FAB (I recently became a
> >>>> maintainer of
> >>>> > the
> >>>> > > > > > >>>> project,
> >>>> > > > > > >>>>>> so I
> >>>> > > > > > >>>>>>>>> could help with that)
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> The only downside of FAB is that it's not as
mature
> >>>> as
> >>>> > > > > > >> something
> >>>> > > > > > >>>> like
> >>>> > > > > > >>>>>>>>> Django, but porting to Django would surely be
much
> >>>> more
> >>>> > > work.
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> Then there's the flask-security suite, but that
> >>>> looks
> >>>> > like
> >>>> > > a
> >>>> > > > > > >> bit
> >>>> > > > > > >>>> of a
> >>>> > > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose
> >>>> which we
> >>>> > > want
> >>>> > > > > > >> to
> >>>> > > > > > >>>> use.
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> Max
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> >>>> > > > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>>> Looks good to me in general, thanks for putting
> >>>> this
> >>>> > > > together!
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>> I think the ability to integrate with external
RBAC
> >>>> > > systems
> >>>> > > > > > >> like
> >>>> > > > > > >>>>> LDAP
> >>>> > > > > > >>>>>> is
> >>>> > > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be
> >>>> decoupled
> >>>> > > with
> >>>> > > > > > >> the
> >>>> > > > > > >>>> RBAC
> >>>> > > > > > >>>>>>>>>> database wherever possible).
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>> I wouldn't be too worried about the permissions
> >>>> about
> >>>> > > > > > >> refreshing
> >>>> > > > > > >>>>>> DAGs, as
> >>>> > > > > > >>>>>>>>>> far as I know this functionality is no longer
> >>>> required
> >>>> > > with
> >>>> > > > > > >> the
> >>>> > > > > > >>>> new
> >>>> > > > > > >>>>>>>>>> webservers which reload state periodically, and
> >>>> will
> >>>> > > > certainly
> >>>> > > > > > >>> be
> >>>> > > > > > >>>>>> removed
> >>>> > > > > > >>>>>>>>>> when we have a better DAG consistency story.
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>> I think it would also be good to think about
this
> >>>> > > > > > >>>>>> proposal/implementation
> >>>> > > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g.
> >>>> when
> >>>> > > > > > >> webserver
> >>>> > > > > > >>>> hits
> >>>> > > > > > >>>>>> APIs
> >>>> > > > > > >>>>>>>>>> like /clear on behalf of users instead of
running
> >>>> > commands
> >>>> > > > > > >>> against
> >>>> > > > > > >>>>> the
> >>>> > > > > > >>>>>>>>>> database directly).
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de
Bruin <
> >>>> > > > > > >>>> bdbruin@gmail.com
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>>>>>> wrote:
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>> Will respond but im traveling at the moment.
Give
> >>>> me a
> >>>> > > few
> >>>> > > > > > >>> days.
> >>>> > > > > > >>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>> Sent from my iPhone
> >>>> > > > > > >>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> >>>> > > > > > >>>> criccomini@apache.org>
> >>>> > > > > > >>>>>>>>>> wrote:
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> Hey all,
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of
> >>>> time
> >>>> > > > thinking
> >>>> > > > > > >>>> about
> >>>> > > > > > >>>>>>>>> this,
> >>>> > > > > > >>>>>>>>>>> and
> >>>> > > > > > >>>>>>>>>>>> want to move forward with it, but want to make
> >>>> sure
> >>>> > > we're
> >>>> > > > > > >> all
> >>>> > > > > > >>> on
> >>>> > > > > > >>>>> the
> >>>> > > > > > >>>>>>>>>> same
> >>>> > > > > > >>>>>>>>>>>> page.
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> Cheers,
> >>>> > > > > > >>>>>>>>>>>> Chris
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh
> >>>> dharwadkar <
> >>>> > > > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> Hello everyone,
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t
have
> >>>> a
> >>>> > > > built-in
> >>>> > > > > > >>> Role
> >>>> > > > > > >>>>>>>>> Based
> >>>> > > > > > >>>>>>>>>>>>> Access Control(RBAC) capability.  It does
> >>>> provide
> >>>> > very
> >>>> > > > > > >>> limited
> >>>> > > > > > >>>>>>>>>>>>> authorization capability by providing admin,
> >>>> > > > data_profiler,
> >>>> > > > > > >>> and
> >>>> > > > > > >>>>>> user
> >>>> > > > > > >>>>>>>>>>> roles.
> >>>> > > > > > >>>>>>>>>>>>> However, associating these roles to
> >>>> authenticated
> >>>> > > > > > >> identities
> >>>> > > > > > >>> is
> >>>> > > > > > >>>>> not
> >>>> > > > > > >>>>>>>>> a
> >>>> > > > > > >>>>>>>>>>>>> simple effort.
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> To address this issue, I have created a
design
> >>>> > proposal
> >>>> > > > for
> >>>> > > > > > >>>>>> building
> >>>> > > > > > >>>>>>>>>>> RBAC
> >>>> > > > > > >>>>>>>>>>>>> into Airflow and simplifying user access
> >>>> management
> >>>> > via
> >>>> > > > the
> >>>> > > > > > >>>>> Airflow
> >>>> > > > > > >>>>>>>>>> UI.
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> The design proposal is located at
> >>>> > > > > > >> https://cwiki.apache.org/
> >>>> > > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Air
> >>>> flow+RBAC+proposal
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much
> >>>> appreciated.
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> Thanks
> >>>> > > > > > >>>>>>>>>>>>> Kalpesh
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>
> >>>> > > > > > >>>>
> >>>> > > > > > >>>
> >>>> > > > > > >>>
> >>>> > > > > > >>>
> >>>> > > > > > >>> --
> >>>> > > > > > >>>
> >>>> > > > > > >>> Joy Gao
> >>>> > > > > > >>> Software Engineer
> >>>> > > > > > >>> 350 Convention Way, Suite 200
> >>>> > > > > > >>> Redwood City, CA 94063
> >>>> > > > > > >>> Mobile:  669-224-9305 <(669)%20224-9305>
> >>>> > > > > > >>>
> >>>> > > > > > >>> Payments partner to the platform economy
> >>>> > > > > > >>>
> >>>> > > > > > >>
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >

Re: Role Based Access Control for Airflow UI

Posted by Chris Riccomini <cr...@apache.org>.

OK! I booked another meeting for Monday at 10am Pacific. Zoom link is in
the invite but just in case:

Join from PC, Mac, Linux, iOS or Android: https://wepay.zoom.us/j/620652517

Or iPhone one-tap (US Toll):  +16465588656 <(646)%20558-8656>,,620652517#
or +14086380968 <(408)%20638-0968>,,620652517#

Or Telephone:
    Dial: +1 646 558 8656 <(646)%20558-8656> (US Toll) or +1 408 638 0968
<(408)%20638-0968> (US Toll)
    +1 855 880 1246 <(855)%20880-1246> (US Toll Free)
    +1 877 369 0926 <(877)%20369-0926> (US Toll Free)
    Meeting ID: 620 652 517
    International numbers available: https://wepay.zoom.us/zoomconference?m=
oRQvXKiEAD4tQlraBOLujkKBPZ_220mB

On Wed, Aug 2, 2017 at 10:46 AM, Chris Riccomini <cr...@apache.org>
wrote:

> Hey all,
>
> Unfortunately, we're going to have to reschedule this meeting. A few
> critical folks ended up having to back out at the last minute. I'll send an
> update when I've coordinated a new time. Ideally, I'd like to do it later
> this week, but we'll see.
>
> Cheers,
> Chris
>
> On Tue, Jul 25, 2017 at 3:48 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
>> Invite sent for 8/2 from 3pm-5pm Pacific time! Here's the dial-in info:
>>
>> Join from PC, Mac, Linux, iOS or Android: https://wepay.zoom.us/j/994482
>> 435
>>
>> Or iPhone one-tap (US Toll):  +16465588656 <(646)%20558-8656>,,994482435#
>> or +14086380968 <(408)%20638-0968>,,994482435#
>>
>> Or Telephone:
>>     Dial: +1 646 558 8656 <(646)%20558-8656> (US Toll) or +1 408 638 0968
>> <(408)%20638-0968> (US Toll)
>>     +1 855 880 1246 <(855)%20880-1246> (US Toll Free)
>>     +1 877 369 0926 <(877)%20369-0926> (US Toll Free)
>>     Meeting ID: 994 482 435
>>     International numbers available: https://wepay.zoom.us/zoomconf
>> erence?m=wHL1YsNLTvoAf8xdE8ki9yNEPVcus_eF
>>
>>
>> On Tue, Jul 25, 2017 at 2:53 PM, Chris Riccomini <cr...@apache.org>
>> wrote:
>>
>>> Great! I'll send an invite.
>>>
>>> On Tue, Jul 25, 2017 at 1:43 PM, Dan Davydov <
>>> dan.davydov@airbnb.com.invalid> wrote:
>>>
>>>> Same as Alex, would be great to be able to remote in though I'm very
>>>> interested.
>>>>
>>>> On Tue, Jul 25, 2017 at 1:37 PM, Alex Guziel <alex.guziel@airbnb.com
>>>> .invalid
>>>> > wrote:
>>>>
>>>> > Yeah, I could call in but I probably won't be able to come down that
>>>> day.
>>>> >
>>>> > On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
>>>> > maximebeauchemin@gmail.com> wrote:
>>>> >
>>>> > > Works for me! Dan said he might confcall in. Alex?
>>>> > >
>>>> > > Max
>>>> > >
>>>> > > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <
>>>> criccomini@apache.org
>>>> > >
>>>> > > wrote:
>>>> > >
>>>> > > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room
>>>> for
>>>> > 12
>>>> > > > people (and video conferencing) at WePay in this time slot. Should
>>>> > allow
>>>> > > > you to head home easily afterwards. :) That work for you guys?
>>>> > > >
>>>> > > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
>>>> > > > maximebeauchemin@gmail.com> wrote:
>>>> > > >
>>>> > > > > The week of the 31st sound good. Wednesday?
>>>> > > > >
>>>> > > > > About React we may not need a frontend lib like it (or at least
>>>> not
>>>> > > just
>>>> > > > > yet). We can talk about it at the meeting.
>>>> > > > >
>>>> > > > > Max
>>>> > > > >
>>>> > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <
>>>> bdbruin@gmail.com>
>>>> > > > > wrote:
>>>> > > > >
>>>> > > > > > We avoid React for the same reasons as the ASF and use
>>>> Polymer 2
>>>> > > > instead.
>>>> > > > > > Would that work?
>>>> > > > > >
>>>> > > > > > Bolke.
>>>> > > > > >
>>>> > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <
>>>> criccomini@apache.org
>>>> > >
>>>> > > > > wrote:
>>>> > > > > > >
>>>> > > > > > > Hey Max,
>>>> > > > > > >
>>>> > > > > > > Want to come down to WePay? We can set up a zoom for those
>>>> that
>>>> > > want
>>>> > > > to
>>>> > > > > > > join online, and record it as well to post for the
>>>> community.
>>>> > > > > > >
>>>> > > > > > > Since Joy is just getting started, and it looks like there's
>>>> > going
>>>> > > to
>>>> > > > > be
>>>> > > > > > a
>>>> > > > > > > K8s discussion next week, maybe we can shoot for the week
>>>> after
>>>> > > (the
>>>> > > > > week
>>>> > > > > > > of the 31st of July)? Care to float a few times that week?
>>>> > > > > > >
>>>> > > > > > > Cheers,
>>>> > > > > > > Chris
>>>> > > > > > >
>>>> > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
>>>> > > > > > > maximebeauchemin@gmail.com> wrote:
>>>> > > > > > >
>>>> > > > > > >> Sounds awesome, count me in!
>>>> > > > > > >>
>>>> > > > > > >> * check out the prototype in my fork, I went far enough to
>>>> hit
>>>> > > some
>>>> > > > > > >> hurdles, try different workarounds. I hooked up the Airflow
>>>> > > > Bootstrap
>>>> > > > > > >> template too so that we feel at home in this new UI
>>>> > > > > > >> * using a single `id` field is a requirement for FAB that
>>>> > airflow
>>>> > > > > > doesn't
>>>> > > > > > >> respect (composite pks), either we add the feature to
>>>> support
>>>> > that
>>>> > > > in
>>>> > > > > > FAB,
>>>> > > > > > >> or we align on the Airflow side and modify the models and
>>>> add a
>>>> > > > > > migration
>>>> > > > > > >> script. This upgrade would require downtime and might be
>>>> > annoying
>>>> > > to
>>>> > > > > the
>>>> > > > > > >> Airflow community, but could help with db performance a bit
>>>> > > (smaller
>>>> > > > > > >> index)... I probably could be convinced either way but I'm
>>>> > leaning
>>>> > > > on
>>>> > > > > > >> improving FAB
>>>> > > > > > >> * I'm a maintainer for FAB so I can help get stuff through
>>>> there
>>>> > > > > > >> * React is in limbo at the ASF for licensing reasons, so no
>>>> > React
>>>> > > at
>>>> > > > > > least
>>>> > > > > > >> for now
>>>> > > > > > >> * npm/webpack/ES6, javascript only in `.js` files
>>>> > > > > > >> * I vote for eslint + eslint-config-airbnb as a set of
>>>> linting
>>>> > > rules
>>>> > > > > > for JS
>>>> > > > > > >> * Keep out of apache (for now), this new app ships as its
>>>> own
>>>> > pypi
>>>> > > > > > package
>>>> > > > > > >> `airflow-webserver`, have a period of overlap (maintaining
>>>> 2 web
>>>> > > > apps)
>>>> > > > > > >> before ripping out `airflow/www` from the core package
>>>> > > > > > >> * You need to get in touch with Marty Kausas, an intern at
>>>> > Airbnb
>>>> > > > > who's
>>>> > > > > > >> been working on a Flask blueprint for improved, more
>>>> > personalized
>>>> > > > > views
>>>> > > > > > on
>>>> > > > > > >> DAGs that we were planning on merging into the main branch
>>>> > > > eventually.
>>>> > > > > > Some
>>>> > > > > > >> of Marty's idea and code could be merged into this effort.
>>>> > > > > > >>
>>>> > > > > > >> These are ideas on how I would proceed personally on this
>>>> but
>>>> > > > > definitely
>>>> > > > > > >> everything here is up for discussion.
>>>> > > > > > >>
>>>> > > > > > >> Let's meet physically at either WePay or Airbnb. Folks
>>>> from the
>>>> > > > > > community,
>>>> > > > > > >> let us know on this thread if you want to be part of this
>>>> > effort,
>>>> > > > > we'll
>>>> > > > > > be
>>>> > > > > > >> happy to include you.
>>>> > > > > > >>
>>>> > > > > > >> Thanks,
>>>> > > > > > >>
>>>> > > > > > >> Max
>>>> > > > > > >>
>>>> > > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com>
>>>> > wrote:
>>>> > > > > > >>
>>>> > > > > > >>> Hey everyone,
>>>> > > > > > >>>
>>>> > > > > > >>> I recently transferred to Data Infra team here at WePay to
>>>> > focus
>>>> > > on
>>>> > > > > > >>> Airflow-related initiatives.
>>>> > > > > > >>>
>>>> > > > > > >>> Given the RBAC design is mostly hashed out, I'm happy to
>>>> get
>>>> > this
>>>> > > > > > feature
>>>> > > > > > >>> off the ground for Q3, starting with converting Airflow
>>>> to Fab,
>>>> > > if
>>>> > > > > > there
>>>> > > > > > >>> are no objections.
>>>> > > > > > >>>
>>>> > > > > > >>> Cheers,
>>>> > > > > > >>> Joy
>>>> > > > > > >>>
>>>> > > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
>>>> > > > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
>>>> > > > > > >>>
>>>> > > > > > >>>> Hey all,
>>>> > > > > > >>>>
>>>> > > > > > >>>> We talked about this internally. We would like to work
>>>> on this
>>>> > > > > feature
>>>> > > > > > >>> but
>>>> > > > > > >>>> given the immediate priorities we are not going to be
>>>> working
>>>> > on
>>>> > > > it
>>>> > > > > in
>>>> > > > > > >>> Q3.
>>>> > > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is
>>>> we can
>>>> > > work
>>>> > > > > on
>>>> > > > > > >> it
>>>> > > > > > >>>> late Q4 or Q12018.
>>>> > > > > > >>>>
>>>> > > > > > >>>> Cheers,
>>>> > > > > > >>>>
>>>> > > > > > >>>> Gurer
>>>> > > > > > >>>>
>>>> > > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
>>>> > > > > > >> criccomini@apache.org>
>>>> > > > > > >>>> wrote:
>>>> > > > > > >>>>
>>>> > > > > > >>>>> I think FAB sounds like the right approach. Waiting to
>>>> hear
>>>> > > back
>>>> > > > > with
>>>> > > > > > >>>> notes
>>>> > > > > > >>>>> on AirBNB H2 discussion to see if they want to take
>>>> this up.
>>>> > > > > > >>>>>
>>>> > > > > > >>>>> @Gurer, any idea when this will happen?
>>>> > > > > > >>>>>
>>>> > > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
>>>> > > > bdbruin@gmail.com
>>>> > > > > >
>>>> > > > > > >>>> wrote:
>>>> > > > > > >>>>>
>>>> > > > > > >>>>>> One downside I see from FAB is that is does not do
>>>> Business
>>>> > > Role
>>>> > > > > > >>>> mapping
>>>> > > > > > >>>>>> to FAB role. I would prefer to create groups in
>>>> IPA/LDAP/AD
>>>> > > and
>>>> > > > > > >> have
>>>> > > > > > >>>>> those
>>>> > > > > > >>>>>> map to FAB roles instead of needing to manage that in
>>>> FAB.
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>> B.
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <
>>>> > bdbruin@gmail.com>
>>>> > > > > > >>> wrote:
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> Hi Guys,
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> Thanks for putting the thinking in! It is about time
>>>> that
>>>> > we
>>>> > > > get
>>>> > > > > > >>> this
>>>> > > > > > >>>>>> moving.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> The design looks pretty sound. One can argue about the
>>>> > > > different
>>>> > > > > > >>>> roles
>>>> > > > > > >>>>>> that are required, but that will be situation
>>>> dependent I
>>>> > > guess.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> Implementation wise I would argue together with Max
>>>> that
>>>> > FAB
>>>> > > > is a
>>>> > > > > > >>>>> better
>>>> > > > > > >>>>>> or best fit. The ER model that is being described is
>>>> pretty
>>>> > > > much a
>>>> > > > > > >>> copy
>>>> > > > > > >>>>> of
>>>> > > > > > >>>>>> a normal security model. So a reimplementation of that
>>>> is 1)
>>>> > > > > > >>>> significant
>>>> > > > > > >>>>>> duplication of effort and 2) bound to have bugs that
>>>> have
>>>> > been
>>>> > > > > > >> solved
>>>> > > > > > >>>> in
>>>> > > > > > >>>>>> the other framework. Moreover, FAB does have
>>>> integration out
>>>> > > of
>>>> > > > > the
>>>> > > > > > >>> box
>>>> > > > > > >>>>>> with some enterprisey systems like IPA,
>>>> ActiveDirectory, and
>>>> > > > LDAP.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> So while you argue that using FAB would increase the
>>>> scope
>>>> > of
>>>> > > > the
>>>> > > > > > >>>>>> proposal significantly, but I think that is not true.
>>>> Using
>>>> > > FAB
>>>> > > > > > >> would
>>>> > > > > > >>>>> allow
>>>> > > > > > >>>>>> you to focus on what kind of out-of-the-box permission
>>>> sets
>>>> > > and
>>>> > > > > > >> roles
>>>> > > > > > >>>> we
>>>> > > > > > >>>>>> would need and maybe address some issues that FAB lacks
>>>> > (maybe
>>>> > > > how
>>>> > > > > > >> to
>>>> > > > > > >>>>> deal
>>>> > > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos,
>>>> probably
>>>> > > how
>>>> > > > to
>>>> > > > > > >>> deal
>>>> > > > > > >>>>>> with API calls that are not CRUD). Implementation wise
>>>> it
>>>> > > > probably
>>>> > > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s
>>>> early POC
>>>> > > as
>>>> > > > an
>>>> > > > > > >>>>> example
>>>> > > > > > >>>>>> - we can slowly move over?
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on
>>>> > > Airflow
>>>> > > > > > >>> coming
>>>> > > > > > >>>>>> year. Improvement of Security, Enterprise Integration,
>>>> > Revamp
>>>> > > UI
>>>> > > > > > >> are
>>>> > > > > > >>> on
>>>> > > > > > >>>>> the
>>>> > > > > > >>>>>> todo list. However, this is not confirmed yet as
>>>> business
>>>> > > > > > >> priorities
>>>> > > > > > >>>>> might
>>>> > > > > > >>>>>> change.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> Bolke.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
>>>> > > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> @Dan:
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> Thanks for your feedback. I will remove the
>>>> REFRESH_DAG
>>>> > > > > > >>> permission.
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> @Max:
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> Thanks for your response.
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> The scope of my proposal was just to add RBAC
>>>> security
>>>> > > feature
>>>> > > > > > >> to
>>>> > > > > > >>>>>> Airflow
>>>> > > > > > >>>>>>>> without replacing any existing frameworks.
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> I understand that adopting FAB would serve Airflow
>>>> better
>>>> > > > moving
>>>> > > > > > >>>>>> forward,
>>>> > > > > > >>>>>>>> however porting Airflow to using FAB significantly
>>>> > increases
>>>> > > > the
>>>> > > > > > >>>> scope
>>>> > > > > > >>>>>> of
>>>> > > > > > >>>>>>>> the proposal and I don't have the time and expertise
>>>> to
>>>> > > carry
>>>> > > > > > >> out
>>>> > > > > > >>>> the
>>>> > > > > > >>>>>> tasks
>>>> > > > > > >>>>>>>> in the extended scope.
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for
>>>> Airflow
>>>> > to
>>>> > > > > > >>> migrate
>>>> > > > > > >>>> to
>>>> > > > > > >>>>>> FAB
>>>> > > > > > >>>>>>>> this year?
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> - Kalpesh
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
>>>> > > > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>>> It would be nice to go with a framework for this. I
>>>> did
>>>> > > some
>>>> > > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this
>>>> > > > direction.
>>>> > > > > > >> It
>>>> > > > > > >>>>>> provides
>>>> > > > > > >>>>>>>>> auth on different authentication backends out of
>>>> the box
>>>> > > > > > >> (oauth,
>>>> > > > > > >>>>>> openid,
>>>> > > > > > >>>>>>>>> ldap, registration, ...), generates perms for each
>>>> view
>>>> > > that
>>>> > > > > > >> has
>>>> > > > > > >>> an
>>>> > > > > > >>>>>>>>> @has_access decorator, generates at set of perms
>>>> for each
>>>> > > ORM
>>>> > > > > > >>> model
>>>> > > > > > >>>>>> (show,
>>>> > > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD
>>>> views
>>>> > > as
>>>> > > > > > >> well
>>>> > > > > > >>>> as
>>>> > > > > > >>>>>> in the
>>>> > > > > > >>>>>>>>> generated REST api that you get for free as a
>>>> byprdoduct
>>>> > of
>>>> > > > > > >>>> deriving
>>>> > > > > > >>>>>> FAB's
>>>> > > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on
>>>> top).
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> I started a POC on FAB here a while ago:
>>>> > > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver
>>>> at the
>>>> > > > time
>>>> > > > > > >> my
>>>> > > > > > >>>>> main
>>>> > > > > > >>>>>>>>> motivation was the free/instantaneous REST api.
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> I think FAB is a decent fit as the porting should be
>>>> > fairly
>>>> > > > > > >>>>>> straightforward
>>>> > > > > > >>>>>>>>> (moving the flask views over and deprecating
>>>> Flask-Admin
>>>> > in
>>>> > > > > > >> favor
>>>> > > > > > >>>> of
>>>> > > > > > >>>>>> FAB's
>>>> > > > > > >>>>>>>>> crud) though there was a few blockers. From memory I
>>>> > think
>>>> > > > FAB
>>>> > > > > > >>>> didn't
>>>> > > > > > >>>>>> like
>>>> > > > > > >>>>>>>>> the compound PKs we use in some of the Airflow
>>>> models.
>>>> > We'd
>>>> > > > > > >> have
>>>> > > > > > >>> to
>>>> > > > > > >>>>>> either
>>>> > > > > > >>>>>>>>> write a db migration script on the Airflow side, or
>>>> add
>>>> > > > support
>>>> > > > > > >>> for
>>>> > > > > > >>>>>>>>> compound keys to FAB (I recently became a
>>>> maintainer of
>>>> > the
>>>> > > > > > >>>> project,
>>>> > > > > > >>>>>> so I
>>>> > > > > > >>>>>>>>> could help with that)
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> The only downside of FAB is that it's not as mature
>>>> as
>>>> > > > > > >> something
>>>> > > > > > >>>> like
>>>> > > > > > >>>>>>>>> Django, but porting to Django would surely be much
>>>> more
>>>> > > work.
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> Then there's the flask-security suite, but that
>>>> looks
>>>> > like
>>>> > > a
>>>> > > > > > >> bit
>>>> > > > > > >>>> of a
>>>> > > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose
>>>> which we
>>>> > > want
>>>> > > > > > >> to
>>>> > > > > > >>>> use.
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> Max
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
>>>> > > > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>>> Looks good to me in general, thanks for putting
>>>> this
>>>> > > > together!
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>> I think the ability to integrate with external RBAC
>>>> > > systems
>>>> > > > > > >> like
>>>> > > > > > >>>>> LDAP
>>>> > > > > > >>>>>> is
>>>> > > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be
>>>> decoupled
>>>> > > with
>>>> > > > > > >> the
>>>> > > > > > >>>> RBAC
>>>> > > > > > >>>>>>>>>> database wherever possible).
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>> I wouldn't be too worried about the permissions
>>>> about
>>>> > > > > > >> refreshing
>>>> > > > > > >>>>>> DAGs, as
>>>> > > > > > >>>>>>>>>> far as I know this functionality is no longer
>>>> required
>>>> > > with
>>>> > > > > > >> the
>>>> > > > > > >>>> new
>>>> > > > > > >>>>>>>>>> webservers which reload state periodically, and
>>>> will
>>>> > > > certainly
>>>> > > > > > >>> be
>>>> > > > > > >>>>>> removed
>>>> > > > > > >>>>>>>>>> when we have a better DAG consistency story.
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>> I think it would also be good to think about this
>>>> > > > > > >>>>>> proposal/implementation
>>>> > > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g.
>>>> when
>>>> > > > > > >> webserver
>>>> > > > > > >>>> hits
>>>> > > > > > >>>>>> APIs
>>>> > > > > > >>>>>>>>>> like /clear on behalf of users instead of running
>>>> > commands
>>>> > > > > > >>> against
>>>> > > > > > >>>>> the
>>>> > > > > > >>>>>>>>>> database directly).
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
>>>> > > > > > >>>> bdbruin@gmail.com
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>>>>>> wrote:
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give
>>>> me a
>>>> > > few
>>>> > > > > > >>> days.
>>>> > > > > > >>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>> Sent from my iPhone
>>>> > > > > > >>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
>>>> > > > > > >>>> criccomini@apache.org>
>>>> > > > > > >>>>>>>>>> wrote:
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> Hey all,
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of
>>>> time
>>>> > > > thinking
>>>> > > > > > >>>> about
>>>> > > > > > >>>>>>>>> this,
>>>> > > > > > >>>>>>>>>>> and
>>>> > > > > > >>>>>>>>>>>> want to move forward with it, but want to make
>>>> sure
>>>> > > we're
>>>> > > > > > >> all
>>>> > > > > > >>> on
>>>> > > > > > >>>>> the
>>>> > > > > > >>>>>>>>>> same
>>>> > > > > > >>>>>>>>>>>> page.
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> Cheers,
>>>> > > > > > >>>>>>>>>>>> Chris
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh
>>>> dharwadkar <
>>>> > > > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> Hello everyone,
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have
>>>> a
>>>> > > > built-in
>>>> > > > > > >>> Role
>>>> > > > > > >>>>>>>>> Based
>>>> > > > > > >>>>>>>>>>>>> Access Control(RBAC) capability.  It does
>>>> provide
>>>> > very
>>>> > > > > > >>> limited
>>>> > > > > > >>>>>>>>>>>>> authorization capability by providing admin,
>>>> > > > data_profiler,
>>>> > > > > > >>> and
>>>> > > > > > >>>>>> user
>>>> > > > > > >>>>>>>>>>> roles.
>>>> > > > > > >>>>>>>>>>>>> However, associating these roles to
>>>> authenticated
>>>> > > > > > >> identities
>>>> > > > > > >>> is
>>>> > > > > > >>>>> not
>>>> > > > > > >>>>>>>>> a
>>>> > > > > > >>>>>>>>>>>>> simple effort.
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> To address this issue, I have created a design
>>>> > proposal
>>>> > > > for
>>>> > > > > > >>>>>> building
>>>> > > > > > >>>>>>>>>>> RBAC
>>>> > > > > > >>>>>>>>>>>>> into Airflow and simplifying user access
>>>> management
>>>> > via
>>>> > > > the
>>>> > > > > > >>>>> Airflow
>>>> > > > > > >>>>>>>>>> UI.
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> The design proposal is located at
>>>> > > > > > >> https://cwiki.apache.org/
>>>> > > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Air
>>>> flow+RBAC+proposal
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much
>>>> appreciated.
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> Thanks
>>>> > > > > > >>>>>>>>>>>>> Kalpesh
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>
>>>> > > > > > >>>>
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>> --
>>>> > > > > > >>>
>>>> > > > > > >>> Joy Gao
>>>> > > > > > >>> Software Engineer
>>>> > > > > > >>> 350 Convention Way, Suite 200
>>>> > > > > > >>> Redwood City, CA 94063
>>>> > > > > > >>> Mobile:  669-224-9305
>>>> > > > > > >>>
>>>> > > > > > >>> Payments partner to the platform economy
>>>> > > > > > >>>
>>>> > > > > > >>
>>>> > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>

Re: Role Based Access Control for Airflow UI

Posted by Chris Riccomini <cr...@apache.org>.

Hey all,

Unfortunately, we're going to have to reschedule this meeting. A few
critical folks ended up having to back out at the last minute. I'll send an
update when I've coordinated a new time. Ideally, I'd like to do it later
this week, but we'll see.

Cheers,
Chris

On Tue, Jul 25, 2017 at 3:48 PM, Chris Riccomini <cr...@apache.org>
wrote:

> Invite sent for 8/2 from 3pm-5pm Pacific time! Here's the dial-in info:
>
> Join from PC, Mac, Linux, iOS or Android: https://wepay.zoom.us/j/
> 994482435
>
> Or iPhone one-tap (US Toll):  +16465588656 <(646)%20558-8656>,,994482435#
> or +14086380968 <(408)%20638-0968>,,994482435#
>
> Or Telephone:
>     Dial: +1 646 558 8656 <(646)%20558-8656> (US Toll) or +1 408 638 0968
> <(408)%20638-0968> (US Toll)
>     +1 855 880 1246 <(855)%20880-1246> (US Toll Free)
>     +1 877 369 0926 <(877)%20369-0926> (US Toll Free)
>     Meeting ID: 994 482 435
>     International numbers available: https://wepay.zoom.us/
> zoomconference?m=wHL1YsNLTvoAf8xdE8ki9yNEPVcus_eF
>
>
> On Tue, Jul 25, 2017 at 2:53 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
>> Great! I'll send an invite.
>>
>> On Tue, Jul 25, 2017 at 1:43 PM, Dan Davydov <
>> dan.davydov@airbnb.com.invalid> wrote:
>>
>>> Same as Alex, would be great to be able to remote in though I'm very
>>> interested.
>>>
>>> On Tue, Jul 25, 2017 at 1:37 PM, Alex Guziel <alex.guziel@airbnb.com
>>> .invalid
>>> > wrote:
>>>
>>> > Yeah, I could call in but I probably won't be able to come down that
>>> day.
>>> >
>>> > On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
>>> > maximebeauchemin@gmail.com> wrote:
>>> >
>>> > > Works for me! Dan said he might confcall in. Alex?
>>> > >
>>> > > Max
>>> > >
>>> > > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <
>>> criccomini@apache.org
>>> > >
>>> > > wrote:
>>> > >
>>> > > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room
>>> for
>>> > 12
>>> > > > people (and video conferencing) at WePay in this time slot. Should
>>> > allow
>>> > > > you to head home easily afterwards. :) That work for you guys?
>>> > > >
>>> > > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
>>> > > > maximebeauchemin@gmail.com> wrote:
>>> > > >
>>> > > > > The week of the 31st sound good. Wednesday?
>>> > > > >
>>> > > > > About React we may not need a frontend lib like it (or at least
>>> not
>>> > > just
>>> > > > > yet). We can talk about it at the meeting.
>>> > > > >
>>> > > > > Max
>>> > > > >
>>> > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <
>>> bdbruin@gmail.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > > We avoid React for the same reasons as the ASF and use Polymer
>>> 2
>>> > > > instead.
>>> > > > > > Would that work?
>>> > > > > >
>>> > > > > > Bolke.
>>> > > > > >
>>> > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <
>>> criccomini@apache.org
>>> > >
>>> > > > > wrote:
>>> > > > > > >
>>> > > > > > > Hey Max,
>>> > > > > > >
>>> > > > > > > Want to come down to WePay? We can set up a zoom for those
>>> that
>>> > > want
>>> > > > to
>>> > > > > > > join online, and record it as well to post for the community.
>>> > > > > > >
>>> > > > > > > Since Joy is just getting started, and it looks like there's
>>> > going
>>> > > to
>>> > > > > be
>>> > > > > > a
>>> > > > > > > K8s discussion next week, maybe we can shoot for the week
>>> after
>>> > > (the
>>> > > > > week
>>> > > > > > > of the 31st of July)? Care to float a few times that week?
>>> > > > > > >
>>> > > > > > > Cheers,
>>> > > > > > > Chris
>>> > > > > > >
>>> > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
>>> > > > > > > maximebeauchemin@gmail.com> wrote:
>>> > > > > > >
>>> > > > > > >> Sounds awesome, count me in!
>>> > > > > > >>
>>> > > > > > >> * check out the prototype in my fork, I went far enough to
>>> hit
>>> > > some
>>> > > > > > >> hurdles, try different workarounds. I hooked up the Airflow
>>> > > > Bootstrap
>>> > > > > > >> template too so that we feel at home in this new UI
>>> > > > > > >> * using a single `id` field is a requirement for FAB that
>>> > airflow
>>> > > > > > doesn't
>>> > > > > > >> respect (composite pks), either we add the feature to
>>> support
>>> > that
>>> > > > in
>>> > > > > > FAB,
>>> > > > > > >> or we align on the Airflow side and modify the models and
>>> add a
>>> > > > > > migration
>>> > > > > > >> script. This upgrade would require downtime and might be
>>> > annoying
>>> > > to
>>> > > > > the
>>> > > > > > >> Airflow community, but could help with db performance a bit
>>> > > (smaller
>>> > > > > > >> index)... I probably could be convinced either way but I'm
>>> > leaning
>>> > > > on
>>> > > > > > >> improving FAB
>>> > > > > > >> * I'm a maintainer for FAB so I can help get stuff through
>>> there
>>> > > > > > >> * React is in limbo at the ASF for licensing reasons, so no
>>> > React
>>> > > at
>>> > > > > > least
>>> > > > > > >> for now
>>> > > > > > >> * npm/webpack/ES6, javascript only in `.js` files
>>> > > > > > >> * I vote for eslint + eslint-config-airbnb as a set of
>>> linting
>>> > > rules
>>> > > > > > for JS
>>> > > > > > >> * Keep out of apache (for now), this new app ships as its
>>> own
>>> > pypi
>>> > > > > > package
>>> > > > > > >> `airflow-webserver`, have a period of overlap (maintaining
>>> 2 web
>>> > > > apps)
>>> > > > > > >> before ripping out `airflow/www` from the core package
>>> > > > > > >> * You need to get in touch with Marty Kausas, an intern at
>>> > Airbnb
>>> > > > > who's
>>> > > > > > >> been working on a Flask blueprint for improved, more
>>> > personalized
>>> > > > > views
>>> > > > > > on
>>> > > > > > >> DAGs that we were planning on merging into the main branch
>>> > > > eventually.
>>> > > > > > Some
>>> > > > > > >> of Marty's idea and code could be merged into this effort.
>>> > > > > > >>
>>> > > > > > >> These are ideas on how I would proceed personally on this
>>> but
>>> > > > > definitely
>>> > > > > > >> everything here is up for discussion.
>>> > > > > > >>
>>> > > > > > >> Let's meet physically at either WePay or Airbnb. Folks from
>>> the
>>> > > > > > community,
>>> > > > > > >> let us know on this thread if you want to be part of this
>>> > effort,
>>> > > > > we'll
>>> > > > > > be
>>> > > > > > >> happy to include you.
>>> > > > > > >>
>>> > > > > > >> Thanks,
>>> > > > > > >>
>>> > > > > > >> Max
>>> > > > > > >>
>>> > > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com>
>>> > wrote:
>>> > > > > > >>
>>> > > > > > >>> Hey everyone,
>>> > > > > > >>>
>>> > > > > > >>> I recently transferred to Data Infra team here at WePay to
>>> > focus
>>> > > on
>>> > > > > > >>> Airflow-related initiatives.
>>> > > > > > >>>
>>> > > > > > >>> Given the RBAC design is mostly hashed out, I'm happy to
>>> get
>>> > this
>>> > > > > > feature
>>> > > > > > >>> off the ground for Q3, starting with converting Airflow to
>>> Fab,
>>> > > if
>>> > > > > > there
>>> > > > > > >>> are no objections.
>>> > > > > > >>>
>>> > > > > > >>> Cheers,
>>> > > > > > >>> Joy
>>> > > > > > >>>
>>> > > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
>>> > > > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
>>> > > > > > >>>
>>> > > > > > >>>> Hey all,
>>> > > > > > >>>>
>>> > > > > > >>>> We talked about this internally. We would like to work on
>>> this
>>> > > > > feature
>>> > > > > > >>> but
>>> > > > > > >>>> given the immediate priorities we are not going to be
>>> working
>>> > on
>>> > > > it
>>> > > > > in
>>> > > > > > >>> Q3.
>>> > > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we
>>> can
>>> > > work
>>> > > > > on
>>> > > > > > >> it
>>> > > > > > >>>> late Q4 or Q12018.
>>> > > > > > >>>>
>>> > > > > > >>>> Cheers,
>>> > > > > > >>>>
>>> > > > > > >>>> Gurer
>>> > > > > > >>>>
>>> > > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
>>> > > > > > >> criccomini@apache.org>
>>> > > > > > >>>> wrote:
>>> > > > > > >>>>
>>> > > > > > >>>>> I think FAB sounds like the right approach. Waiting to
>>> hear
>>> > > back
>>> > > > > with
>>> > > > > > >>>> notes
>>> > > > > > >>>>> on AirBNB H2 discussion to see if they want to take this
>>> up.
>>> > > > > > >>>>>
>>> > > > > > >>>>> @Gurer, any idea when this will happen?
>>> > > > > > >>>>>
>>> > > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
>>> > > > bdbruin@gmail.com
>>> > > > > >
>>> > > > > > >>>> wrote:
>>> > > > > > >>>>>
>>> > > > > > >>>>>> One downside I see from FAB is that is does not do
>>> Business
>>> > > Role
>>> > > > > > >>>> mapping
>>> > > > > > >>>>>> to FAB role. I would prefer to create groups in
>>> IPA/LDAP/AD
>>> > > and
>>> > > > > > >> have
>>> > > > > > >>>>> those
>>> > > > > > >>>>>> map to FAB roles instead of needing to manage that in
>>> FAB.
>>> > > > > > >>>>>>
>>> > > > > > >>>>>> B.
>>> > > > > > >>>>>>
>>> > > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <
>>> > bdbruin@gmail.com>
>>> > > > > > >>> wrote:
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> Hi Guys,
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> Thanks for putting the thinking in! It is about time
>>> that
>>> > we
>>> > > > get
>>> > > > > > >>> this
>>> > > > > > >>>>>> moving.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> The design looks pretty sound. One can argue about the
>>> > > > different
>>> > > > > > >>>> roles
>>> > > > > > >>>>>> that are required, but that will be situation dependent
>>> I
>>> > > guess.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> Implementation wise I would argue together with Max
>>> that
>>> > FAB
>>> > > > is a
>>> > > > > > >>>>> better
>>> > > > > > >>>>>> or best fit. The ER model that is being described is
>>> pretty
>>> > > > much a
>>> > > > > > >>> copy
>>> > > > > > >>>>> of
>>> > > > > > >>>>>> a normal security model. So a reimplementation of that
>>> is 1)
>>> > > > > > >>>> significant
>>> > > > > > >>>>>> duplication of effort and 2) bound to have bugs that
>>> have
>>> > been
>>> > > > > > >> solved
>>> > > > > > >>>> in
>>> > > > > > >>>>>> the other framework. Moreover, FAB does have
>>> integration out
>>> > > of
>>> > > > > the
>>> > > > > > >>> box
>>> > > > > > >>>>>> with some enterprisey systems like IPA,
>>> ActiveDirectory, and
>>> > > > LDAP.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> So while you argue that using FAB would increase the
>>> scope
>>> > of
>>> > > > the
>>> > > > > > >>>>>> proposal significantly, but I think that is not true.
>>> Using
>>> > > FAB
>>> > > > > > >> would
>>> > > > > > >>>>> allow
>>> > > > > > >>>>>> you to focus on what kind of out-of-the-box permission
>>> sets
>>> > > and
>>> > > > > > >> roles
>>> > > > > > >>>> we
>>> > > > > > >>>>>> would need and maybe address some issues that FAB lacks
>>> > (maybe
>>> > > > how
>>> > > > > > >> to
>>> > > > > > >>>>> deal
>>> > > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos,
>>> probably
>>> > > how
>>> > > > to
>>> > > > > > >>> deal
>>> > > > > > >>>>>> with API calls that are not CRUD). Implementation wise
>>> it
>>> > > > probably
>>> > > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s
>>> early POC
>>> > > as
>>> > > > an
>>> > > > > > >>>>> example
>>> > > > > > >>>>>> - we can slowly move over?
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on
>>> > > Airflow
>>> > > > > > >>> coming
>>> > > > > > >>>>>> year. Improvement of Security, Enterprise Integration,
>>> > Revamp
>>> > > UI
>>> > > > > > >> are
>>> > > > > > >>> on
>>> > > > > > >>>>> the
>>> > > > > > >>>>>> todo list. However, this is not confirmed yet as
>>> business
>>> > > > > > >> priorities
>>> > > > > > >>>>> might
>>> > > > > > >>>>>> change.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> Bolke.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
>>> > > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> @Dan:
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> Thanks for your feedback. I will remove the
>>> REFRESH_DAG
>>> > > > > > >>> permission.
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> @Max:
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> Thanks for your response.
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> The scope of my proposal was just to add RBAC security
>>> > > feature
>>> > > > > > >> to
>>> > > > > > >>>>>> Airflow
>>> > > > > > >>>>>>>> without replacing any existing frameworks.
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> I understand that adopting FAB would serve Airflow
>>> better
>>> > > > moving
>>> > > > > > >>>>>> forward,
>>> > > > > > >>>>>>>> however porting Airflow to using FAB significantly
>>> > increases
>>> > > > the
>>> > > > > > >>>> scope
>>> > > > > > >>>>>> of
>>> > > > > > >>>>>>>> the proposal and I don't have the time and expertise
>>> to
>>> > > carry
>>> > > > > > >> out
>>> > > > > > >>>> the
>>> > > > > > >>>>>> tasks
>>> > > > > > >>>>>>>> in the extended scope.
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for
>>> Airflow
>>> > to
>>> > > > > > >>> migrate
>>> > > > > > >>>> to
>>> > > > > > >>>>>> FAB
>>> > > > > > >>>>>>>> this year?
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> - Kalpesh
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
>>> > > > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>>> It would be nice to go with a framework for this. I
>>> did
>>> > > some
>>> > > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this
>>> > > > direction.
>>> > > > > > >> It
>>> > > > > > >>>>>> provides
>>> > > > > > >>>>>>>>> auth on different authentication backends out of the
>>> box
>>> > > > > > >> (oauth,
>>> > > > > > >>>>>> openid,
>>> > > > > > >>>>>>>>> ldap, registration, ...), generates perms for each
>>> view
>>> > > that
>>> > > > > > >> has
>>> > > > > > >>> an
>>> > > > > > >>>>>>>>> @has_access decorator, generates at set of perms for
>>> each
>>> > > ORM
>>> > > > > > >>> model
>>> > > > > > >>>>>> (show,
>>> > > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD
>>> views
>>> > > as
>>> > > > > > >> well
>>> > > > > > >>>> as
>>> > > > > > >>>>>> in the
>>> > > > > > >>>>>>>>> generated REST api that you get for free as a
>>> byprdoduct
>>> > of
>>> > > > > > >>>> deriving
>>> > > > > > >>>>>> FAB's
>>> > > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on
>>> top).
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> I started a POC on FAB here a while ago:
>>> > > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver
>>> at the
>>> > > > time
>>> > > > > > >> my
>>> > > > > > >>>>> main
>>> > > > > > >>>>>>>>> motivation was the free/instantaneous REST api.
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> I think FAB is a decent fit as the porting should be
>>> > fairly
>>> > > > > > >>>>>> straightforward
>>> > > > > > >>>>>>>>> (moving the flask views over and deprecating
>>> Flask-Admin
>>> > in
>>> > > > > > >> favor
>>> > > > > > >>>> of
>>> > > > > > >>>>>> FAB's
>>> > > > > > >>>>>>>>> crud) though there was a few blockers. From memory I
>>> > think
>>> > > > FAB
>>> > > > > > >>>> didn't
>>> > > > > > >>>>>> like
>>> > > > > > >>>>>>>>> the compound PKs we use in some of the Airflow
>>> models.
>>> > We'd
>>> > > > > > >> have
>>> > > > > > >>> to
>>> > > > > > >>>>>> either
>>> > > > > > >>>>>>>>> write a db migration script on the Airflow side, or
>>> add
>>> > > > support
>>> > > > > > >>> for
>>> > > > > > >>>>>>>>> compound keys to FAB (I recently became a maintainer
>>> of
>>> > the
>>> > > > > > >>>> project,
>>> > > > > > >>>>>> so I
>>> > > > > > >>>>>>>>> could help with that)
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> The only downside of FAB is that it's not as mature
>>> as
>>> > > > > > >> something
>>> > > > > > >>>> like
>>> > > > > > >>>>>>>>> Django, but porting to Django would surely be much
>>> more
>>> > > work.
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> Then there's the flask-security suite, but that looks
>>> > like
>>> > > a
>>> > > > > > >> bit
>>> > > > > > >>>> of a
>>> > > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose
>>> which we
>>> > > want
>>> > > > > > >> to
>>> > > > > > >>>> use.
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> Max
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
>>> > > > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>>> Looks good to me in general, thanks for putting this
>>> > > > together!
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>> I think the ability to integrate with external RBAC
>>> > > systems
>>> > > > > > >> like
>>> > > > > > >>>>> LDAP
>>> > > > > > >>>>>> is
>>> > > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be
>>> decoupled
>>> > > with
>>> > > > > > >> the
>>> > > > > > >>>> RBAC
>>> > > > > > >>>>>>>>>> database wherever possible).
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>> I wouldn't be too worried about the permissions
>>> about
>>> > > > > > >> refreshing
>>> > > > > > >>>>>> DAGs, as
>>> > > > > > >>>>>>>>>> far as I know this functionality is no longer
>>> required
>>> > > with
>>> > > > > > >> the
>>> > > > > > >>>> new
>>> > > > > > >>>>>>>>>> webservers which reload state periodically, and will
>>> > > > certainly
>>> > > > > > >>> be
>>> > > > > > >>>>>> removed
>>> > > > > > >>>>>>>>>> when we have a better DAG consistency story.
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>> I think it would also be good to think about this
>>> > > > > > >>>>>> proposal/implementation
>>> > > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g.
>>> when
>>> > > > > > >> webserver
>>> > > > > > >>>> hits
>>> > > > > > >>>>>> APIs
>>> > > > > > >>>>>>>>>> like /clear on behalf of users instead of running
>>> > commands
>>> > > > > > >>> against
>>> > > > > > >>>>> the
>>> > > > > > >>>>>>>>>> database directly).
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
>>> > > > > > >>>> bdbruin@gmail.com
>>> > > > > > >>>>>>
>>> > > > > > >>>>>>>>>> wrote:
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give
>>> me a
>>> > > few
>>> > > > > > >>> days.
>>> > > > > > >>>>>>>>>>>
>>> > > > > > >>>>>>>>>>> Sent from my iPhone
>>> > > > > > >>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
>>> > > > > > >>>> criccomini@apache.org>
>>> > > > > > >>>>>>>>>> wrote:
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> Hey all,
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time
>>> > > > thinking
>>> > > > > > >>>> about
>>> > > > > > >>>>>>>>> this,
>>> > > > > > >>>>>>>>>>> and
>>> > > > > > >>>>>>>>>>>> want to move forward with it, but want to make
>>> sure
>>> > > we're
>>> > > > > > >> all
>>> > > > > > >>> on
>>> > > > > > >>>>> the
>>> > > > > > >>>>>>>>>> same
>>> > > > > > >>>>>>>>>>>> page.
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> Cheers,
>>> > > > > > >>>>>>>>>>>> Chris
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh
>>> dharwadkar <
>>> > > > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> Hello everyone,
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a
>>> > > > built-in
>>> > > > > > >>> Role
>>> > > > > > >>>>>>>>> Based
>>> > > > > > >>>>>>>>>>>>> Access Control(RBAC) capability.  It does provide
>>> > very
>>> > > > > > >>> limited
>>> > > > > > >>>>>>>>>>>>> authorization capability by providing admin,
>>> > > > data_profiler,
>>> > > > > > >>> and
>>> > > > > > >>>>>> user
>>> > > > > > >>>>>>>>>>> roles.
>>> > > > > > >>>>>>>>>>>>> However, associating these roles to authenticated
>>> > > > > > >> identities
>>> > > > > > >>> is
>>> > > > > > >>>>> not
>>> > > > > > >>>>>>>>> a
>>> > > > > > >>>>>>>>>>>>> simple effort.
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> To address this issue, I have created a design
>>> > proposal
>>> > > > for
>>> > > > > > >>>>>> building
>>> > > > > > >>>>>>>>>>> RBAC
>>> > > > > > >>>>>>>>>>>>> into Airflow and simplifying user access
>>> management
>>> > via
>>> > > > the
>>> > > > > > >>>>> Airflow
>>> > > > > > >>>>>>>>>> UI.
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> The design proposal is located at
>>> > > > > > >> https://cwiki.apache.org/
>>> > > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much
>>> appreciated.
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> Thanks
>>> > > > > > >>>>>>>>>>>>> Kalpesh
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>
>>> > > > > > >>>>>>
>>> > > > > > >>>>>
>>> > > > > > >>>>
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>> --
>>> > > > > > >>>
>>> > > > > > >>> Joy Gao
>>> > > > > > >>> Software Engineer
>>> > > > > > >>> 350 Convention Way, Suite 200
>>> > > > > > >>> Redwood City, CA 94063
>>> > > > > > >>> Mobile:  669-224-9305
>>> > > > > > >>>
>>> > > > > > >>> Payments partner to the platform economy
>>> > > > > > >>>
>>> > > > > > >>
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>

Re: Role Based Access Control for Airflow UI

Posted by Chris Riccomini <cr...@apache.org>.

Invite sent for 8/2 from 3pm-5pm Pacific time! Here's the dial-in info:

Join from PC, Mac, Linux, iOS or Android: https://wepay.zoom.us/j/994482435

Or iPhone one-tap (US Toll):  +16465588656,,994482435# or
+14086380968,,994482435#

Or Telephone:
    Dial: +1 646 558 8656 (US Toll) or +1 408 638 0968 (US Toll)
    +1 855 880 1246 (US Toll Free)
    +1 877 369 0926 (US Toll Free)
    Meeting ID: 994 482 435
    International numbers available:
https://wepay.zoom.us/zoomconference?m=wHL1YsNLTvoAf8xdE8ki9yNEPVcus_eF


On Tue, Jul 25, 2017 at 2:53 PM, Chris Riccomini <cr...@apache.org>
wrote:

> Great! I'll send an invite.
>
> On Tue, Jul 25, 2017 at 1:43 PM, Dan Davydov <dan.davydov@airbnb.com.
> invalid> wrote:
>
>> Same as Alex, would be great to be able to remote in though I'm very
>> interested.
>>
>> On Tue, Jul 25, 2017 at 1:37 PM, Alex Guziel <alex.guziel@airbnb.com
>> .invalid
>> > wrote:
>>
>> > Yeah, I could call in but I probably won't be able to come down that
>> day.
>> >
>> > On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
>> > maximebeauchemin@gmail.com> wrote:
>> >
>> > > Works for me! Dan said he might confcall in. Alex?
>> > >
>> > > Max
>> > >
>> > > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <
>> criccomini@apache.org
>> > >
>> > > wrote:
>> > >
>> > > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room
>> for
>> > 12
>> > > > people (and video conferencing) at WePay in this time slot. Should
>> > allow
>> > > > you to head home easily afterwards. :) That work for you guys?
>> > > >
>> > > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
>> > > > maximebeauchemin@gmail.com> wrote:
>> > > >
>> > > > > The week of the 31st sound good. Wednesday?
>> > > > >
>> > > > > About React we may not need a frontend lib like it (or at least
>> not
>> > > just
>> > > > > yet). We can talk about it at the meeting.
>> > > > >
>> > > > > Max
>> > > > >
>> > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <
>> bdbruin@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > We avoid React for the same reasons as the ASF and use Polymer 2
>> > > > instead.
>> > > > > > Would that work?
>> > > > > >
>> > > > > > Bolke.
>> > > > > >
>> > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <
>> criccomini@apache.org
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > Hey Max,
>> > > > > > >
>> > > > > > > Want to come down to WePay? We can set up a zoom for those
>> that
>> > > want
>> > > > to
>> > > > > > > join online, and record it as well to post for the community.
>> > > > > > >
>> > > > > > > Since Joy is just getting started, and it looks like there's
>> > going
>> > > to
>> > > > > be
>> > > > > > a
>> > > > > > > K8s discussion next week, maybe we can shoot for the week
>> after
>> > > (the
>> > > > > week
>> > > > > > > of the 31st of July)? Care to float a few times that week?
>> > > > > > >
>> > > > > > > Cheers,
>> > > > > > > Chris
>> > > > > > >
>> > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
>> > > > > > > maximebeauchemin@gmail.com> wrote:
>> > > > > > >
>> > > > > > >> Sounds awesome, count me in!
>> > > > > > >>
>> > > > > > >> * check out the prototype in my fork, I went far enough to
>> hit
>> > > some
>> > > > > > >> hurdles, try different workarounds. I hooked up the Airflow
>> > > > Bootstrap
>> > > > > > >> template too so that we feel at home in this new UI
>> > > > > > >> * using a single `id` field is a requirement for FAB that
>> > airflow
>> > > > > > doesn't
>> > > > > > >> respect (composite pks), either we add the feature to support
>> > that
>> > > > in
>> > > > > > FAB,
>> > > > > > >> or we align on the Airflow side and modify the models and
>> add a
>> > > > > > migration
>> > > > > > >> script. This upgrade would require downtime and might be
>> > annoying
>> > > to
>> > > > > the
>> > > > > > >> Airflow community, but could help with db performance a bit
>> > > (smaller
>> > > > > > >> index)... I probably could be convinced either way but I'm
>> > leaning
>> > > > on
>> > > > > > >> improving FAB
>> > > > > > >> * I'm a maintainer for FAB so I can help get stuff through
>> there
>> > > > > > >> * React is in limbo at the ASF for licensing reasons, so no
>> > React
>> > > at
>> > > > > > least
>> > > > > > >> for now
>> > > > > > >> * npm/webpack/ES6, javascript only in `.js` files
>> > > > > > >> * I vote for eslint + eslint-config-airbnb as a set of
>> linting
>> > > rules
>> > > > > > for JS
>> > > > > > >> * Keep out of apache (for now), this new app ships as its own
>> > pypi
>> > > > > > package
>> > > > > > >> `airflow-webserver`, have a period of overlap (maintaining 2
>> web
>> > > > apps)
>> > > > > > >> before ripping out `airflow/www` from the core package
>> > > > > > >> * You need to get in touch with Marty Kausas, an intern at
>> > Airbnb
>> > > > > who's
>> > > > > > >> been working on a Flask blueprint for improved, more
>> > personalized
>> > > > > views
>> > > > > > on
>> > > > > > >> DAGs that we were planning on merging into the main branch
>> > > > eventually.
>> > > > > > Some
>> > > > > > >> of Marty's idea and code could be merged into this effort.
>> > > > > > >>
>> > > > > > >> These are ideas on how I would proceed personally on this but
>> > > > > definitely
>> > > > > > >> everything here is up for discussion.
>> > > > > > >>
>> > > > > > >> Let's meet physically at either WePay or Airbnb. Folks from
>> the
>> > > > > > community,
>> > > > > > >> let us know on this thread if you want to be part of this
>> > effort,
>> > > > > we'll
>> > > > > > be
>> > > > > > >> happy to include you.
>> > > > > > >>
>> > > > > > >> Thanks,
>> > > > > > >>
>> > > > > > >> Max
>> > > > > > >>
>> > > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com>
>> > wrote:
>> > > > > > >>
>> > > > > > >>> Hey everyone,
>> > > > > > >>>
>> > > > > > >>> I recently transferred to Data Infra team here at WePay to
>> > focus
>> > > on
>> > > > > > >>> Airflow-related initiatives.
>> > > > > > >>>
>> > > > > > >>> Given the RBAC design is mostly hashed out, I'm happy to get
>> > this
>> > > > > > feature
>> > > > > > >>> off the ground for Q3, starting with converting Airflow to
>> Fab,
>> > > if
>> > > > > > there
>> > > > > > >>> are no objections.
>> > > > > > >>>
>> > > > > > >>> Cheers,
>> > > > > > >>> Joy
>> > > > > > >>>
>> > > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
>> > > > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
>> > > > > > >>>
>> > > > > > >>>> Hey all,
>> > > > > > >>>>
>> > > > > > >>>> We talked about this internally. We would like to work on
>> this
>> > > > > feature
>> > > > > > >>> but
>> > > > > > >>>> given the immediate priorities we are not going to be
>> working
>> > on
>> > > > it
>> > > > > in
>> > > > > > >>> Q3.
>> > > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we
>> can
>> > > work
>> > > > > on
>> > > > > > >> it
>> > > > > > >>>> late Q4 or Q12018.
>> > > > > > >>>>
>> > > > > > >>>> Cheers,
>> > > > > > >>>>
>> > > > > > >>>> Gurer
>> > > > > > >>>>
>> > > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
>> > > > > > >> criccomini@apache.org>
>> > > > > > >>>> wrote:
>> > > > > > >>>>
>> > > > > > >>>>> I think FAB sounds like the right approach. Waiting to
>> hear
>> > > back
>> > > > > with
>> > > > > > >>>> notes
>> > > > > > >>>>> on AirBNB H2 discussion to see if they want to take this
>> up.
>> > > > > > >>>>>
>> > > > > > >>>>> @Gurer, any idea when this will happen?
>> > > > > > >>>>>
>> > > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
>> > > > bdbruin@gmail.com
>> > > > > >
>> > > > > > >>>> wrote:
>> > > > > > >>>>>
>> > > > > > >>>>>> One downside I see from FAB is that is does not do
>> Business
>> > > Role
>> > > > > > >>>> mapping
>> > > > > > >>>>>> to FAB role. I would prefer to create groups in
>> IPA/LDAP/AD
>> > > and
>> > > > > > >> have
>> > > > > > >>>>> those
>> > > > > > >>>>>> map to FAB roles instead of needing to manage that in
>> FAB.
>> > > > > > >>>>>>
>> > > > > > >>>>>> B.
>> > > > > > >>>>>>
>> > > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <
>> > bdbruin@gmail.com>
>> > > > > > >>> wrote:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Hi Guys,
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Thanks for putting the thinking in! It is about time
>> that
>> > we
>> > > > get
>> > > > > > >>> this
>> > > > > > >>>>>> moving.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> The design looks pretty sound. One can argue about the
>> > > > different
>> > > > > > >>>> roles
>> > > > > > >>>>>> that are required, but that will be situation dependent I
>> > > guess.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Implementation wise I would argue together with Max that
>> > FAB
>> > > > is a
>> > > > > > >>>>> better
>> > > > > > >>>>>> or best fit. The ER model that is being described is
>> pretty
>> > > > much a
>> > > > > > >>> copy
>> > > > > > >>>>> of
>> > > > > > >>>>>> a normal security model. So a reimplementation of that
>> is 1)
>> > > > > > >>>> significant
>> > > > > > >>>>>> duplication of effort and 2) bound to have bugs that have
>> > been
>> > > > > > >> solved
>> > > > > > >>>> in
>> > > > > > >>>>>> the other framework. Moreover, FAB does have integration
>> out
>> > > of
>> > > > > the
>> > > > > > >>> box
>> > > > > > >>>>>> with some enterprisey systems like IPA, ActiveDirectory,
>> and
>> > > > LDAP.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> So while you argue that using FAB would increase the
>> scope
>> > of
>> > > > the
>> > > > > > >>>>>> proposal significantly, but I think that is not true.
>> Using
>> > > FAB
>> > > > > > >> would
>> > > > > > >>>>> allow
>> > > > > > >>>>>> you to focus on what kind of out-of-the-box permission
>> sets
>> > > and
>> > > > > > >> roles
>> > > > > > >>>> we
>> > > > > > >>>>>> would need and maybe address some issues that FAB lacks
>> > (maybe
>> > > > how
>> > > > > > >> to
>> > > > > > >>>>> deal
>> > > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos,
>> probably
>> > > how
>> > > > to
>> > > > > > >>> deal
>> > > > > > >>>>>> with API calls that are not CRUD). Implementation wise it
>> > > > probably
>> > > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s early
>> POC
>> > > as
>> > > > an
>> > > > > > >>>>> example
>> > > > > > >>>>>> - we can slowly move over?
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on
>> > > Airflow
>> > > > > > >>> coming
>> > > > > > >>>>>> year. Improvement of Security, Enterprise Integration,
>> > Revamp
>> > > UI
>> > > > > > >> are
>> > > > > > >>> on
>> > > > > > >>>>> the
>> > > > > > >>>>>> todo list. However, this is not confirmed yet as business
>> > > > > > >> priorities
>> > > > > > >>>>> might
>> > > > > > >>>>>> change.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Bolke.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
>> > > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> @Dan:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG
>> > > > > > >>> permission.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> @Max:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Thanks for your response.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> The scope of my proposal was just to add RBAC security
>> > > feature
>> > > > > > >> to
>> > > > > > >>>>>> Airflow
>> > > > > > >>>>>>>> without replacing any existing frameworks.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> I understand that adopting FAB would serve Airflow
>> better
>> > > > moving
>> > > > > > >>>>>> forward,
>> > > > > > >>>>>>>> however porting Airflow to using FAB significantly
>> > increases
>> > > > the
>> > > > > > >>>> scope
>> > > > > > >>>>>> of
>> > > > > > >>>>>>>> the proposal and I don't have the time and expertise to
>> > > carry
>> > > > > > >> out
>> > > > > > >>>> the
>> > > > > > >>>>>> tasks
>> > > > > > >>>>>>>> in the extended scope.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for
>> Airflow
>> > to
>> > > > > > >>> migrate
>> > > > > > >>>> to
>> > > > > > >>>>>> FAB
>> > > > > > >>>>>>>> this year?
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> - Kalpesh
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
>> > > > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> It would be nice to go with a framework for this. I
>> did
>> > > some
>> > > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this
>> > > > direction.
>> > > > > > >> It
>> > > > > > >>>>>> provides
>> > > > > > >>>>>>>>> auth on different authentication backends out of the
>> box
>> > > > > > >> (oauth,
>> > > > > > >>>>>> openid,
>> > > > > > >>>>>>>>> ldap, registration, ...), generates perms for each
>> view
>> > > that
>> > > > > > >> has
>> > > > > > >>> an
>> > > > > > >>>>>>>>> @has_access decorator, generates at set of perms for
>> each
>> > > ORM
>> > > > > > >>> model
>> > > > > > >>>>>> (show,
>> > > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD
>> views
>> > > as
>> > > > > > >> well
>> > > > > > >>>> as
>> > > > > > >>>>>> in the
>> > > > > > >>>>>>>>> generated REST api that you get for free as a
>> byprdoduct
>> > of
>> > > > > > >>>> deriving
>> > > > > > >>>>>> FAB's
>> > > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on
>> top).
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> I started a POC on FAB here a while ago:
>> > > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at
>> the
>> > > > time
>> > > > > > >> my
>> > > > > > >>>>> main
>> > > > > > >>>>>>>>> motivation was the free/instantaneous REST api.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> I think FAB is a decent fit as the porting should be
>> > fairly
>> > > > > > >>>>>> straightforward
>> > > > > > >>>>>>>>> (moving the flask views over and deprecating
>> Flask-Admin
>> > in
>> > > > > > >> favor
>> > > > > > >>>> of
>> > > > > > >>>>>> FAB's
>> > > > > > >>>>>>>>> crud) though there was a few blockers. From memory I
>> > think
>> > > > FAB
>> > > > > > >>>> didn't
>> > > > > > >>>>>> like
>> > > > > > >>>>>>>>> the compound PKs we use in some of the Airflow models.
>> > We'd
>> > > > > > >> have
>> > > > > > >>> to
>> > > > > > >>>>>> either
>> > > > > > >>>>>>>>> write a db migration script on the Airflow side, or
>> add
>> > > > support
>> > > > > > >>> for
>> > > > > > >>>>>>>>> compound keys to FAB (I recently became a maintainer
>> of
>> > the
>> > > > > > >>>> project,
>> > > > > > >>>>>> so I
>> > > > > > >>>>>>>>> could help with that)
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> The only downside of FAB is that it's not as mature as
>> > > > > > >> something
>> > > > > > >>>> like
>> > > > > > >>>>>>>>> Django, but porting to Django would surely be much
>> more
>> > > work.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> Then there's the flask-security suite, but that looks
>> > like
>> > > a
>> > > > > > >> bit
>> > > > > > >>>> of a
>> > > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose which
>> we
>> > > want
>> > > > > > >> to
>> > > > > > >>>> use.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> Max
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
>> > > > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> Looks good to me in general, thanks for putting this
>> > > > together!
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> I think the ability to integrate with external RBAC
>> > > systems
>> > > > > > >> like
>> > > > > > >>>>> LDAP
>> > > > > > >>>>>> is
>> > > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be
>> decoupled
>> > > with
>> > > > > > >> the
>> > > > > > >>>> RBAC
>> > > > > > >>>>>>>>>> database wherever possible).
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> I wouldn't be too worried about the permissions about
>> > > > > > >> refreshing
>> > > > > > >>>>>> DAGs, as
>> > > > > > >>>>>>>>>> far as I know this functionality is no longer
>> required
>> > > with
>> > > > > > >> the
>> > > > > > >>>> new
>> > > > > > >>>>>>>>>> webservers which reload state periodically, and will
>> > > > certainly
>> > > > > > >>> be
>> > > > > > >>>>>> removed
>> > > > > > >>>>>>>>>> when we have a better DAG consistency story.
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> I think it would also be good to think about this
>> > > > > > >>>>>> proposal/implementation
>> > > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g. when
>> > > > > > >> webserver
>> > > > > > >>>> hits
>> > > > > > >>>>>> APIs
>> > > > > > >>>>>>>>>> like /clear on behalf of users instead of running
>> > commands
>> > > > > > >>> against
>> > > > > > >>>>> the
>> > > > > > >>>>>>>>>> database directly).
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
>> > > > > > >>>> bdbruin@gmail.com
>> > > > > > >>>>>>
>> > > > > > >>>>>>>>>> wrote:
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give
>> me a
>> > > few
>> > > > > > >>> days.
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> Sent from my iPhone
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
>> > > > > > >>>> criccomini@apache.org>
>> > > > > > >>>>>>>>>> wrote:
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> Hey all,
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time
>> > > > thinking
>> > > > > > >>>> about
>> > > > > > >>>>>>>>> this,
>> > > > > > >>>>>>>>>>> and
>> > > > > > >>>>>>>>>>>> want to move forward with it, but want to make sure
>> > > we're
>> > > > > > >> all
>> > > > > > >>> on
>> > > > > > >>>>> the
>> > > > > > >>>>>>>>>> same
>> > > > > > >>>>>>>>>>>> page.
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> Cheers,
>> > > > > > >>>>>>>>>>>> Chris
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar
>> <
>> > > > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Hello everyone,
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a
>> > > > built-in
>> > > > > > >>> Role
>> > > > > > >>>>>>>>> Based
>> > > > > > >>>>>>>>>>>>> Access Control(RBAC) capability.  It does provide
>> > very
>> > > > > > >>> limited
>> > > > > > >>>>>>>>>>>>> authorization capability by providing admin,
>> > > > data_profiler,
>> > > > > > >>> and
>> > > > > > >>>>>> user
>> > > > > > >>>>>>>>>>> roles.
>> > > > > > >>>>>>>>>>>>> However, associating these roles to authenticated
>> > > > > > >> identities
>> > > > > > >>> is
>> > > > > > >>>>> not
>> > > > > > >>>>>>>>> a
>> > > > > > >>>>>>>>>>>>> simple effort.
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> To address this issue, I have created a design
>> > proposal
>> > > > for
>> > > > > > >>>>>> building
>> > > > > > >>>>>>>>>>> RBAC
>> > > > > > >>>>>>>>>>>>> into Airflow and simplifying user access
>> management
>> > via
>> > > > the
>> > > > > > >>>>> Airflow
>> > > > > > >>>>>>>>>> UI.
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> The design proposal is located at
>> > > > > > >> https://cwiki.apache.org/
>> > > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much
>> appreciated.
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Thanks
>> > > > > > >>>>>>>>>>>>> Kalpesh
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>
>> > > > > > >>>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>> --
>> > > > > > >>>
>> > > > > > >>> Joy Gao
>> > > > > > >>> Software Engineer
>> > > > > > >>> 350 Convention Way, Suite 200
>> > > > > > >>> Redwood City, CA 94063
>> > > > > > >>> Mobile:  669-224-9305
>> > > > > > >>>
>> > > > > > >>> Payments partner to the platform economy
>> > > > > > >>>
>> > > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Role Based Access Control for Airflow UI

Posted by Chris Riccomini <cr...@apache.org>.

Great! I'll send an invite.

On Tue, Jul 25, 2017 at 1:43 PM, Dan Davydov <dan.davydov@airbnb.com.invalid
> wrote:

> Same as Alex, would be great to be able to remote in though I'm very
> interested.
>
> On Tue, Jul 25, 2017 at 1:37 PM, Alex Guziel <alex.guziel@airbnb.com.
> invalid
> > wrote:
>
> > Yeah, I could call in but I probably won't be able to come down that day.
> >
> > On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> >
> > > Works for me! Dan said he might confcall in. Alex?
> > >
> > > Max
> > >
> > > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <
> criccomini@apache.org
> > >
> > > wrote:
> > >
> > > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room for
> > 12
> > > > people (and video conferencing) at WePay in this time slot. Should
> > allow
> > > > you to head home easily afterwards. :) That work for you guys?
> > > >
> > > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
> > > > maximebeauchemin@gmail.com> wrote:
> > > >
> > > > > The week of the 31st sound good. Wednesday?
> > > > >
> > > > > About React we may not need a frontend lib like it (or at least not
> > > just
> > > > > yet). We can talk about it at the meeting.
> > > > >
> > > > > Max
> > > > >
> > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <
> bdbruin@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > We avoid React for the same reasons as the ASF and use Polymer 2
> > > > instead.
> > > > > > Would that work?
> > > > > >
> > > > > > Bolke.
> > > > > >
> > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <
> criccomini@apache.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > Hey Max,
> > > > > > >
> > > > > > > Want to come down to WePay? We can set up a zoom for those that
> > > want
> > > > to
> > > > > > > join online, and record it as well to post for the community.
> > > > > > >
> > > > > > > Since Joy is just getting started, and it looks like there's
> > going
> > > to
> > > > > be
> > > > > > a
> > > > > > > K8s discussion next week, maybe we can shoot for the week after
> > > (the
> > > > > week
> > > > > > > of the 31st of July)? Care to float a few times that week?
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Chris
> > > > > > >
> > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
> > > > > > > maximebeauchemin@gmail.com> wrote:
> > > > > > >
> > > > > > >> Sounds awesome, count me in!
> > > > > > >>
> > > > > > >> * check out the prototype in my fork, I went far enough to hit
> > > some
> > > > > > >> hurdles, try different workarounds. I hooked up the Airflow
> > > > Bootstrap
> > > > > > >> template too so that we feel at home in this new UI
> > > > > > >> * using a single `id` field is a requirement for FAB that
> > airflow
> > > > > > doesn't
> > > > > > >> respect (composite pks), either we add the feature to support
> > that
> > > > in
> > > > > > FAB,
> > > > > > >> or we align on the Airflow side and modify the models and add
> a
> > > > > > migration
> > > > > > >> script. This upgrade would require downtime and might be
> > annoying
> > > to
> > > > > the
> > > > > > >> Airflow community, but could help with db performance a bit
> > > (smaller
> > > > > > >> index)... I probably could be convinced either way but I'm
> > leaning
> > > > on
> > > > > > >> improving FAB
> > > > > > >> * I'm a maintainer for FAB so I can help get stuff through
> there
> > > > > > >> * React is in limbo at the ASF for licensing reasons, so no
> > React
> > > at
> > > > > > least
> > > > > > >> for now
> > > > > > >> * npm/webpack/ES6, javascript only in `.js` files
> > > > > > >> * I vote for eslint + eslint-config-airbnb as a set of linting
> > > rules
> > > > > > for JS
> > > > > > >> * Keep out of apache (for now), this new app ships as its own
> > pypi
> > > > > > package
> > > > > > >> `airflow-webserver`, have a period of overlap (maintaining 2
> web
> > > > apps)
> > > > > > >> before ripping out `airflow/www` from the core package
> > > > > > >> * You need to get in touch with Marty Kausas, an intern at
> > Airbnb
> > > > > who's
> > > > > > >> been working on a Flask blueprint for improved, more
> > personalized
> > > > > views
> > > > > > on
> > > > > > >> DAGs that we were planning on merging into the main branch
> > > > eventually.
> > > > > > Some
> > > > > > >> of Marty's idea and code could be merged into this effort.
> > > > > > >>
> > > > > > >> These are ideas on how I would proceed personally on this but
> > > > > definitely
> > > > > > >> everything here is up for discussion.
> > > > > > >>
> > > > > > >> Let's meet physically at either WePay or Airbnb. Folks from
> the
> > > > > > community,
> > > > > > >> let us know on this thread if you want to be part of this
> > effort,
> > > > > we'll
> > > > > > be
> > > > > > >> happy to include you.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >>
> > > > > > >> Max
> > > > > > >>
> > > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com>
> > wrote:
> > > > > > >>
> > > > > > >>> Hey everyone,
> > > > > > >>>
> > > > > > >>> I recently transferred to Data Infra team here at WePay to
> > focus
> > > on
> > > > > > >>> Airflow-related initiatives.
> > > > > > >>>
> > > > > > >>> Given the RBAC design is mostly hashed out, I'm happy to get
> > this
> > > > > > feature
> > > > > > >>> off the ground for Q3, starting with converting Airflow to
> Fab,
> > > if
> > > > > > there
> > > > > > >>> are no objections.
> > > > > > >>>
> > > > > > >>> Cheers,
> > > > > > >>> Joy
> > > > > > >>>
> > > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> > > > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
> > > > > > >>>
> > > > > > >>>> Hey all,
> > > > > > >>>>
> > > > > > >>>> We talked about this internally. We would like to work on
> this
> > > > > feature
> > > > > > >>> but
> > > > > > >>>> given the immediate priorities we are not going to be
> working
> > on
> > > > it
> > > > > in
> > > > > > >>> Q3.
> > > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we
> can
> > > work
> > > > > on
> > > > > > >> it
> > > > > > >>>> late Q4 or Q12018.
> > > > > > >>>>
> > > > > > >>>> Cheers,
> > > > > > >>>>
> > > > > > >>>> Gurer
> > > > > > >>>>
> > > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
> > > > > > >> criccomini@apache.org>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>>> I think FAB sounds like the right approach. Waiting to hear
> > > back
> > > > > with
> > > > > > >>>> notes
> > > > > > >>>>> on AirBNB H2 discussion to see if they want to take this
> up.
> > > > > > >>>>>
> > > > > > >>>>> @Gurer, any idea when this will happen?
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
> > > > bdbruin@gmail.com
> > > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> One downside I see from FAB is that is does not do
> Business
> > > Role
> > > > > > >>>> mapping
> > > > > > >>>>>> to FAB role. I would prefer to create groups in
> IPA/LDAP/AD
> > > and
> > > > > > >> have
> > > > > > >>>>> those
> > > > > > >>>>>> map to FAB roles instead of needing to manage that in FAB.
> > > > > > >>>>>>
> > > > > > >>>>>> B.
> > > > > > >>>>>>
> > > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <
> > bdbruin@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>> Hi Guys,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thanks for putting the thinking in! It is about time that
> > we
> > > > get
> > > > > > >>> this
> > > > > > >>>>>> moving.
> > > > > > >>>>>>>
> > > > > > >>>>>>> The design looks pretty sound. One can argue about the
> > > > different
> > > > > > >>>> roles
> > > > > > >>>>>> that are required, but that will be situation dependent I
> > > guess.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Implementation wise I would argue together with Max that
> > FAB
> > > > is a
> > > > > > >>>>> better
> > > > > > >>>>>> or best fit. The ER model that is being described is
> pretty
> > > > much a
> > > > > > >>> copy
> > > > > > >>>>> of
> > > > > > >>>>>> a normal security model. So a reimplementation of that is
> 1)
> > > > > > >>>> significant
> > > > > > >>>>>> duplication of effort and 2) bound to have bugs that have
> > been
> > > > > > >> solved
> > > > > > >>>> in
> > > > > > >>>>>> the other framework. Moreover, FAB does have integration
> out
> > > of
> > > > > the
> > > > > > >>> box
> > > > > > >>>>>> with some enterprisey systems like IPA, ActiveDirectory,
> and
> > > > LDAP.
> > > > > > >>>>>>>
> > > > > > >>>>>>> So while you argue that using FAB would increase the
> scope
> > of
> > > > the
> > > > > > >>>>>> proposal significantly, but I think that is not true.
> Using
> > > FAB
> > > > > > >> would
> > > > > > >>>>> allow
> > > > > > >>>>>> you to focus on what kind of out-of-the-box permission
> sets
> > > and
> > > > > > >> roles
> > > > > > >>>> we
> > > > > > >>>>>> would need and maybe address some issues that FAB lacks
> > (maybe
> > > > how
> > > > > > >> to
> > > > > > >>>>> deal
> > > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos,
> probably
> > > how
> > > > to
> > > > > > >>> deal
> > > > > > >>>>>> with API calls that are not CRUD). Implementation wise it
> > > > probably
> > > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s early
> POC
> > > as
> > > > an
> > > > > > >>>>> example
> > > > > > >>>>>> - we can slowly move over?
> > > > > > >>>>>>>
> > > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on
> > > Airflow
> > > > > > >>> coming
> > > > > > >>>>>> year. Improvement of Security, Enterprise Integration,
> > Revamp
> > > UI
> > > > > > >> are
> > > > > > >>> on
> > > > > > >>>>> the
> > > > > > >>>>>> todo list. However, this is not confirmed yet as business
> > > > > > >> priorities
> > > > > > >>>>> might
> > > > > > >>>>>> change.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Bolke.
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> > > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> @Dan:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG
> > > > > > >>> permission.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> @Max:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thanks for your response.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> The scope of my proposal was just to add RBAC security
> > > feature
> > > > > > >> to
> > > > > > >>>>>> Airflow
> > > > > > >>>>>>>> without replacing any existing frameworks.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I understand that adopting FAB would serve Airflow
> better
> > > > moving
> > > > > > >>>>>> forward,
> > > > > > >>>>>>>> however porting Airflow to using FAB significantly
> > increases
> > > > the
> > > > > > >>>> scope
> > > > > > >>>>>> of
> > > > > > >>>>>>>> the proposal and I don't have the time and expertise to
> > > carry
> > > > > > >> out
> > > > > > >>>> the
> > > > > > >>>>>> tasks
> > > > > > >>>>>>>> in the extended scope.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for Airflow
> > to
> > > > > > >>> migrate
> > > > > > >>>> to
> > > > > > >>>>>> FAB
> > > > > > >>>>>>>> this year?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> - Kalpesh
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
> > > > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> It would be nice to go with a framework for this. I did
> > > some
> > > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this
> > > > direction.
> > > > > > >> It
> > > > > > >>>>>> provides
> > > > > > >>>>>>>>> auth on different authentication backends out of the
> box
> > > > > > >> (oauth,
> > > > > > >>>>>> openid,
> > > > > > >>>>>>>>> ldap, registration, ...), generates perms for each view
> > > that
> > > > > > >> has
> > > > > > >>> an
> > > > > > >>>>>>>>> @has_access decorator, generates at set of perms for
> each
> > > ORM
> > > > > > >>> model
> > > > > > >>>>>> (show,
> > > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD
> views
> > > as
> > > > > > >> well
> > > > > > >>>> as
> > > > > > >>>>>> in the
> > > > > > >>>>>>>>> generated REST api that you get for free as a
> byprdoduct
> > of
> > > > > > >>>> deriving
> > > > > > >>>>>> FAB's
> > > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on
> top).
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I started a POC on FAB here a while ago:
> > > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at
> the
> > > > time
> > > > > > >> my
> > > > > > >>>>> main
> > > > > > >>>>>>>>> motivation was the free/instantaneous REST api.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I think FAB is a decent fit as the porting should be
> > fairly
> > > > > > >>>>>> straightforward
> > > > > > >>>>>>>>> (moving the flask views over and deprecating
> Flask-Admin
> > in
> > > > > > >> favor
> > > > > > >>>> of
> > > > > > >>>>>> FAB's
> > > > > > >>>>>>>>> crud) though there was a few blockers. From memory I
> > think
> > > > FAB
> > > > > > >>>> didn't
> > > > > > >>>>>> like
> > > > > > >>>>>>>>> the compound PKs we use in some of the Airflow models.
> > We'd
> > > > > > >> have
> > > > > > >>> to
> > > > > > >>>>>> either
> > > > > > >>>>>>>>> write a db migration script on the Airflow side, or add
> > > > support
> > > > > > >>> for
> > > > > > >>>>>>>>> compound keys to FAB (I recently became a maintainer of
> > the
> > > > > > >>>> project,
> > > > > > >>>>>> so I
> > > > > > >>>>>>>>> could help with that)
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> The only downside of FAB is that it's not as mature as
> > > > > > >> something
> > > > > > >>>> like
> > > > > > >>>>>>>>> Django, but porting to Django would surely be much more
> > > work.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Then there's the flask-security suite, but that looks
> > like
> > > a
> > > > > > >> bit
> > > > > > >>>> of a
> > > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose which
> we
> > > want
> > > > > > >> to
> > > > > > >>>> use.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Max
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> > > > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Looks good to me in general, thanks for putting this
> > > > together!
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> I think the ability to integrate with external RBAC
> > > systems
> > > > > > >> like
> > > > > > >>>>> LDAP
> > > > > > >>>>>> is
> > > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be decoupled
> > > with
> > > > > > >> the
> > > > > > >>>> RBAC
> > > > > > >>>>>>>>>> database wherever possible).
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> I wouldn't be too worried about the permissions about
> > > > > > >> refreshing
> > > > > > >>>>>> DAGs, as
> > > > > > >>>>>>>>>> far as I know this functionality is no longer required
> > > with
> > > > > > >> the
> > > > > > >>>> new
> > > > > > >>>>>>>>>> webservers which reload state periodically, and will
> > > > certainly
> > > > > > >>> be
> > > > > > >>>>>> removed
> > > > > > >>>>>>>>>> when we have a better DAG consistency story.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> I think it would also be good to think about this
> > > > > > >>>>>> proposal/implementation
> > > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g. when
> > > > > > >> webserver
> > > > > > >>>> hits
> > > > > > >>>>>> APIs
> > > > > > >>>>>>>>>> like /clear on behalf of users instead of running
> > commands
> > > > > > >>> against
> > > > > > >>>>> the
> > > > > > >>>>>>>>>> database directly).
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
> > > > > > >>>> bdbruin@gmail.com
> > > > > > >>>>>>
> > > > > > >>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give me
> a
> > > few
> > > > > > >>> days.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Sent from my iPhone
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> > > > > > >>>> criccomini@apache.org>
> > > > > > >>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Hey all,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time
> > > > thinking
> > > > > > >>>> about
> > > > > > >>>>>>>>> this,
> > > > > > >>>>>>>>>>> and
> > > > > > >>>>>>>>>>>> want to move forward with it, but want to make sure
> > > we're
> > > > > > >> all
> > > > > > >>> on
> > > > > > >>>>> the
> > > > > > >>>>>>>>>> same
> > > > > > >>>>>>>>>>>> page.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Cheers,
> > > > > > >>>>>>>>>>>> Chris
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
> > > > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Hello everyone,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a
> > > > built-in
> > > > > > >>> Role
> > > > > > >>>>>>>>> Based
> > > > > > >>>>>>>>>>>>> Access Control(RBAC) capability.  It does provide
> > very
> > > > > > >>> limited
> > > > > > >>>>>>>>>>>>> authorization capability by providing admin,
> > > > data_profiler,
> > > > > > >>> and
> > > > > > >>>>>> user
> > > > > > >>>>>>>>>>> roles.
> > > > > > >>>>>>>>>>>>> However, associating these roles to authenticated
> > > > > > >> identities
> > > > > > >>> is
> > > > > > >>>>> not
> > > > > > >>>>>>>>> a
> > > > > > >>>>>>>>>>>>> simple effort.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> To address this issue, I have created a design
> > proposal
> > > > for
> > > > > > >>>>>> building
> > > > > > >>>>>>>>>>> RBAC
> > > > > > >>>>>>>>>>>>> into Airflow and simplifying user access management
> > via
> > > > the
> > > > > > >>>>> Airflow
> > > > > > >>>>>>>>>> UI.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> The design proposal is located at
> > > > > > >> https://cwiki.apache.org/
> > > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much
> appreciated.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Thanks
> > > > > > >>>>>>>>>>>>> Kalpesh
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>>
> > > > > > >>> Joy Gao
> > > > > > >>> Software Engineer
> > > > > > >>> 350 Convention Way, Suite 200
> > > > > > >>> Redwood City, CA 94063
> > > > > > >>> Mobile:  669-224-9305
> > > > > > >>>
> > > > > > >>> Payments partner to the platform economy
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Role Based Access Control for Airflow UI

Posted by Dan Davydov <da...@airbnb.com.INVALID>.

Same as Alex, would be great to be able to remote in though I'm very
interested.

On Tue, Jul 25, 2017 at 1:37 PM, Alex Guziel <alex.guziel@airbnb.com.invalid
> wrote:

> Yeah, I could call in but I probably won't be able to come down that day.
>
> On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > Works for me! Dan said he might confcall in. Alex?
> >
> > Max
> >
> > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <criccomini@apache.org
> >
> > wrote:
> >
> > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room for
> 12
> > > people (and video conferencing) at WePay in this time slot. Should
> allow
> > > you to head home easily afterwards. :) That work for you guys?
> > >
> > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
> > > maximebeauchemin@gmail.com> wrote:
> > >
> > > > The week of the 31st sound good. Wednesday?
> > > >
> > > > About React we may not need a frontend lib like it (or at least not
> > just
> > > > yet). We can talk about it at the meeting.
> > > >
> > > > Max
> > > >
> > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <bd...@gmail.com>
> > > > wrote:
> > > >
> > > > > We avoid React for the same reasons as the ASF and use Polymer 2
> > > instead.
> > > > > Would that work?
> > > > >
> > > > > Bolke.
> > > > >
> > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <criccomini@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > Hey Max,
> > > > > >
> > > > > > Want to come down to WePay? We can set up a zoom for those that
> > want
> > > to
> > > > > > join online, and record it as well to post for the community.
> > > > > >
> > > > > > Since Joy is just getting started, and it looks like there's
> going
> > to
> > > > be
> > > > > a
> > > > > > K8s discussion next week, maybe we can shoot for the week after
> > (the
> > > > week
> > > > > > of the 31st of July)? Care to float a few times that week?
> > > > > >
> > > > > > Cheers,
> > > > > > Chris
> > > > > >
> > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
> > > > > > maximebeauchemin@gmail.com> wrote:
> > > > > >
> > > > > >> Sounds awesome, count me in!
> > > > > >>
> > > > > >> * check out the prototype in my fork, I went far enough to hit
> > some
> > > > > >> hurdles, try different workarounds. I hooked up the Airflow
> > > Bootstrap
> > > > > >> template too so that we feel at home in this new UI
> > > > > >> * using a single `id` field is a requirement for FAB that
> airflow
> > > > > doesn't
> > > > > >> respect (composite pks), either we add the feature to support
> that
> > > in
> > > > > FAB,
> > > > > >> or we align on the Airflow side and modify the models and add a
> > > > > migration
> > > > > >> script. This upgrade would require downtime and might be
> annoying
> > to
> > > > the
> > > > > >> Airflow community, but could help with db performance a bit
> > (smaller
> > > > > >> index)... I probably could be convinced either way but I'm
> leaning
> > > on
> > > > > >> improving FAB
> > > > > >> * I'm a maintainer for FAB so I can help get stuff through there
> > > > > >> * React is in limbo at the ASF for licensing reasons, so no
> React
> > at
> > > > > least
> > > > > >> for now
> > > > > >> * npm/webpack/ES6, javascript only in `.js` files
> > > > > >> * I vote for eslint + eslint-config-airbnb as a set of linting
> > rules
> > > > > for JS
> > > > > >> * Keep out of apache (for now), this new app ships as its own
> pypi
> > > > > package
> > > > > >> `airflow-webserver`, have a period of overlap (maintaining 2 web
> > > apps)
> > > > > >> before ripping out `airflow/www` from the core package
> > > > > >> * You need to get in touch with Marty Kausas, an intern at
> Airbnb
> > > > who's
> > > > > >> been working on a Flask blueprint for improved, more
> personalized
> > > > views
> > > > > on
> > > > > >> DAGs that we were planning on merging into the main branch
> > > eventually.
> > > > > Some
> > > > > >> of Marty's idea and code could be merged into this effort.
> > > > > >>
> > > > > >> These are ideas on how I would proceed personally on this but
> > > > definitely
> > > > > >> everything here is up for discussion.
> > > > > >>
> > > > > >> Let's meet physically at either WePay or Airbnb. Folks from the
> > > > > community,
> > > > > >> let us know on this thread if you want to be part of this
> effort,
> > > > we'll
> > > > > be
> > > > > >> happy to include you.
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Max
> > > > > >>
> > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com>
> wrote:
> > > > > >>
> > > > > >>> Hey everyone,
> > > > > >>>
> > > > > >>> I recently transferred to Data Infra team here at WePay to
> focus
> > on
> > > > > >>> Airflow-related initiatives.
> > > > > >>>
> > > > > >>> Given the RBAC design is mostly hashed out, I'm happy to get
> this
> > > > > feature
> > > > > >>> off the ground for Q3, starting with converting Airflow to Fab,
> > if
> > > > > there
> > > > > >>> are no objections.
> > > > > >>>
> > > > > >>> Cheers,
> > > > > >>> Joy
> > > > > >>>
> > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> > > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
> > > > > >>>
> > > > > >>>> Hey all,
> > > > > >>>>
> > > > > >>>> We talked about this internally. We would like to work on this
> > > > feature
> > > > > >>> but
> > > > > >>>> given the immediate priorities we are not going to be working
> on
> > > it
> > > > in
> > > > > >>> Q3.
> > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we can
> > work
> > > > on
> > > > > >> it
> > > > > >>>> late Q4 or Q12018.
> > > > > >>>>
> > > > > >>>> Cheers,
> > > > > >>>>
> > > > > >>>> Gurer
> > > > > >>>>
> > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
> > > > > >> criccomini@apache.org>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> I think FAB sounds like the right approach. Waiting to hear
> > back
> > > > with
> > > > > >>>> notes
> > > > > >>>>> on AirBNB H2 discussion to see if they want to take this up.
> > > > > >>>>>
> > > > > >>>>> @Gurer, any idea when this will happen?
> > > > > >>>>>
> > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
> > > bdbruin@gmail.com
> > > > >
> > > > > >>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> One downside I see from FAB is that is does not do Business
> > Role
> > > > > >>>> mapping
> > > > > >>>>>> to FAB role. I would prefer to create groups in IPA/LDAP/AD
> > and
> > > > > >> have
> > > > > >>>>> those
> > > > > >>>>>> map to FAB roles instead of needing to manage that in FAB.
> > > > > >>>>>>
> > > > > >>>>>> B.
> > > > > >>>>>>
> > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <
> bdbruin@gmail.com>
> > > > > >>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>> Hi Guys,
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks for putting the thinking in! It is about time that
> we
> > > get
> > > > > >>> this
> > > > > >>>>>> moving.
> > > > > >>>>>>>
> > > > > >>>>>>> The design looks pretty sound. One can argue about the
> > > different
> > > > > >>>> roles
> > > > > >>>>>> that are required, but that will be situation dependent I
> > guess.
> > > > > >>>>>>>
> > > > > >>>>>>> Implementation wise I would argue together with Max that
> FAB
> > > is a
> > > > > >>>>> better
> > > > > >>>>>> or best fit. The ER model that is being described is pretty
> > > much a
> > > > > >>> copy
> > > > > >>>>> of
> > > > > >>>>>> a normal security model. So a reimplementation of that is 1)
> > > > > >>>> significant
> > > > > >>>>>> duplication of effort and 2) bound to have bugs that have
> been
> > > > > >> solved
> > > > > >>>> in
> > > > > >>>>>> the other framework. Moreover, FAB does have integration out
> > of
> > > > the
> > > > > >>> box
> > > > > >>>>>> with some enterprisey systems like IPA, ActiveDirectory, and
> > > LDAP.
> > > > > >>>>>>>
> > > > > >>>>>>> So while you argue that using FAB would increase the scope
> of
> > > the
> > > > > >>>>>> proposal significantly, but I think that is not true. Using
> > FAB
> > > > > >> would
> > > > > >>>>> allow
> > > > > >>>>>> you to focus on what kind of out-of-the-box permission sets
> > and
> > > > > >> roles
> > > > > >>>> we
> > > > > >>>>>> would need and maybe address some issues that FAB lacks
> (maybe
> > > how
> > > > > >> to
> > > > > >>>>> deal
> > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos, probably
> > how
> > > to
> > > > > >>> deal
> > > > > >>>>>> with API calls that are not CRUD). Implementation wise it
> > > probably
> > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s early POC
> > as
> > > an
> > > > > >>>>> example
> > > > > >>>>>> - we can slowly move over?
> > > > > >>>>>>>
> > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on
> > Airflow
> > > > > >>> coming
> > > > > >>>>>> year. Improvement of Security, Enterprise Integration,
> Revamp
> > UI
> > > > > >> are
> > > > > >>> on
> > > > > >>>>> the
> > > > > >>>>>> todo list. However, this is not confirmed yet as business
> > > > > >> priorities
> > > > > >>>>> might
> > > > > >>>>>> change.
> > > > > >>>>>>>
> > > > > >>>>>>> Bolke.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> @Dan:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG
> > > > > >>> permission.
> > > > > >>>>>>>>
> > > > > >>>>>>>> @Max:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thanks for your response.
> > > > > >>>>>>>>
> > > > > >>>>>>>> The scope of my proposal was just to add RBAC security
> > feature
> > > > > >> to
> > > > > >>>>>> Airflow
> > > > > >>>>>>>> without replacing any existing frameworks.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I understand that adopting FAB would serve Airflow better
> > > moving
> > > > > >>>>>> forward,
> > > > > >>>>>>>> however porting Airflow to using FAB significantly
> increases
> > > the
> > > > > >>>> scope
> > > > > >>>>>> of
> > > > > >>>>>>>> the proposal and I don't have the time and expertise to
> > carry
> > > > > >> out
> > > > > >>>> the
> > > > > >>>>>> tasks
> > > > > >>>>>>>> in the extended scope.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for Airflow
> to
> > > > > >>> migrate
> > > > > >>>> to
> > > > > >>>>>> FAB
> > > > > >>>>>>>> this year?
> > > > > >>>>>>>>
> > > > > >>>>>>>> - Kalpesh
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
> > > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> It would be nice to go with a framework for this. I did
> > some
> > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this
> > > direction.
> > > > > >> It
> > > > > >>>>>> provides
> > > > > >>>>>>>>> auth on different authentication backends out of the box
> > > > > >> (oauth,
> > > > > >>>>>> openid,
> > > > > >>>>>>>>> ldap, registration, ...), generates perms for each view
> > that
> > > > > >> has
> > > > > >>> an
> > > > > >>>>>>>>> @has_access decorator, generates at set of perms for each
> > ORM
> > > > > >>> model
> > > > > >>>>>> (show,
> > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD views
> > as
> > > > > >> well
> > > > > >>>> as
> > > > > >>>>>> in the
> > > > > >>>>>>>>> generated REST api that you get for free as a byprdoduct
> of
> > > > > >>>> deriving
> > > > > >>>>>> FAB's
> > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on top).
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I started a POC on FAB here a while ago:
> > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at the
> > > time
> > > > > >> my
> > > > > >>>>> main
> > > > > >>>>>>>>> motivation was the free/instantaneous REST api.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I think FAB is a decent fit as the porting should be
> fairly
> > > > > >>>>>> straightforward
> > > > > >>>>>>>>> (moving the flask views over and deprecating Flask-Admin
> in
> > > > > >> favor
> > > > > >>>> of
> > > > > >>>>>> FAB's
> > > > > >>>>>>>>> crud) though there was a few blockers. From memory I
> think
> > > FAB
> > > > > >>>> didn't
> > > > > >>>>>> like
> > > > > >>>>>>>>> the compound PKs we use in some of the Airflow models.
> We'd
> > > > > >> have
> > > > > >>> to
> > > > > >>>>>> either
> > > > > >>>>>>>>> write a db migration script on the Airflow side, or add
> > > support
> > > > > >>> for
> > > > > >>>>>>>>> compound keys to FAB (I recently became a maintainer of
> the
> > > > > >>>> project,
> > > > > >>>>>> so I
> > > > > >>>>>>>>> could help with that)
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> The only downside of FAB is that it's not as mature as
> > > > > >> something
> > > > > >>>> like
> > > > > >>>>>>>>> Django, but porting to Django would surely be much more
> > work.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Then there's the flask-security suite, but that looks
> like
> > a
> > > > > >> bit
> > > > > >>>> of a
> > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose which we
> > want
> > > > > >> to
> > > > > >>>> use.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Max
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> > > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Looks good to me in general, thanks for putting this
> > > together!
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> I think the ability to integrate with external RBAC
> > systems
> > > > > >> like
> > > > > >>>>> LDAP
> > > > > >>>>>> is
> > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be decoupled
> > with
> > > > > >> the
> > > > > >>>> RBAC
> > > > > >>>>>>>>>> database wherever possible).
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> I wouldn't be too worried about the permissions about
> > > > > >> refreshing
> > > > > >>>>>> DAGs, as
> > > > > >>>>>>>>>> far as I know this functionality is no longer required
> > with
> > > > > >> the
> > > > > >>>> new
> > > > > >>>>>>>>>> webservers which reload state periodically, and will
> > > certainly
> > > > > >>> be
> > > > > >>>>>> removed
> > > > > >>>>>>>>>> when we have a better DAG consistency story.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> I think it would also be good to think about this
> > > > > >>>>>> proposal/implementation
> > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g. when
> > > > > >> webserver
> > > > > >>>> hits
> > > > > >>>>>> APIs
> > > > > >>>>>>>>>> like /clear on behalf of users instead of running
> commands
> > > > > >>> against
> > > > > >>>>> the
> > > > > >>>>>>>>>> database directly).
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
> > > > > >>>> bdbruin@gmail.com
> > > > > >>>>>>
> > > > > >>>>>>>>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give me a
> > few
> > > > > >>> days.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Sent from my iPhone
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> > > > > >>>> criccomini@apache.org>
> > > > > >>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Hey all,
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time
> > > thinking
> > > > > >>>> about
> > > > > >>>>>>>>> this,
> > > > > >>>>>>>>>>> and
> > > > > >>>>>>>>>>>> want to move forward with it, but want to make sure
> > we're
> > > > > >> all
> > > > > >>> on
> > > > > >>>>> the
> > > > > >>>>>>>>>> same
> > > > > >>>>>>>>>>>> page.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Cheers,
> > > > > >>>>>>>>>>>> Chris
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
> > > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Hello everyone,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a
> > > built-in
> > > > > >>> Role
> > > > > >>>>>>>>> Based
> > > > > >>>>>>>>>>>>> Access Control(RBAC) capability.  It does provide
> very
> > > > > >>> limited
> > > > > >>>>>>>>>>>>> authorization capability by providing admin,
> > > data_profiler,
> > > > > >>> and
> > > > > >>>>>> user
> > > > > >>>>>>>>>>> roles.
> > > > > >>>>>>>>>>>>> However, associating these roles to authenticated
> > > > > >> identities
> > > > > >>> is
> > > > > >>>>> not
> > > > > >>>>>>>>> a
> > > > > >>>>>>>>>>>>> simple effort.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> To address this issue, I have created a design
> proposal
> > > for
> > > > > >>>>>> building
> > > > > >>>>>>>>>>> RBAC
> > > > > >>>>>>>>>>>>> into Airflow and simplifying user access management
> via
> > > the
> > > > > >>>>> Airflow
> > > > > >>>>>>>>>> UI.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> The design proposal is located at
> > > > > >> https://cwiki.apache.org/
> > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much appreciated.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Thanks
> > > > > >>>>>>>>>>>>> Kalpesh
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>>
> > > > > >>> Joy Gao
> > > > > >>> Software Engineer
> > > > > >>> 350 Convention Way, Suite 200
> > > > > >>> Redwood City, CA 94063
> > > > > >>> Mobile:  669-224-9305
> > > > > >>>
> > > > > >>> Payments partner to the platform economy
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Role Based Access Control for Airflow UI

Posted by Alex Guziel <al...@airbnb.com.INVALID>.

Yeah, I could call in but I probably won't be able to come down that day.

On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> Works for me! Dan said he might confcall in. Alex?
>
> Max
>
> On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room for 12
> > people (and video conferencing) at WePay in this time slot. Should allow
> > you to head home easily afterwards. :) That work for you guys?
> >
> > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> >
> > > The week of the 31st sound good. Wednesday?
> > >
> > > About React we may not need a frontend lib like it (or at least not
> just
> > > yet). We can talk about it at the meeting.
> > >
> > > Max
> > >
> > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <bd...@gmail.com>
> > > wrote:
> > >
> > > > We avoid React for the same reasons as the ASF and use Polymer 2
> > instead.
> > > > Would that work?
> > > >
> > > > Bolke.
> > > >
> > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <cr...@apache.org>
> > > wrote:
> > > > >
> > > > > Hey Max,
> > > > >
> > > > > Want to come down to WePay? We can set up a zoom for those that
> want
> > to
> > > > > join online, and record it as well to post for the community.
> > > > >
> > > > > Since Joy is just getting started, and it looks like there's going
> to
> > > be
> > > > a
> > > > > K8s discussion next week, maybe we can shoot for the week after
> (the
> > > week
> > > > > of the 31st of July)? Care to float a few times that week?
> > > > >
> > > > > Cheers,
> > > > > Chris
> > > > >
> > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
> > > > > maximebeauchemin@gmail.com> wrote:
> > > > >
> > > > >> Sounds awesome, count me in!
> > > > >>
> > > > >> * check out the prototype in my fork, I went far enough to hit
> some
> > > > >> hurdles, try different workarounds. I hooked up the Airflow
> > Bootstrap
> > > > >> template too so that we feel at home in this new UI
> > > > >> * using a single `id` field is a requirement for FAB that airflow
> > > > doesn't
> > > > >> respect (composite pks), either we add the feature to support that
> > in
> > > > FAB,
> > > > >> or we align on the Airflow side and modify the models and add a
> > > > migration
> > > > >> script. This upgrade would require downtime and might be annoying
> to
> > > the
> > > > >> Airflow community, but could help with db performance a bit
> (smaller
> > > > >> index)... I probably could be convinced either way but I'm leaning
> > on
> > > > >> improving FAB
> > > > >> * I'm a maintainer for FAB so I can help get stuff through there
> > > > >> * React is in limbo at the ASF for licensing reasons, so no React
> at
> > > > least
> > > > >> for now
> > > > >> * npm/webpack/ES6, javascript only in `.js` files
> > > > >> * I vote for eslint + eslint-config-airbnb as a set of linting
> rules
> > > > for JS
> > > > >> * Keep out of apache (for now), this new app ships as its own pypi
> > > > package
> > > > >> `airflow-webserver`, have a period of overlap (maintaining 2 web
> > apps)
> > > > >> before ripping out `airflow/www` from the core package
> > > > >> * You need to get in touch with Marty Kausas, an intern at Airbnb
> > > who's
> > > > >> been working on a Flask blueprint for improved, more personalized
> > > views
> > > > on
> > > > >> DAGs that we were planning on merging into the main branch
> > eventually.
> > > > Some
> > > > >> of Marty's idea and code could be merged into this effort.
> > > > >>
> > > > >> These are ideas on how I would proceed personally on this but
> > > definitely
> > > > >> everything here is up for discussion.
> > > > >>
> > > > >> Let's meet physically at either WePay or Airbnb. Folks from the
> > > > community,
> > > > >> let us know on this thread if you want to be part of this effort,
> > > we'll
> > > > be
> > > > >> happy to include you.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Max
> > > > >>
> > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com> wrote:
> > > > >>
> > > > >>> Hey everyone,
> > > > >>>
> > > > >>> I recently transferred to Data Infra team here at WePay to focus
> on
> > > > >>> Airflow-related initiatives.
> > > > >>>
> > > > >>> Given the RBAC design is mostly hashed out, I'm happy to get this
> > > > feature
> > > > >>> off the ground for Q3, starting with converting Airflow to Fab,
> if
> > > > there
> > > > >>> are no objections.
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Joy
> > > > >>>
> > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
> > > > >>>
> > > > >>>> Hey all,
> > > > >>>>
> > > > >>>> We talked about this internally. We would like to work on this
> > > feature
> > > > >>> but
> > > > >>>> given the immediate priorities we are not going to be working on
> > it
> > > in
> > > > >>> Q3.
> > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we can
> work
> > > on
> > > > >> it
> > > > >>>> late Q4 or Q12018.
> > > > >>>>
> > > > >>>> Cheers,
> > > > >>>>
> > > > >>>> Gurer
> > > > >>>>
> > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
> > > > >> criccomini@apache.org>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> I think FAB sounds like the right approach. Waiting to hear
> back
> > > with
> > > > >>>> notes
> > > > >>>>> on AirBNB H2 discussion to see if they want to take this up.
> > > > >>>>>
> > > > >>>>> @Gurer, any idea when this will happen?
> > > > >>>>>
> > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
> > bdbruin@gmail.com
> > > >
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>>> One downside I see from FAB is that is does not do Business
> Role
> > > > >>>> mapping
> > > > >>>>>> to FAB role. I would prefer to create groups in IPA/LDAP/AD
> and
> > > > >> have
> > > > >>>>> those
> > > > >>>>>> map to FAB roles instead of needing to manage that in FAB.
> > > > >>>>>>
> > > > >>>>>> B.
> > > > >>>>>>
> > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <bd...@gmail.com>
> > > > >>> wrote:
> > > > >>>>>>>
> > > > >>>>>>> Hi Guys,
> > > > >>>>>>>
> > > > >>>>>>> Thanks for putting the thinking in! It is about time that we
> > get
> > > > >>> this
> > > > >>>>>> moving.
> > > > >>>>>>>
> > > > >>>>>>> The design looks pretty sound. One can argue about the
> > different
> > > > >>>> roles
> > > > >>>>>> that are required, but that will be situation dependent I
> guess.
> > > > >>>>>>>
> > > > >>>>>>> Implementation wise I would argue together with Max that FAB
> > is a
> > > > >>>>> better
> > > > >>>>>> or best fit. The ER model that is being described is pretty
> > much a
> > > > >>> copy
> > > > >>>>> of
> > > > >>>>>> a normal security model. So a reimplementation of that is 1)
> > > > >>>> significant
> > > > >>>>>> duplication of effort and 2) bound to have bugs that have been
> > > > >> solved
> > > > >>>> in
> > > > >>>>>> the other framework. Moreover, FAB does have integration out
> of
> > > the
> > > > >>> box
> > > > >>>>>> with some enterprisey systems like IPA, ActiveDirectory, and
> > LDAP.
> > > > >>>>>>>
> > > > >>>>>>> So while you argue that using FAB would increase the scope of
> > the
> > > > >>>>>> proposal significantly, but I think that is not true. Using
> FAB
> > > > >> would
> > > > >>>>> allow
> > > > >>>>>> you to focus on what kind of out-of-the-box permission sets
> and
> > > > >> roles
> > > > >>>> we
> > > > >>>>>> would need and maybe address some issues that FAB lacks (maybe
> > how
> > > > >> to
> > > > >>>>> deal
> > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos, probably
> how
> > to
> > > > >>> deal
> > > > >>>>>> with API calls that are not CRUD). Implementation wise it
> > probably
> > > > >>>>>> simplifies what we need to do. Maybe - using Max’s early POC
> as
> > an
> > > > >>>>> example
> > > > >>>>>> - we can slowly move over?
> > > > >>>>>>>
> > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on
> Airflow
> > > > >>> coming
> > > > >>>>>> year. Improvement of Security, Enterprise Integration, Revamp
> UI
> > > > >> are
> > > > >>> on
> > > > >>>>> the
> > > > >>>>>> todo list. However, this is not confirmed yet as business
> > > > >> priorities
> > > > >>>>> might
> > > > >>>>>> change.
> > > > >>>>>>>
> > > > >>>>>>> Bolke.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> @Dan:
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG
> > > > >>> permission.
> > > > >>>>>>>>
> > > > >>>>>>>> @Max:
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks for your response.
> > > > >>>>>>>>
> > > > >>>>>>>> The scope of my proposal was just to add RBAC security
> feature
> > > > >> to
> > > > >>>>>> Airflow
> > > > >>>>>>>> without replacing any existing frameworks.
> > > > >>>>>>>>
> > > > >>>>>>>> I understand that adopting FAB would serve Airflow better
> > moving
> > > > >>>>>> forward,
> > > > >>>>>>>> however porting Airflow to using FAB significantly increases
> > the
> > > > >>>> scope
> > > > >>>>>> of
> > > > >>>>>>>> the proposal and I don't have the time and expertise to
> carry
> > > > >> out
> > > > >>>> the
> > > > >>>>>> tasks
> > > > >>>>>>>> in the extended scope.
> > > > >>>>>>>>
> > > > >>>>>>>> Hence, I'm curious to know if there's a plan for Airflow to
> > > > >>> migrate
> > > > >>>> to
> > > > >>>>>> FAB
> > > > >>>>>>>> this year?
> > > > >>>>>>>>
> > > > >>>>>>>> - Kalpesh
> > > > >>>>>>>>
> > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
> > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> It would be nice to go with a framework for this. I did
> some
> > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this
> > direction.
> > > > >> It
> > > > >>>>>> provides
> > > > >>>>>>>>> auth on different authentication backends out of the box
> > > > >> (oauth,
> > > > >>>>>> openid,
> > > > >>>>>>>>> ldap, registration, ...), generates perms for each view
> that
> > > > >> has
> > > > >>> an
> > > > >>>>>>>>> @has_access decorator, generates at set of perms for each
> ORM
> > > > >>> model
> > > > >>>>>> (show,
> > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD views
> as
> > > > >> well
> > > > >>>> as
> > > > >>>>>> in the
> > > > >>>>>>>>> generated REST api that you get for free as a byprdoduct of
> > > > >>>> deriving
> > > > >>>>>> FAB's
> > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on top).
> > > > >>>>>>>>>
> > > > >>>>>>>>> I started a POC on FAB here a while ago:
> > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at the
> > time
> > > > >> my
> > > > >>>>> main
> > > > >>>>>>>>> motivation was the free/instantaneous REST api.
> > > > >>>>>>>>>
> > > > >>>>>>>>> I think FAB is a decent fit as the porting should be fairly
> > > > >>>>>> straightforward
> > > > >>>>>>>>> (moving the flask views over and deprecating Flask-Admin in
> > > > >> favor
> > > > >>>> of
> > > > >>>>>> FAB's
> > > > >>>>>>>>> crud) though there was a few blockers. From memory I think
> > FAB
> > > > >>>> didn't
> > > > >>>>>> like
> > > > >>>>>>>>> the compound PKs we use in some of the Airflow models. We'd
> > > > >> have
> > > > >>> to
> > > > >>>>>> either
> > > > >>>>>>>>> write a db migration script on the Airflow side, or add
> > support
> > > > >>> for
> > > > >>>>>>>>> compound keys to FAB (I recently became a maintainer of the
> > > > >>>> project,
> > > > >>>>>> so I
> > > > >>>>>>>>> could help with that)
> > > > >>>>>>>>>
> > > > >>>>>>>>> The only downside of FAB is that it's not as mature as
> > > > >> something
> > > > >>>> like
> > > > >>>>>>>>> Django, but porting to Django would surely be much more
> work.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Then there's the flask-security suite, but that looks like
> a
> > > > >> bit
> > > > >>>> of a
> > > > >>>>>>>>> patchwork to me, I guess we can pick and choose which we
> want
> > > > >> to
> > > > >>>> use.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Max
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Looks good to me in general, thanks for putting this
> > together!
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I think the ability to integrate with external RBAC
> systems
> > > > >> like
> > > > >>>>> LDAP
> > > > >>>>>> is
> > > > >>>>>>>>>> important (i.e. the Airflow DB should not be decoupled
> with
> > > > >> the
> > > > >>>> RBAC
> > > > >>>>>>>>>> database wherever possible).
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I wouldn't be too worried about the permissions about
> > > > >> refreshing
> > > > >>>>>> DAGs, as
> > > > >>>>>>>>>> far as I know this functionality is no longer required
> with
> > > > >> the
> > > > >>>> new
> > > > >>>>>>>>>> webservers which reload state periodically, and will
> > certainly
> > > > >>> be
> > > > >>>>>> removed
> > > > >>>>>>>>>> when we have a better DAG consistency story.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I think it would also be good to think about this
> > > > >>>>>> proposal/implementation
> > > > >>>>>>>>>> and how it applied in the API-driven world (e.g. when
> > > > >> webserver
> > > > >>>> hits
> > > > >>>>>> APIs
> > > > >>>>>>>>>> like /clear on behalf of users instead of running commands
> > > > >>> against
> > > > >>>>> the
> > > > >>>>>>>>>> database directly).
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
> > > > >>>> bdbruin@gmail.com
> > > > >>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give me a
> few
> > > > >>> days.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Sent from my iPhone
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> > > > >>>> criccomini@apache.org>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Hey all,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time
> > thinking
> > > > >>>> about
> > > > >>>>>>>>> this,
> > > > >>>>>>>>>>> and
> > > > >>>>>>>>>>>> want to move forward with it, but want to make sure
> we're
> > > > >> all
> > > > >>> on
> > > > >>>>> the
> > > > >>>>>>>>>> same
> > > > >>>>>>>>>>>> page.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Cheers,
> > > > >>>>>>>>>>>> Chris
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
> > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Hello everyone,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a
> > built-in
> > > > >>> Role
> > > > >>>>>>>>> Based
> > > > >>>>>>>>>>>>> Access Control(RBAC) capability.  It does provide very
> > > > >>> limited
> > > > >>>>>>>>>>>>> authorization capability by providing admin,
> > data_profiler,
> > > > >>> and
> > > > >>>>>> user
> > > > >>>>>>>>>>> roles.
> > > > >>>>>>>>>>>>> However, associating these roles to authenticated
> > > > >> identities
> > > > >>> is
> > > > >>>>> not
> > > > >>>>>>>>> a
> > > > >>>>>>>>>>>>> simple effort.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> To address this issue, I have created a design proposal
> > for
> > > > >>>>>> building
> > > > >>>>>>>>>>> RBAC
> > > > >>>>>>>>>>>>> into Airflow and simplifying user access management via
> > the
> > > > >>>>> Airflow
> > > > >>>>>>>>>> UI.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> The design proposal is located at
> > > > >> https://cwiki.apache.org/
> > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Any comments/questions/feedback are much appreciated.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>>> Kalpesh
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>>
> > > > >>> Joy Gao
> > > > >>> Software Engineer
> > > > >>> 350 Convention Way, Suite 200
> > > > >>> Redwood City, CA 94063
> > > > >>> Mobile:  669-224-9305
> > > > >>>
> > > > >>> Payments partner to the platform economy
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: Role Based Access Control for Airflow UI

Posted by Maxime Beauchemin <ma...@gmail.com>.

Works for me! Dan said he might confcall in. Alex?

Max

On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <cr...@apache.org>
wrote:

> Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room for 12
> people (and video conferencing) at WePay in this time slot. Should allow
> you to head home easily afterwards. :) That work for you guys?
>
> On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > The week of the 31st sound good. Wednesday?
> >
> > About React we may not need a frontend lib like it (or at least not just
> > yet). We can talk about it at the meeting.
> >
> > Max
> >
> > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> >
> > > We avoid React for the same reasons as the ASF and use Polymer 2
> instead.
> > > Would that work?
> > >
> > > Bolke.
> > >
> > > > On 20 Jul 2017, at 19:35, Chris Riccomini <cr...@apache.org>
> > wrote:
> > > >
> > > > Hey Max,
> > > >
> > > > Want to come down to WePay? We can set up a zoom for those that want
> to
> > > > join online, and record it as well to post for the community.
> > > >
> > > > Since Joy is just getting started, and it looks like there's going to
> > be
> > > a
> > > > K8s discussion next week, maybe we can shoot for the week after (the
> > week
> > > > of the 31st of July)? Care to float a few times that week?
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
> > > > maximebeauchemin@gmail.com> wrote:
> > > >
> > > >> Sounds awesome, count me in!
> > > >>
> > > >> * check out the prototype in my fork, I went far enough to hit some
> > > >> hurdles, try different workarounds. I hooked up the Airflow
> Bootstrap
> > > >> template too so that we feel at home in this new UI
> > > >> * using a single `id` field is a requirement for FAB that airflow
> > > doesn't
> > > >> respect (composite pks), either we add the feature to support that
> in
> > > FAB,
> > > >> or we align on the Airflow side and modify the models and add a
> > > migration
> > > >> script. This upgrade would require downtime and might be annoying to
> > the
> > > >> Airflow community, but could help with db performance a bit (smaller
> > > >> index)... I probably could be convinced either way but I'm leaning
> on
> > > >> improving FAB
> > > >> * I'm a maintainer for FAB so I can help get stuff through there
> > > >> * React is in limbo at the ASF for licensing reasons, so no React at
> > > least
> > > >> for now
> > > >> * npm/webpack/ES6, javascript only in `.js` files
> > > >> * I vote for eslint + eslint-config-airbnb as a set of linting rules
> > > for JS
> > > >> * Keep out of apache (for now), this new app ships as its own pypi
> > > package
> > > >> `airflow-webserver`, have a period of overlap (maintaining 2 web
> apps)
> > > >> before ripping out `airflow/www` from the core package
> > > >> * You need to get in touch with Marty Kausas, an intern at Airbnb
> > who's
> > > >> been working on a Flask blueprint for improved, more personalized
> > views
> > > on
> > > >> DAGs that we were planning on merging into the main branch
> eventually.
> > > Some
> > > >> of Marty's idea and code could be merged into this effort.
> > > >>
> > > >> These are ideas on how I would proceed personally on this but
> > definitely
> > > >> everything here is up for discussion.
> > > >>
> > > >> Let's meet physically at either WePay or Airbnb. Folks from the
> > > community,
> > > >> let us know on this thread if you want to be part of this effort,
> > we'll
> > > be
> > > >> happy to include you.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Max
> > > >>
> > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com> wrote:
> > > >>
> > > >>> Hey everyone,
> > > >>>
> > > >>> I recently transferred to Data Infra team here at WePay to focus on
> > > >>> Airflow-related initiatives.
> > > >>>
> > > >>> Given the RBAC design is mostly hashed out, I'm happy to get this
> > > feature
> > > >>> off the ground for Q3, starting with converting Airflow to Fab, if
> > > there
> > > >>> are no objections.
> > > >>>
> > > >>> Cheers,
> > > >>> Joy
> > > >>>
> > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
> > > >>>
> > > >>>> Hey all,
> > > >>>>
> > > >>>> We talked about this internally. We would like to work on this
> > feature
> > > >>> but
> > > >>>> given the immediate priorities we are not going to be working on
> it
> > in
> > > >>> Q3.
> > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we can work
> > on
> > > >> it
> > > >>>> late Q4 or Q12018.
> > > >>>>
> > > >>>> Cheers,
> > > >>>>
> > > >>>> Gurer
> > > >>>>
> > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
> > > >> criccomini@apache.org>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> I think FAB sounds like the right approach. Waiting to hear back
> > with
> > > >>>> notes
> > > >>>>> on AirBNB H2 discussion to see if they want to take this up.
> > > >>>>>
> > > >>>>> @Gurer, any idea when this will happen?
> > > >>>>>
> > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
> bdbruin@gmail.com
> > >
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> One downside I see from FAB is that is does not do Business Role
> > > >>>> mapping
> > > >>>>>> to FAB role. I would prefer to create groups in IPA/LDAP/AD and
> > > >> have
> > > >>>>> those
> > > >>>>>> map to FAB roles instead of needing to manage that in FAB.
> > > >>>>>>
> > > >>>>>> B.
> > > >>>>>>
> > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <bd...@gmail.com>
> > > >>> wrote:
> > > >>>>>>>
> > > >>>>>>> Hi Guys,
> > > >>>>>>>
> > > >>>>>>> Thanks for putting the thinking in! It is about time that we
> get
> > > >>> this
> > > >>>>>> moving.
> > > >>>>>>>
> > > >>>>>>> The design looks pretty sound. One can argue about the
> different
> > > >>>> roles
> > > >>>>>> that are required, but that will be situation dependent I guess.
> > > >>>>>>>
> > > >>>>>>> Implementation wise I would argue together with Max that FAB
> is a
> > > >>>>> better
> > > >>>>>> or best fit. The ER model that is being described is pretty
> much a
> > > >>> copy
> > > >>>>> of
> > > >>>>>> a normal security model. So a reimplementation of that is 1)
> > > >>>> significant
> > > >>>>>> duplication of effort and 2) bound to have bugs that have been
> > > >> solved
> > > >>>> in
> > > >>>>>> the other framework. Moreover, FAB does have integration out of
> > the
> > > >>> box
> > > >>>>>> with some enterprisey systems like IPA, ActiveDirectory, and
> LDAP.
> > > >>>>>>>
> > > >>>>>>> So while you argue that using FAB would increase the scope of
> the
> > > >>>>>> proposal significantly, but I think that is not true. Using FAB
> > > >> would
> > > >>>>> allow
> > > >>>>>> you to focus on what kind of out-of-the-box permission sets and
> > > >> roles
> > > >>>> we
> > > >>>>>> would need and maybe address some issues that FAB lacks (maybe
> how
> > > >> to
> > > >>>>> deal
> > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos, probably how
> to
> > > >>> deal
> > > >>>>>> with API calls that are not CRUD). Implementation wise it
> probably
> > > >>>>>> simplifies what we need to do. Maybe - using Max’s early POC as
> an
> > > >>>>> example
> > > >>>>>> - we can slowly move over?
> > > >>>>>>>
> > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on Airflow
> > > >>> coming
> > > >>>>>> year. Improvement of Security, Enterprise Integration, Revamp UI
> > > >> are
> > > >>> on
> > > >>>>> the
> > > >>>>>> todo list. However, this is not confirmed yet as business
> > > >> priorities
> > > >>>>> might
> > > >>>>>> change.
> > > >>>>>>>
> > > >>>>>>> Bolke.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > >>>>>>>>
> > > >>>>>>>> @Dan:
> > > >>>>>>>>
> > > >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG
> > > >>> permission.
> > > >>>>>>>>
> > > >>>>>>>> @Max:
> > > >>>>>>>>
> > > >>>>>>>> Thanks for your response.
> > > >>>>>>>>
> > > >>>>>>>> The scope of my proposal was just to add RBAC security feature
> > > >> to
> > > >>>>>> Airflow
> > > >>>>>>>> without replacing any existing frameworks.
> > > >>>>>>>>
> > > >>>>>>>> I understand that adopting FAB would serve Airflow better
> moving
> > > >>>>>> forward,
> > > >>>>>>>> however porting Airflow to using FAB significantly increases
> the
> > > >>>> scope
> > > >>>>>> of
> > > >>>>>>>> the proposal and I don't have the time and expertise to carry
> > > >> out
> > > >>>> the
> > > >>>>>> tasks
> > > >>>>>>>> in the extended scope.
> > > >>>>>>>>
> > > >>>>>>>> Hence, I'm curious to know if there's a plan for Airflow to
> > > >>> migrate
> > > >>>> to
> > > >>>>>> FAB
> > > >>>>>>>> this year?
> > > >>>>>>>>
> > > >>>>>>>> - Kalpesh
> > > >>>>>>>>
> > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
> > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> It would be nice to go with a framework for this. I did some
> > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this
> direction.
> > > >> It
> > > >>>>>> provides
> > > >>>>>>>>> auth on different authentication backends out of the box
> > > >> (oauth,
> > > >>>>>> openid,
> > > >>>>>>>>> ldap, registration, ...), generates perms for each view that
> > > >> has
> > > >>> an
> > > >>>>>>>>> @has_access decorator, generates at set of perms for each ORM
> > > >>> model
> > > >>>>>> (show,
> > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD views as
> > > >> well
> > > >>>> as
> > > >>>>>> in the
> > > >>>>>>>>> generated REST api that you get for free as a byprdoduct of
> > > >>>> deriving
> > > >>>>>> FAB's
> > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on top).
> > > >>>>>>>>>
> > > >>>>>>>>> I started a POC on FAB here a while ago:
> > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at the
> time
> > > >> my
> > > >>>>> main
> > > >>>>>>>>> motivation was the free/instantaneous REST api.
> > > >>>>>>>>>
> > > >>>>>>>>> I think FAB is a decent fit as the porting should be fairly
> > > >>>>>> straightforward
> > > >>>>>>>>> (moving the flask views over and deprecating Flask-Admin in
> > > >> favor
> > > >>>> of
> > > >>>>>> FAB's
> > > >>>>>>>>> crud) though there was a few blockers. From memory I think
> FAB
> > > >>>> didn't
> > > >>>>>> like
> > > >>>>>>>>> the compound PKs we use in some of the Airflow models. We'd
> > > >> have
> > > >>> to
> > > >>>>>> either
> > > >>>>>>>>> write a db migration script on the Airflow side, or add
> support
> > > >>> for
> > > >>>>>>>>> compound keys to FAB (I recently became a maintainer of the
> > > >>>> project,
> > > >>>>>> so I
> > > >>>>>>>>> could help with that)
> > > >>>>>>>>>
> > > >>>>>>>>> The only downside of FAB is that it's not as mature as
> > > >> something
> > > >>>> like
> > > >>>>>>>>> Django, but porting to Django would surely be much more work.
> > > >>>>>>>>>
> > > >>>>>>>>> Then there's the flask-security suite, but that looks like a
> > > >> bit
> > > >>>> of a
> > > >>>>>>>>> patchwork to me, I guess we can pick and choose which we want
> > > >> to
> > > >>>> use.
> > > >>>>>>>>>
> > > >>>>>>>>> Max
> > > >>>>>>>>>
> > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Looks good to me in general, thanks for putting this
> together!
> > > >>>>>>>>>>
> > > >>>>>>>>>> I think the ability to integrate with external RBAC systems
> > > >> like
> > > >>>>> LDAP
> > > >>>>>> is
> > > >>>>>>>>>> important (i.e. the Airflow DB should not be decoupled with
> > > >> the
> > > >>>> RBAC
> > > >>>>>>>>>> database wherever possible).
> > > >>>>>>>>>>
> > > >>>>>>>>>> I wouldn't be too worried about the permissions about
> > > >> refreshing
> > > >>>>>> DAGs, as
> > > >>>>>>>>>> far as I know this functionality is no longer required with
> > > >> the
> > > >>>> new
> > > >>>>>>>>>> webservers which reload state periodically, and will
> certainly
> > > >>> be
> > > >>>>>> removed
> > > >>>>>>>>>> when we have a better DAG consistency story.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I think it would also be good to think about this
> > > >>>>>> proposal/implementation
> > > >>>>>>>>>> and how it applied in the API-driven world (e.g. when
> > > >> webserver
> > > >>>> hits
> > > >>>>>> APIs
> > > >>>>>>>>>> like /clear on behalf of users instead of running commands
> > > >>> against
> > > >>>>> the
> > > >>>>>>>>>> database directly).
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
> > > >>>> bdbruin@gmail.com
> > > >>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Will respond but im traveling at the moment. Give me a few
> > > >>> days.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Sent from my iPhone
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> > > >>>> criccomini@apache.org>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Hey all,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time
> thinking
> > > >>>> about
> > > >>>>>>>>> this,
> > > >>>>>>>>>>> and
> > > >>>>>>>>>>>> want to move forward with it, but want to make sure we're
> > > >> all
> > > >>> on
> > > >>>>> the
> > > >>>>>>>>>> same
> > > >>>>>>>>>>>> page.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Cheers,
> > > >>>>>>>>>>>> Chris
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
> > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a
> built-in
> > > >>> Role
> > > >>>>>>>>> Based
> > > >>>>>>>>>>>>> Access Control(RBAC) capability.  It does provide very
> > > >>> limited
> > > >>>>>>>>>>>>> authorization capability by providing admin,
> data_profiler,
> > > >>> and
> > > >>>>>> user
> > > >>>>>>>>>>> roles.
> > > >>>>>>>>>>>>> However, associating these roles to authenticated
> > > >> identities
> > > >>> is
> > > >>>>> not
> > > >>>>>>>>> a
> > > >>>>>>>>>>>>> simple effort.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> To address this issue, I have created a design proposal
> for
> > > >>>>>> building
> > > >>>>>>>>>>> RBAC
> > > >>>>>>>>>>>>> into Airflow and simplifying user access management via
> the
> > > >>>>> Airflow
> > > >>>>>>>>>> UI.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> The design proposal is located at
> > > >> https://cwiki.apache.org/
> > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Any comments/questions/feedback are much appreciated.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks
> > > >>>>>>>>>>>>> Kalpesh
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>>
> > > >>> Joy Gao
> > > >>> Software Engineer
> > > >>> 350 Convention Way, Suite 200
> > > >>> Redwood City, CA 94063
> > > >>> Mobile:  669-224-9305
> > > >>>
> > > >>> Payments partner to the platform economy
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: Role Based Access Control for Airflow UI

Posted by Chris Riccomini <cr...@apache.org>.

Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room for 12
people (and video conferencing) at WePay in this time slot. Should allow
you to head home easily afterwards. :) That work for you guys?

On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> The week of the 31st sound good. Wednesday?
>
> About React we may not need a frontend lib like it (or at least not just
> yet). We can talk about it at the meeting.
>
> Max
>
> On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
>
> > We avoid React for the same reasons as the ASF and use Polymer 2 instead.
> > Would that work?
> >
> > Bolke.
> >
> > > On 20 Jul 2017, at 19:35, Chris Riccomini <cr...@apache.org>
> wrote:
> > >
> > > Hey Max,
> > >
> > > Want to come down to WePay? We can set up a zoom for those that want to
> > > join online, and record it as well to post for the community.
> > >
> > > Since Joy is just getting started, and it looks like there's going to
> be
> > a
> > > K8s discussion next week, maybe we can shoot for the week after (the
> week
> > > of the 31st of July)? Care to float a few times that week?
> > >
> > > Cheers,
> > > Chris
> > >
> > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
> > > maximebeauchemin@gmail.com> wrote:
> > >
> > >> Sounds awesome, count me in!
> > >>
> > >> * check out the prototype in my fork, I went far enough to hit some
> > >> hurdles, try different workarounds. I hooked up the Airflow Bootstrap
> > >> template too so that we feel at home in this new UI
> > >> * using a single `id` field is a requirement for FAB that airflow
> > doesn't
> > >> respect (composite pks), either we add the feature to support that in
> > FAB,
> > >> or we align on the Airflow side and modify the models and add a
> > migration
> > >> script. This upgrade would require downtime and might be annoying to
> the
> > >> Airflow community, but could help with db performance a bit (smaller
> > >> index)... I probably could be convinced either way but I'm leaning on
> > >> improving FAB
> > >> * I'm a maintainer for FAB so I can help get stuff through there
> > >> * React is in limbo at the ASF for licensing reasons, so no React at
> > least
> > >> for now
> > >> * npm/webpack/ES6, javascript only in `.js` files
> > >> * I vote for eslint + eslint-config-airbnb as a set of linting rules
> > for JS
> > >> * Keep out of apache (for now), this new app ships as its own pypi
> > package
> > >> `airflow-webserver`, have a period of overlap (maintaining 2 web apps)
> > >> before ripping out `airflow/www` from the core package
> > >> * You need to get in touch with Marty Kausas, an intern at Airbnb
> who's
> > >> been working on a Flask blueprint for improved, more personalized
> views
> > on
> > >> DAGs that we were planning on merging into the main branch eventually.
> > Some
> > >> of Marty's idea and code could be merged into this effort.
> > >>
> > >> These are ideas on how I would proceed personally on this but
> definitely
> > >> everything here is up for discussion.
> > >>
> > >> Let's meet physically at either WePay or Airbnb. Folks from the
> > community,
> > >> let us know on this thread if you want to be part of this effort,
> we'll
> > be
> > >> happy to include you.
> > >>
> > >> Thanks,
> > >>
> > >> Max
> > >>
> > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com> wrote:
> > >>
> > >>> Hey everyone,
> > >>>
> > >>> I recently transferred to Data Infra team here at WePay to focus on
> > >>> Airflow-related initiatives.
> > >>>
> > >>> Given the RBAC design is mostly hashed out, I'm happy to get this
> > feature
> > >>> off the ground for Q3, starting with converting Airflow to Fab, if
> > there
> > >>> are no objections.
> > >>>
> > >>> Cheers,
> > >>> Joy
> > >>>
> > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> > >>> gurer.kiratli@airbnb.com.invalid> wrote:
> > >>>
> > >>>> Hey all,
> > >>>>
> > >>>> We talked about this internally. We would like to work on this
> feature
> > >>> but
> > >>>> given the immediate priorities we are not going to be working on it
> in
> > >>> Q3.
> > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we can work
> on
> > >> it
> > >>>> late Q4 or Q12018.
> > >>>>
> > >>>> Cheers,
> > >>>>
> > >>>> Gurer
> > >>>>
> > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
> > >> criccomini@apache.org>
> > >>>> wrote:
> > >>>>
> > >>>>> I think FAB sounds like the right approach. Waiting to hear back
> with
> > >>>> notes
> > >>>>> on AirBNB H2 discussion to see if they want to take this up.
> > >>>>>
> > >>>>> @Gurer, any idea when this will happen?
> > >>>>>
> > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <bdbruin@gmail.com
> >
> > >>>> wrote:
> > >>>>>
> > >>>>>> One downside I see from FAB is that is does not do Business Role
> > >>>> mapping
> > >>>>>> to FAB role. I would prefer to create groups in IPA/LDAP/AD and
> > >> have
> > >>>>> those
> > >>>>>> map to FAB roles instead of needing to manage that in FAB.
> > >>>>>>
> > >>>>>> B.
> > >>>>>>
> > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <bd...@gmail.com>
> > >>> wrote:
> > >>>>>>>
> > >>>>>>> Hi Guys,
> > >>>>>>>
> > >>>>>>> Thanks for putting the thinking in! It is about time that we get
> > >>> this
> > >>>>>> moving.
> > >>>>>>>
> > >>>>>>> The design looks pretty sound. One can argue about the different
> > >>>> roles
> > >>>>>> that are required, but that will be situation dependent I guess.
> > >>>>>>>
> > >>>>>>> Implementation wise I would argue together with Max that FAB is a
> > >>>>> better
> > >>>>>> or best fit. The ER model that is being described is pretty much a
> > >>> copy
> > >>>>> of
> > >>>>>> a normal security model. So a reimplementation of that is 1)
> > >>>> significant
> > >>>>>> duplication of effort and 2) bound to have bugs that have been
> > >> solved
> > >>>> in
> > >>>>>> the other framework. Moreover, FAB does have integration out of
> the
> > >>> box
> > >>>>>> with some enterprisey systems like IPA, ActiveDirectory, and LDAP.
> > >>>>>>>
> > >>>>>>> So while you argue that using FAB would increase the scope of the
> > >>>>>> proposal significantly, but I think that is not true. Using FAB
> > >> would
> > >>>>> allow
> > >>>>>> you to focus on what kind of out-of-the-box permission sets and
> > >> roles
> > >>>> we
> > >>>>>> would need and maybe address some issues that FAB lacks (maybe how
> > >> to
> > >>>>> deal
> > >>>>>> with non web access - ie. in DAGs, maybe Kerberos, probably how to
> > >>> deal
> > >>>>>> with API calls that are not CRUD). Implementation wise it probably
> > >>>>>> simplifies what we need to do. Maybe - using Max’s early POC as an
> > >>>>> example
> > >>>>>> - we can slowly move over?
> > >>>>>>>
> > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on Airflow
> > >>> coming
> > >>>>>> year. Improvement of Security, Enterprise Integration, Revamp UI
> > >> are
> > >>> on
> > >>>>> the
> > >>>>>> todo list. However, this is not confirmed yet as business
> > >> priorities
> > >>>>> might
> > >>>>>> change.
> > >>>>>>>
> > >>>>>>> Bolke.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > >>>>>>>>
> > >>>>>>>> @Dan:
> > >>>>>>>>
> > >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG
> > >>> permission.
> > >>>>>>>>
> > >>>>>>>> @Max:
> > >>>>>>>>
> > >>>>>>>> Thanks for your response.
> > >>>>>>>>
> > >>>>>>>> The scope of my proposal was just to add RBAC security feature
> > >> to
> > >>>>>> Airflow
> > >>>>>>>> without replacing any existing frameworks.
> > >>>>>>>>
> > >>>>>>>> I understand that adopting FAB would serve Airflow better moving
> > >>>>>> forward,
> > >>>>>>>> however porting Airflow to using FAB significantly increases the
> > >>>> scope
> > >>>>>> of
> > >>>>>>>> the proposal and I don't have the time and expertise to carry
> > >> out
> > >>>> the
> > >>>>>> tasks
> > >>>>>>>> in the extended scope.
> > >>>>>>>>
> > >>>>>>>> Hence, I'm curious to know if there's a plan for Airflow to
> > >>> migrate
> > >>>> to
> > >>>>>> FAB
> > >>>>>>>> this year?
> > >>>>>>>>
> > >>>>>>>> - Kalpesh
> > >>>>>>>>
> > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
> > >>>>>>>> maximebeauchemin@gmail.com> wrote:
> > >>>>>>>>
> > >>>>>>>>> It would be nice to go with a framework for this. I did some
> > >>>>>>>>> experimentation using FlaskAppBuilder to go in this direction.
> > >> It
> > >>>>>> provides
> > >>>>>>>>> auth on different authentication backends out of the box
> > >> (oauth,
> > >>>>>> openid,
> > >>>>>>>>> ldap, registration, ...), generates perms for each view that
> > >> has
> > >>> an
> > >>>>>>>>> @has_access decorator, generates at set of perms for each ORM
> > >>> model
> > >>>>>> (show,
> > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD views as
> > >> well
> > >>>> as
> > >>>>>> in the
> > >>>>>>>>> generated REST api that you get for free as a byprdoduct of
> > >>>> deriving
> > >>>>>> FAB's
> > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on top).
> > >>>>>>>>>
> > >>>>>>>>> I started a POC on FAB here a while ago:
> > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at the time
> > >> my
> > >>>>> main
> > >>>>>>>>> motivation was the free/instantaneous REST api.
> > >>>>>>>>>
> > >>>>>>>>> I think FAB is a decent fit as the porting should be fairly
> > >>>>>> straightforward
> > >>>>>>>>> (moving the flask views over and deprecating Flask-Admin in
> > >> favor
> > >>>> of
> > >>>>>> FAB's
> > >>>>>>>>> crud) though there was a few blockers. From memory I think FAB
> > >>>> didn't
> > >>>>>> like
> > >>>>>>>>> the compound PKs we use in some of the Airflow models. We'd
> > >> have
> > >>> to
> > >>>>>> either
> > >>>>>>>>> write a db migration script on the Airflow side, or add support
> > >>> for
> > >>>>>>>>> compound keys to FAB (I recently became a maintainer of the
> > >>>> project,
> > >>>>>> so I
> > >>>>>>>>> could help with that)
> > >>>>>>>>>
> > >>>>>>>>> The only downside of FAB is that it's not as mature as
> > >> something
> > >>>> like
> > >>>>>>>>> Django, but porting to Django would surely be much more work.
> > >>>>>>>>>
> > >>>>>>>>> Then there's the flask-security suite, but that looks like a
> > >> bit
> > >>>> of a
> > >>>>>>>>> patchwork to me, I guess we can pick and choose which we want
> > >> to
> > >>>> use.
> > >>>>>>>>>
> > >>>>>>>>> Max
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Looks good to me in general, thanks for putting this together!
> > >>>>>>>>>>
> > >>>>>>>>>> I think the ability to integrate with external RBAC systems
> > >> like
> > >>>>> LDAP
> > >>>>>> is
> > >>>>>>>>>> important (i.e. the Airflow DB should not be decoupled with
> > >> the
> > >>>> RBAC
> > >>>>>>>>>> database wherever possible).
> > >>>>>>>>>>
> > >>>>>>>>>> I wouldn't be too worried about the permissions about
> > >> refreshing
> > >>>>>> DAGs, as
> > >>>>>>>>>> far as I know this functionality is no longer required with
> > >> the
> > >>>> new
> > >>>>>>>>>> webservers which reload state periodically, and will certainly
> > >>> be
> > >>>>>> removed
> > >>>>>>>>>> when we have a better DAG consistency story.
> > >>>>>>>>>>
> > >>>>>>>>>> I think it would also be good to think about this
> > >>>>>> proposal/implementation
> > >>>>>>>>>> and how it applied in the API-driven world (e.g. when
> > >> webserver
> > >>>> hits
> > >>>>>> APIs
> > >>>>>>>>>> like /clear on behalf of users instead of running commands
> > >>> against
> > >>>>> the
> > >>>>>>>>>> database directly).
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
> > >>>> bdbruin@gmail.com
> > >>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Will respond but im traveling at the moment. Give me a few
> > >>> days.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Sent from my iPhone
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> > >>>> criccomini@apache.org>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hey all,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time thinking
> > >>>> about
> > >>>>>>>>> this,
> > >>>>>>>>>>> and
> > >>>>>>>>>>>> want to move forward with it, but want to make sure we're
> > >> all
> > >>> on
> > >>>>> the
> > >>>>>>>>>> same
> > >>>>>>>>>>>> page.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>> Chris
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
> > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a built-in
> > >>> Role
> > >>>>>>>>> Based
> > >>>>>>>>>>>>> Access Control(RBAC) capability.  It does provide very
> > >>> limited
> > >>>>>>>>>>>>> authorization capability by providing admin, data_profiler,
> > >>> and
> > >>>>>> user
> > >>>>>>>>>>> roles.
> > >>>>>>>>>>>>> However, associating these roles to authenticated
> > >> identities
> > >>> is
> > >>>>> not
> > >>>>>>>>> a
> > >>>>>>>>>>>>> simple effort.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> To address this issue, I have created a design proposal for
> > >>>>>> building
> > >>>>>>>>>>> RBAC
> > >>>>>>>>>>>>> into Airflow and simplifying user access management via the
> > >>>>> Airflow
> > >>>>>>>>>> UI.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> The design proposal is located at
> > >> https://cwiki.apache.org/
> > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Any comments/questions/feedback are much appreciated.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks
> > >>>>>>>>>>>>> Kalpesh
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Joy Gao
> > >>> Software Engineer
> > >>> 350 Convention Way, Suite 200
> > >>> Redwood City, CA 94063
> > >>> Mobile:  669-224-9305
> > >>>
> > >>> Payments partner to the platform economy
> > >>>
> > >>
> >
> >
>

Re: Role Based Access Control for Airflow UI

Posted by Maxime Beauchemin <ma...@gmail.com>.

The week of the 31st sound good. Wednesday?

About React we may not need a frontend lib like it (or at least not just
yet). We can talk about it at the meeting.

Max

On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <bd...@gmail.com> wrote:

> We avoid React for the same reasons as the ASF and use Polymer 2 instead.
> Would that work?
>
> Bolke.
>
> > On 20 Jul 2017, at 19:35, Chris Riccomini <cr...@apache.org> wrote:
> >
> > Hey Max,
> >
> > Want to come down to WePay? We can set up a zoom for those that want to
> > join online, and record it as well to post for the community.
> >
> > Since Joy is just getting started, and it looks like there's going to be
> a
> > K8s discussion next week, maybe we can shoot for the week after (the week
> > of the 31st of July)? Care to float a few times that week?
> >
> > Cheers,
> > Chris
> >
> > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> >
> >> Sounds awesome, count me in!
> >>
> >> * check out the prototype in my fork, I went far enough to hit some
> >> hurdles, try different workarounds. I hooked up the Airflow Bootstrap
> >> template too so that we feel at home in this new UI
> >> * using a single `id` field is a requirement for FAB that airflow
> doesn't
> >> respect (composite pks), either we add the feature to support that in
> FAB,
> >> or we align on the Airflow side and modify the models and add a
> migration
> >> script. This upgrade would require downtime and might be annoying to the
> >> Airflow community, but could help with db performance a bit (smaller
> >> index)... I probably could be convinced either way but I'm leaning on
> >> improving FAB
> >> * I'm a maintainer for FAB so I can help get stuff through there
> >> * React is in limbo at the ASF for licensing reasons, so no React at
> least
> >> for now
> >> * npm/webpack/ES6, javascript only in `.js` files
> >> * I vote for eslint + eslint-config-airbnb as a set of linting rules
> for JS
> >> * Keep out of apache (for now), this new app ships as its own pypi
> package
> >> `airflow-webserver`, have a period of overlap (maintaining 2 web apps)
> >> before ripping out `airflow/www` from the core package
> >> * You need to get in touch with Marty Kausas, an intern at Airbnb who's
> >> been working on a Flask blueprint for improved, more personalized views
> on
> >> DAGs that we were planning on merging into the main branch eventually.
> Some
> >> of Marty's idea and code could be merged into this effort.
> >>
> >> These are ideas on how I would proceed personally on this but definitely
> >> everything here is up for discussion.
> >>
> >> Let's meet physically at either WePay or Airbnb. Folks from the
> community,
> >> let us know on this thread if you want to be part of this effort, we'll
> be
> >> happy to include you.
> >>
> >> Thanks,
> >>
> >> Max
> >>
> >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com> wrote:
> >>
> >>> Hey everyone,
> >>>
> >>> I recently transferred to Data Infra team here at WePay to focus on
> >>> Airflow-related initiatives.
> >>>
> >>> Given the RBAC design is mostly hashed out, I'm happy to get this
> feature
> >>> off the ground for Q3, starting with converting Airflow to Fab, if
> there
> >>> are no objections.
> >>>
> >>> Cheers,
> >>> Joy
> >>>
> >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> >>> gurer.kiratli@airbnb.com.invalid> wrote:
> >>>
> >>>> Hey all,
> >>>>
> >>>> We talked about this internally. We would like to work on this feature
> >>> but
> >>>> given the immediate priorities we are not going to be working on it in
> >>> Q3.
> >>>> Comes end of Q3 we will reevaluate. Likely scenario is we can work on
> >> it
> >>>> late Q4 or Q12018.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Gurer
> >>>>
> >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
> >> criccomini@apache.org>
> >>>> wrote:
> >>>>
> >>>>> I think FAB sounds like the right approach. Waiting to hear back with
> >>>> notes
> >>>>> on AirBNB H2 discussion to see if they want to take this up.
> >>>>>
> >>>>> @Gurer, any idea when this will happen?
> >>>>>
> >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <bd...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> One downside I see from FAB is that is does not do Business Role
> >>>> mapping
> >>>>>> to FAB role. I would prefer to create groups in IPA/LDAP/AD and
> >> have
> >>>>> those
> >>>>>> map to FAB roles instead of needing to manage that in FAB.
> >>>>>>
> >>>>>> B.
> >>>>>>
> >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <bd...@gmail.com>
> >>> wrote:
> >>>>>>>
> >>>>>>> Hi Guys,
> >>>>>>>
> >>>>>>> Thanks for putting the thinking in! It is about time that we get
> >>> this
> >>>>>> moving.
> >>>>>>>
> >>>>>>> The design looks pretty sound. One can argue about the different
> >>>> roles
> >>>>>> that are required, but that will be situation dependent I guess.
> >>>>>>>
> >>>>>>> Implementation wise I would argue together with Max that FAB is a
> >>>>> better
> >>>>>> or best fit. The ER model that is being described is pretty much a
> >>> copy
> >>>>> of
> >>>>>> a normal security model. So a reimplementation of that is 1)
> >>>> significant
> >>>>>> duplication of effort and 2) bound to have bugs that have been
> >> solved
> >>>> in
> >>>>>> the other framework. Moreover, FAB does have integration out of the
> >>> box
> >>>>>> with some enterprisey systems like IPA, ActiveDirectory, and LDAP.
> >>>>>>>
> >>>>>>> So while you argue that using FAB would increase the scope of the
> >>>>>> proposal significantly, but I think that is not true. Using FAB
> >> would
> >>>>> allow
> >>>>>> you to focus on what kind of out-of-the-box permission sets and
> >> roles
> >>>> we
> >>>>>> would need and maybe address some issues that FAB lacks (maybe how
> >> to
> >>>>> deal
> >>>>>> with non web access - ie. in DAGs, maybe Kerberos, probably how to
> >>> deal
> >>>>>> with API calls that are not CRUD). Implementation wise it probably
> >>>>>> simplifies what we need to do. Maybe - using Max’s early POC as an
> >>>>> example
> >>>>>> - we can slowly move over?
> >>>>>>>
> >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on Airflow
> >>> coming
> >>>>>> year. Improvement of Security, Enterprise Integration, Revamp UI
> >> are
> >>> on
> >>>>> the
> >>>>>> todo list. However, this is not confirmed yet as business
> >> priorities
> >>>>> might
> >>>>>> change.
> >>>>>>>
> >>>>>>> Bolke.
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> @Dan:
> >>>>>>>>
> >>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG
> >>> permission.
> >>>>>>>>
> >>>>>>>> @Max:
> >>>>>>>>
> >>>>>>>> Thanks for your response.
> >>>>>>>>
> >>>>>>>> The scope of my proposal was just to add RBAC security feature
> >> to
> >>>>>> Airflow
> >>>>>>>> without replacing any existing frameworks.
> >>>>>>>>
> >>>>>>>> I understand that adopting FAB would serve Airflow better moving
> >>>>>> forward,
> >>>>>>>> however porting Airflow to using FAB significantly increases the
> >>>> scope
> >>>>>> of
> >>>>>>>> the proposal and I don't have the time and expertise to carry
> >> out
> >>>> the
> >>>>>> tasks
> >>>>>>>> in the extended scope.
> >>>>>>>>
> >>>>>>>> Hence, I'm curious to know if there's a plan for Airflow to
> >>> migrate
> >>>> to
> >>>>>> FAB
> >>>>>>>> this year?
> >>>>>>>>
> >>>>>>>> - Kalpesh
> >>>>>>>>
> >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
> >>>>>>>> maximebeauchemin@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> It would be nice to go with a framework for this. I did some
> >>>>>>>>> experimentation using FlaskAppBuilder to go in this direction.
> >> It
> >>>>>> provides
> >>>>>>>>> auth on different authentication backends out of the box
> >> (oauth,
> >>>>>> openid,
> >>>>>>>>> ldap, registration, ...), generates perms for each view that
> >> has
> >>> an
> >>>>>>>>> @has_access decorator, generates at set of perms for each ORM
> >>> model
> >>>>>> (show,
> >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD views as
> >> well
> >>>> as
> >>>>>> in the
> >>>>>>>>> generated REST api that you get for free as a byprdoduct of
> >>>> deriving
> >>>>>> FAB's
> >>>>>>>>> models (essentially it's SqlAlchemy with a layer on top).
> >>>>>>>>>
> >>>>>>>>> I started a POC on FAB here a while ago:
> >>>>>>>>> https://github.com/mistercrunch/airflow_webserver at the time
> >> my
> >>>>> main
> >>>>>>>>> motivation was the free/instantaneous REST api.
> >>>>>>>>>
> >>>>>>>>> I think FAB is a decent fit as the porting should be fairly
> >>>>>> straightforward
> >>>>>>>>> (moving the flask views over and deprecating Flask-Admin in
> >> favor
> >>>> of
> >>>>>> FAB's
> >>>>>>>>> crud) though there was a few blockers. From memory I think FAB
> >>>> didn't
> >>>>>> like
> >>>>>>>>> the compound PKs we use in some of the Airflow models. We'd
> >> have
> >>> to
> >>>>>> either
> >>>>>>>>> write a db migration script on the Airflow side, or add support
> >>> for
> >>>>>>>>> compound keys to FAB (I recently became a maintainer of the
> >>>> project,
> >>>>>> so I
> >>>>>>>>> could help with that)
> >>>>>>>>>
> >>>>>>>>> The only downside of FAB is that it's not as mature as
> >> something
> >>>> like
> >>>>>>>>> Django, but porting to Django would surely be much more work.
> >>>>>>>>>
> >>>>>>>>> Then there's the flask-security suite, but that looks like a
> >> bit
> >>>> of a
> >>>>>>>>> patchwork to me, I guess we can pick and choose which we want
> >> to
> >>>> use.
> >>>>>>>>>
> >>>>>>>>> Max
> >>>>>>>>>
> >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
> >>>>>>>>>
> >>>>>>>>>> Looks good to me in general, thanks for putting this together!
> >>>>>>>>>>
> >>>>>>>>>> I think the ability to integrate with external RBAC systems
> >> like
> >>>>> LDAP
> >>>>>> is
> >>>>>>>>>> important (i.e. the Airflow DB should not be decoupled with
> >> the
> >>>> RBAC
> >>>>>>>>>> database wherever possible).
> >>>>>>>>>>
> >>>>>>>>>> I wouldn't be too worried about the permissions about
> >> refreshing
> >>>>>> DAGs, as
> >>>>>>>>>> far as I know this functionality is no longer required with
> >> the
> >>>> new
> >>>>>>>>>> webservers which reload state periodically, and will certainly
> >>> be
> >>>>>> removed
> >>>>>>>>>> when we have a better DAG consistency story.
> >>>>>>>>>>
> >>>>>>>>>> I think it would also be good to think about this
> >>>>>> proposal/implementation
> >>>>>>>>>> and how it applied in the API-driven world (e.g. when
> >> webserver
> >>>> hits
> >>>>>> APIs
> >>>>>>>>>> like /clear on behalf of users instead of running commands
> >>> against
> >>>>> the
> >>>>>>>>>> database directly).
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
> >>>> bdbruin@gmail.com
> >>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Will respond but im traveling at the moment. Give me a few
> >>> days.
> >>>>>>>>>>>
> >>>>>>>>>>> Sent from my iPhone
> >>>>>>>>>>>
> >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> >>>> criccomini@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Checking in on this. We spent a good chunk of time thinking
> >>>> about
> >>>>>>>>> this,
> >>>>>>>>>>> and
> >>>>>>>>>>>> want to move forward with it, but want to make sure we're
> >> all
> >>> on
> >>>>> the
> >>>>>>>>>> same
> >>>>>>>>>>>> page.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Chris
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
> >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a built-in
> >>> Role
> >>>>>>>>> Based
> >>>>>>>>>>>>> Access Control(RBAC) capability.  It does provide very
> >>> limited
> >>>>>>>>>>>>> authorization capability by providing admin, data_profiler,
> >>> and
> >>>>>> user
> >>>>>>>>>>> roles.
> >>>>>>>>>>>>> However, associating these roles to authenticated
> >> identities
> >>> is
> >>>>> not
> >>>>>>>>> a
> >>>>>>>>>>>>> simple effort.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> To address this issue, I have created a design proposal for
> >>>>>> building
> >>>>>>>>>>> RBAC
> >>>>>>>>>>>>> into Airflow and simplifying user access management via the
> >>>>> Airflow
> >>>>>>>>>> UI.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The design proposal is located at
> >> https://cwiki.apache.org/
> >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Any comments/questions/feedback are much appreciated.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>> Kalpesh
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Joy Gao
> >>> Software Engineer
> >>> 350 Convention Way, Suite 200
> >>> Redwood City, CA 94063
> >>> Mobile:  669-224-9305
> >>>
> >>> Payments partner to the platform economy
> >>>
> >>
>
>

Re: Role Based Access Control for Airflow UI

Posted by Bolke de Bruin <bd...@gmail.com>.

We avoid React for the same reasons as the ASF and use Polymer 2 instead. Would that work?

Bolke.

> On 20 Jul 2017, at 19:35, Chris Riccomini <cr...@apache.org> wrote:
> 
> Hey Max,
> 
> Want to come down to WePay? We can set up a zoom for those that want to
> join online, and record it as well to post for the community.
> 
> Since Joy is just getting started, and it looks like there's going to be a
> K8s discussion next week, maybe we can shoot for the week after (the week
> of the 31st of July)? Care to float a few times that week?
> 
> Cheers,
> Chris
> 
> On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
> 
>> Sounds awesome, count me in!
>> 
>> * check out the prototype in my fork, I went far enough to hit some
>> hurdles, try different workarounds. I hooked up the Airflow Bootstrap
>> template too so that we feel at home in this new UI
>> * using a single `id` field is a requirement for FAB that airflow doesn't
>> respect (composite pks), either we add the feature to support that in FAB,
>> or we align on the Airflow side and modify the models and add a migration
>> script. This upgrade would require downtime and might be annoying to the
>> Airflow community, but could help with db performance a bit (smaller
>> index)... I probably could be convinced either way but I'm leaning on
>> improving FAB
>> * I'm a maintainer for FAB so I can help get stuff through there
>> * React is in limbo at the ASF for licensing reasons, so no React at least
>> for now
>> * npm/webpack/ES6, javascript only in `.js` files
>> * I vote for eslint + eslint-config-airbnb as a set of linting rules for JS
>> * Keep out of apache (for now), this new app ships as its own pypi package
>> `airflow-webserver`, have a period of overlap (maintaining 2 web apps)
>> before ripping out `airflow/www` from the core package
>> * You need to get in touch with Marty Kausas, an intern at Airbnb who's
>> been working on a Flask blueprint for improved, more personalized views on
>> DAGs that we were planning on merging into the main branch eventually. Some
>> of Marty's idea and code could be merged into this effort.
>> 
>> These are ideas on how I would proceed personally on this but definitely
>> everything here is up for discussion.
>> 
>> Let's meet physically at either WePay or Airbnb. Folks from the community,
>> let us know on this thread if you want to be part of this effort, we'll be
>> happy to include you.
>> 
>> Thanks,
>> 
>> Max
>> 
>> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com> wrote:
>> 
>>> Hey everyone,
>>> 
>>> I recently transferred to Data Infra team here at WePay to focus on
>>> Airflow-related initiatives.
>>> 
>>> Given the RBAC design is mostly hashed out, I'm happy to get this feature
>>> off the ground for Q3, starting with converting Airflow to Fab, if there
>>> are no objections.
>>> 
>>> Cheers,
>>> Joy
>>> 
>>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
>>> gurer.kiratli@airbnb.com.invalid> wrote:
>>> 
>>>> Hey all,
>>>> 
>>>> We talked about this internally. We would like to work on this feature
>>> but
>>>> given the immediate priorities we are not going to be working on it in
>>> Q3.
>>>> Comes end of Q3 we will reevaluate. Likely scenario is we can work on
>> it
>>>> late Q4 or Q12018.
>>>> 
>>>> Cheers,
>>>> 
>>>> Gurer
>>>> 
>>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
>> criccomini@apache.org>
>>>> wrote:
>>>> 
>>>>> I think FAB sounds like the right approach. Waiting to hear back with
>>>> notes
>>>>> on AirBNB H2 discussion to see if they want to take this up.
>>>>> 
>>>>> @Gurer, any idea when this will happen?
>>>>> 
>>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <bd...@gmail.com>
>>>> wrote:
>>>>> 
>>>>>> One downside I see from FAB is that is does not do Business Role
>>>> mapping
>>>>>> to FAB role. I would prefer to create groups in IPA/LDAP/AD and
>> have
>>>>> those
>>>>>> map to FAB roles instead of needing to manage that in FAB.
>>>>>> 
>>>>>> B.
>>>>>> 
>>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <bd...@gmail.com>
>>> wrote:
>>>>>>> 
>>>>>>> Hi Guys,
>>>>>>> 
>>>>>>> Thanks for putting the thinking in! It is about time that we get
>>> this
>>>>>> moving.
>>>>>>> 
>>>>>>> The design looks pretty sound. One can argue about the different
>>>> roles
>>>>>> that are required, but that will be situation dependent I guess.
>>>>>>> 
>>>>>>> Implementation wise I would argue together with Max that FAB is a
>>>>> better
>>>>>> or best fit. The ER model that is being described is pretty much a
>>> copy
>>>>> of
>>>>>> a normal security model. So a reimplementation of that is 1)
>>>> significant
>>>>>> duplication of effort and 2) bound to have bugs that have been
>> solved
>>>> in
>>>>>> the other framework. Moreover, FAB does have integration out of the
>>> box
>>>>>> with some enterprisey systems like IPA, ActiveDirectory, and LDAP.
>>>>>>> 
>>>>>>> So while you argue that using FAB would increase the scope of the
>>>>>> proposal significantly, but I think that is not true. Using FAB
>> would
>>>>> allow
>>>>>> you to focus on what kind of out-of-the-box permission sets and
>> roles
>>>> we
>>>>>> would need and maybe address some issues that FAB lacks (maybe how
>> to
>>>>> deal
>>>>>> with non web access - ie. in DAGs, maybe Kerberos, probably how to
>>> deal
>>>>>> with API calls that are not CRUD). Implementation wise it probably
>>>>>> simplifies what we need to do. Maybe - using Max’s early POC as an
>>>>> example
>>>>>> - we can slowly move over?
>>>>>>> 
>>>>>>> On a side note: Im planning to hire 2-3 ppl to work on Airflow
>>> coming
>>>>>> year. Improvement of Security, Enterprise Integration, Revamp UI
>> are
>>> on
>>>>> the
>>>>>> todo list. However, this is not confirmed yet as business
>> priorities
>>>>> might
>>>>>> change.
>>>>>>> 
>>>>>>> Bolke.
>>>>>>> 
>>>>>>> 
>>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
>>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>>>>>>> 
>>>>>>>> @Dan:
>>>>>>>> 
>>>>>>>> Thanks for your feedback. I will remove the REFRESH_DAG
>>> permission.
>>>>>>>> 
>>>>>>>> @Max:
>>>>>>>> 
>>>>>>>> Thanks for your response.
>>>>>>>> 
>>>>>>>> The scope of my proposal was just to add RBAC security feature
>> to
>>>>>> Airflow
>>>>>>>> without replacing any existing frameworks.
>>>>>>>> 
>>>>>>>> I understand that adopting FAB would serve Airflow better moving
>>>>>> forward,
>>>>>>>> however porting Airflow to using FAB significantly increases the
>>>> scope
>>>>>> of
>>>>>>>> the proposal and I don't have the time and expertise to carry
>> out
>>>> the
>>>>>> tasks
>>>>>>>> in the extended scope.
>>>>>>>> 
>>>>>>>> Hence, I'm curious to know if there's a plan for Airflow to
>>> migrate
>>>> to
>>>>>> FAB
>>>>>>>> this year?
>>>>>>>> 
>>>>>>>> - Kalpesh
>>>>>>>> 
>>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> It would be nice to go with a framework for this. I did some
>>>>>>>>> experimentation using FlaskAppBuilder to go in this direction.
>> It
>>>>>> provides
>>>>>>>>> auth on different authentication backends out of the box
>> (oauth,
>>>>>> openid,
>>>>>>>>> ldap, registration, ...), generates perms for each view that
>> has
>>> an
>>>>>>>>> @has_access decorator, generates at set of perms for each ORM
>>> model
>>>>>> (show,
>>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD views as
>> well
>>>> as
>>>>>> in the
>>>>>>>>> generated REST api that you get for free as a byprdoduct of
>>>> deriving
>>>>>> FAB's
>>>>>>>>> models (essentially it's SqlAlchemy with a layer on top).
>>>>>>>>> 
>>>>>>>>> I started a POC on FAB here a while ago:
>>>>>>>>> https://github.com/mistercrunch/airflow_webserver at the time
>> my
>>>>> main
>>>>>>>>> motivation was the free/instantaneous REST api.
>>>>>>>>> 
>>>>>>>>> I think FAB is a decent fit as the porting should be fairly
>>>>>> straightforward
>>>>>>>>> (moving the flask views over and deprecating Flask-Admin in
>> favor
>>>> of
>>>>>> FAB's
>>>>>>>>> crud) though there was a few blockers. From memory I think FAB
>>>> didn't
>>>>>> like
>>>>>>>>> the compound PKs we use in some of the Airflow models. We'd
>> have
>>> to
>>>>>> either
>>>>>>>>> write a db migration script on the Airflow side, or add support
>>> for
>>>>>>>>> compound keys to FAB (I recently became a maintainer of the
>>>> project,
>>>>>> so I
>>>>>>>>> could help with that)
>>>>>>>>> 
>>>>>>>>> The only downside of FAB is that it's not as mature as
>> something
>>>> like
>>>>>>>>> Django, but porting to Django would surely be much more work.
>>>>>>>>> 
>>>>>>>>> Then there's the flask-security suite, but that looks like a
>> bit
>>>> of a
>>>>>>>>> patchwork to me, I guess we can pick and choose which we want
>> to
>>>> use.
>>>>>>>>> 
>>>>>>>>> Max
>>>>>>>>> 
>>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
>>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
>>>>>>>>> 
>>>>>>>>>> Looks good to me in general, thanks for putting this together!
>>>>>>>>>> 
>>>>>>>>>> I think the ability to integrate with external RBAC systems
>> like
>>>>> LDAP
>>>>>> is
>>>>>>>>>> important (i.e. the Airflow DB should not be decoupled with
>> the
>>>> RBAC
>>>>>>>>>> database wherever possible).
>>>>>>>>>> 
>>>>>>>>>> I wouldn't be too worried about the permissions about
>> refreshing
>>>>>> DAGs, as
>>>>>>>>>> far as I know this functionality is no longer required with
>> the
>>>> new
>>>>>>>>>> webservers which reload state periodically, and will certainly
>>> be
>>>>>> removed
>>>>>>>>>> when we have a better DAG consistency story.
>>>>>>>>>> 
>>>>>>>>>> I think it would also be good to think about this
>>>>>> proposal/implementation
>>>>>>>>>> and how it applied in the API-driven world (e.g. when
>> webserver
>>>> hits
>>>>>> APIs
>>>>>>>>>> like /clear on behalf of users instead of running commands
>>> against
>>>>> the
>>>>>>>>>> database directly).
>>>>>>>>>> 
>>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
>>>> bdbruin@gmail.com
>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Will respond but im traveling at the moment. Give me a few
>>> days.
>>>>>>>>>>> 
>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>> 
>>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
>>>> criccomini@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hey all,
>>>>>>>>>>>> 
>>>>>>>>>>>> Checking in on this. We spent a good chunk of time thinking
>>>> about
>>>>>>>>> this,
>>>>>>>>>>> and
>>>>>>>>>>>> want to move forward with it, but want to make sure we're
>> all
>>> on
>>>>> the
>>>>>>>>>> same
>>>>>>>>>>>> page.
>>>>>>>>>>>> 
>>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Chris
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
>>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a built-in
>>> Role
>>>>>>>>> Based
>>>>>>>>>>>>> Access Control(RBAC) capability.  It does provide very
>>> limited
>>>>>>>>>>>>> authorization capability by providing admin, data_profiler,
>>> and
>>>>>> user
>>>>>>>>>>> roles.
>>>>>>>>>>>>> However, associating these roles to authenticated
>> identities
>>> is
>>>>> not
>>>>>>>>> a
>>>>>>>>>>>>> simple effort.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> To address this issue, I have created a design proposal for
>>>>>> building
>>>>>>>>>>> RBAC
>>>>>>>>>>>>> into Airflow and simplifying user access management via the
>>>>> Airflow
>>>>>>>>>> UI.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The design proposal is located at
>> https://cwiki.apache.org/
>>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Any comments/questions/feedback are much appreciated.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Kalpesh
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> 
>>> Joy Gao
>>> Software Engineer
>>> 350 Convention Way, Suite 200
>>> Redwood City, CA 94063
>>> Mobile:  669-224-9305
>>> 
>>> Payments partner to the platform economy
>>> 
>>

Re: Role Based Access Control for Airflow UI

Posted by Chris Riccomini <cr...@apache.org>.

Hey Max,

Want to come down to WePay? We can set up a zoom for those that want to
join online, and record it as well to post for the community.

Since Joy is just getting started, and it looks like there's going to be a
K8s discussion next week, maybe we can shoot for the week after (the week
of the 31st of July)? Care to float a few times that week?

Cheers,
Chris

On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> Sounds awesome, count me in!
>
> * check out the prototype in my fork, I went far enough to hit some
> hurdles, try different workarounds. I hooked up the Airflow Bootstrap
> template too so that we feel at home in this new UI
> * using a single `id` field is a requirement for FAB that airflow doesn't
> respect (composite pks), either we add the feature to support that in FAB,
> or we align on the Airflow side and modify the models and add a migration
> script. This upgrade would require downtime and might be annoying to the
> Airflow community, but could help with db performance a bit (smaller
> index)... I probably could be convinced either way but I'm leaning on
> improving FAB
> * I'm a maintainer for FAB so I can help get stuff through there
> * React is in limbo at the ASF for licensing reasons, so no React at least
> for now
> * npm/webpack/ES6, javascript only in `.js` files
> * I vote for eslint + eslint-config-airbnb as a set of linting rules for JS
> * Keep out of apache (for now), this new app ships as its own pypi package
> `airflow-webserver`, have a period of overlap (maintaining 2 web apps)
> before ripping out `airflow/www` from the core package
> * You need to get in touch with Marty Kausas, an intern at Airbnb who's
> been working on a Flask blueprint for improved, more personalized views on
> DAGs that we were planning on merging into the main branch eventually. Some
> of Marty's idea and code could be merged into this effort.
>
> These are ideas on how I would proceed personally on this but definitely
> everything here is up for discussion.
>
> Let's meet physically at either WePay or Airbnb. Folks from the community,
> let us know on this thread if you want to be part of this effort, we'll be
> happy to include you.
>
> Thanks,
>
> Max
>
> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hey everyone,
> >
> > I recently transferred to Data Infra team here at WePay to focus on
> > Airflow-related initiatives.
> >
> > Given the RBAC design is mostly hashed out, I'm happy to get this feature
> > off the ground for Q3, starting with converting Airflow to Fab, if there
> > are no objections.
> >
> > Cheers,
> > Joy
> >
> > On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> > gurer.kiratli@airbnb.com.invalid> wrote:
> >
> > > Hey all,
> > >
> > > We talked about this internally. We would like to work on this feature
> > but
> > > given the immediate priorities we are not going to be working on it in
> > Q3.
> > > Comes end of Q3 we will reevaluate. Likely scenario is we can work on
> it
> > > late Q4 or Q12018.
> > >
> > > Cheers,
> > >
> > > Gurer
> > >
> > > On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
> criccomini@apache.org>
> > > wrote:
> > >
> > > > I think FAB sounds like the right approach. Waiting to hear back with
> > > notes
> > > > on AirBNB H2 discussion to see if they want to take this up.
> > > >
> > > > @Gurer, any idea when this will happen?
> > > >
> > > > On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <bd...@gmail.com>
> > > wrote:
> > > >
> > > > > One downside I see from FAB is that is does not do Business Role
> > > mapping
> > > > > to FAB role. I would prefer to create groups in IPA/LDAP/AD and
> have
> > > > those
> > > > > map to FAB roles instead of needing to manage that in FAB.
> > > > >
> > > > > B.
> > > > >
> > > > > > On 22 Jun 2017, at 09:36, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > > > > >
> > > > > > Hi Guys,
> > > > > >
> > > > > > Thanks for putting the thinking in! It is about time that we get
> > this
> > > > > moving.
> > > > > >
> > > > > > The design looks pretty sound. One can argue about the different
> > > roles
> > > > > that are required, but that will be situation dependent I guess.
> > > > > >
> > > > > > Implementation wise I would argue together with Max that FAB is a
> > > > better
> > > > > or best fit. The ER model that is being described is pretty much a
> > copy
> > > > of
> > > > > a normal security model. So a reimplementation of that is 1)
> > > significant
> > > > > duplication of effort and 2) bound to have bugs that have been
> solved
> > > in
> > > > > the other framework. Moreover, FAB does have integration out of the
> > box
> > > > > with some enterprisey systems like IPA, ActiveDirectory, and LDAP.
> > > > > >
> > > > > > So while you argue that using FAB would increase the scope of the
> > > > > proposal significantly, but I think that is not true. Using FAB
> would
> > > > allow
> > > > > you to focus on what kind of out-of-the-box permission sets and
> roles
> > > we
> > > > > would need and maybe address some issues that FAB lacks (maybe how
> to
> > > > deal
> > > > > with non web access - ie. in DAGs, maybe Kerberos, probably how to
> > deal
> > > > > with API calls that are not CRUD). Implementation wise it probably
> > > > > simplifies what we need to do. Maybe - using Max’s early POC as an
> > > > example
> > > > > - we can slowly move over?
> > > > > >
> > > > > > On a side note: Im planning to hire 2-3 ppl to work on Airflow
> > coming
> > > > > year. Improvement of Security, Enterprise Integration, Revamp UI
> are
> > on
> > > > the
> > > > > todo list. However, this is not confirmed yet as business
> priorities
> > > > might
> > > > > change.
> > > > > >
> > > > > > Bolke.
> > > > > >
> > > > > >
> > > > > >> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> > > > > kalpeshdharwadkar@gmail.com> wrote:
> > > > > >>
> > > > > >> @Dan:
> > > > > >>
> > > > > >> Thanks for your feedback. I will remove the REFRESH_DAG
> > permission.
> > > > > >>
> > > > > >> @Max:
> > > > > >>
> > > > > >> Thanks for your response.
> > > > > >>
> > > > > >> The scope of my proposal was just to add RBAC security feature
> to
> > > > > Airflow
> > > > > >> without replacing any existing frameworks.
> > > > > >>
> > > > > >> I understand that adopting FAB would serve Airflow better moving
> > > > > forward,
> > > > > >> however porting Airflow to using FAB significantly increases the
> > > scope
> > > > > of
> > > > > >> the proposal and I don't have the time and expertise to carry
> out
> > > the
> > > > > tasks
> > > > > >> in the extended scope.
> > > > > >>
> > > > > >> Hence, I'm curious to know if there's a plan for Airflow to
> > migrate
> > > to
> > > > > FAB
> > > > > >> this year?
> > > > > >>
> > > > > >> - Kalpesh
> > > > > >>
> > > > > >> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
> > > > > >> maximebeauchemin@gmail.com> wrote:
> > > > > >>
> > > > > >>> It would be nice to go with a framework for this. I did some
> > > > > >>> experimentation using FlaskAppBuilder to go in this direction.
> It
> > > > > provides
> > > > > >>> auth on different authentication backends out of the box
> (oauth,
> > > > > openid,
> > > > > >>> ldap, registration, ...), generates perms for each view that
> has
> > an
> > > > > >>> @has_access decorator, generates at set of perms for each ORM
> > model
> > > > > (show,
> > > > > >>> edit, delete, add, ...) and enforces it in the CRUD views as
> well
> > > as
> > > > > in the
> > > > > >>> generated REST api that you get for free as a byprdoduct of
> > > deriving
> > > > > FAB's
> > > > > >>> models (essentially it's SqlAlchemy with a layer on top).
> > > > > >>>
> > > > > >>> I started a POC on FAB here a while ago:
> > > > > >>> https://github.com/mistercrunch/airflow_webserver at the time
> my
> > > > main
> > > > > >>> motivation was the free/instantaneous REST api.
> > > > > >>>
> > > > > >>> I think FAB is a decent fit as the porting should be fairly
> > > > > straightforward
> > > > > >>> (moving the flask views over and deprecating Flask-Admin in
> favor
> > > of
> > > > > FAB's
> > > > > >>> crud) though there was a few blockers. From memory I think FAB
> > > didn't
> > > > > like
> > > > > >>> the compound PKs we use in some of the Airflow models. We'd
> have
> > to
> > > > > either
> > > > > >>> write a db migration script on the Airflow side, or add support
> > for
> > > > > >>> compound keys to FAB (I recently became a maintainer of the
> > > project,
> > > > > so I
> > > > > >>> could help with that)
> > > > > >>>
> > > > > >>> The only downside of FAB is that it's not as mature as
> something
> > > like
> > > > > >>> Django, but porting to Django would surely be much more work.
> > > > > >>>
> > > > > >>> Then there's the flask-security suite, but that looks like a
> bit
> > > of a
> > > > > >>> patchwork to me, I guess we can pick and choose which we want
> to
> > > use.
> > > > > >>>
> > > > > >>> Max
> > > > > >>>
> > > > > >>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> > > > > >>> dan.davydov@airbnb.com.invalid> wrote:
> > > > > >>>
> > > > > >>>> Looks good to me in general, thanks for putting this together!
> > > > > >>>>
> > > > > >>>> I think the ability to integrate with external RBAC systems
> like
> > > > LDAP
> > > > > is
> > > > > >>>> important (i.e. the Airflow DB should not be decoupled with
> the
> > > RBAC
> > > > > >>>> database wherever possible).
> > > > > >>>>
> > > > > >>>> I wouldn't be too worried about the permissions about
> refreshing
> > > > > DAGs, as
> > > > > >>>> far as I know this functionality is no longer required with
> the
> > > new
> > > > > >>>> webservers which reload state periodically, and will certainly
> > be
> > > > > removed
> > > > > >>>> when we have a better DAG consistency story.
> > > > > >>>>
> > > > > >>>> I think it would also be good to think about this
> > > > > proposal/implementation
> > > > > >>>> and how it applied in the API-driven world (e.g. when
> webserver
> > > hits
> > > > > APIs
> > > > > >>>> like /clear on behalf of users instead of running commands
> > against
> > > > the
> > > > > >>>> database directly).
> > > > > >>>>
> > > > > >>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
> > > bdbruin@gmail.com
> > > > >
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> Will respond but im traveling at the moment. Give me a few
> > days.
> > > > > >>>>>
> > > > > >>>>> Sent from my iPhone
> > > > > >>>>>
> > > > > >>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> > > criccomini@apache.org>
> > > > > >>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>> Hey all,
> > > > > >>>>>>
> > > > > >>>>>> Checking in on this. We spent a good chunk of time thinking
> > > about
> > > > > >>> this,
> > > > > >>>>> and
> > > > > >>>>>> want to move forward with it, but want to make sure we're
> all
> > on
> > > > the
> > > > > >>>> same
> > > > > >>>>>> page.
> > > > > >>>>>>
> > > > > >>>>>> Max? Bolke? Dan? Jeremiah?
> > > > > >>>>>>
> > > > > >>>>>> Cheers,
> > > > > >>>>>> Chris
> > > > > >>>>>>
> > > > > >>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
> > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hello everyone,
> > > > > >>>>>>>
> > > > > >>>>>>> As you all know, currently Airflow doesn’t have a built-in
> > Role
> > > > > >>> Based
> > > > > >>>>>>> Access Control(RBAC) capability.  It does provide very
> > limited
> > > > > >>>>>>> authorization capability by providing admin, data_profiler,
> > and
> > > > > user
> > > > > >>>>> roles.
> > > > > >>>>>>> However, associating these roles to authenticated
> identities
> > is
> > > > not
> > > > > >>> a
> > > > > >>>>>>> simple effort.
> > > > > >>>>>>>
> > > > > >>>>>>> To address this issue, I have created a design proposal for
> > > > > building
> > > > > >>>>> RBAC
> > > > > >>>>>>> into Airflow and simplifying user access management via the
> > > > Airflow
> > > > > >>>> UI.
> > > > > >>>>>>>
> > > > > >>>>>>> The design proposal is located at
> https://cwiki.apache.org/
> > > > > >>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
> > > > > >>>>>>>
> > > > > >>>>>>> Any comments/questions/feedback are much appreciated.
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks
> > > > > >>>>>>> Kalpesh
> > > > > >>>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Joy Gao
> > Software Engineer
> > 350 Convention Way, Suite 200
> > Redwood City, CA 94063
> > Mobile:  669-224-9305
> >
> > Payments partner to the platform economy
> >
>

Re: Role Based Access Control for Airflow UI

Posted by Maxime Beauchemin <ma...@gmail.com>.

Sounds awesome, count me in!

* check out the prototype in my fork, I went far enough to hit some
hurdles, try different workarounds. I hooked up the Airflow Bootstrap
template too so that we feel at home in this new UI
* using a single `id` field is a requirement for FAB that airflow doesn't
respect (composite pks), either we add the feature to support that in FAB,
or we align on the Airflow side and modify the models and add a migration
script. This upgrade would require downtime and might be annoying to the
Airflow community, but could help with db performance a bit (smaller
index)... I probably could be convinced either way but I'm leaning on
improving FAB
* I'm a maintainer for FAB so I can help get stuff through there
* React is in limbo at the ASF for licensing reasons, so no React at least
for now
* npm/webpack/ES6, javascript only in `.js` files
* I vote for eslint + eslint-config-airbnb as a set of linting rules for JS
* Keep out of apache (for now), this new app ships as its own pypi package
`airflow-webserver`, have a period of overlap (maintaining 2 web apps)
before ripping out `airflow/www` from the core package
* You need to get in touch with Marty Kausas, an intern at Airbnb who's
been working on a Flask blueprint for improved, more personalized views on
DAGs that we were planning on merging into the main branch eventually. Some
of Marty's idea and code could be merged into this effort.

These are ideas on how I would proceed personally on this but definitely
everything here is up for discussion.

Let's meet physically at either WePay or Airbnb. Folks from the community,
let us know on this thread if you want to be part of this effort, we'll be
happy to include you.

Thanks,

Max

On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com> wrote:

> Hey everyone,
>
> I recently transferred to Data Infra team here at WePay to focus on
> Airflow-related initiatives.
>
> Given the RBAC design is mostly hashed out, I'm happy to get this feature
> off the ground for Q3, starting with converting Airflow to Fab, if there
> are no objections.
>
> Cheers,
> Joy
>
> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> gurer.kiratli@airbnb.com.invalid> wrote:
>
> > Hey all,
> >
> > We talked about this internally. We would like to work on this feature
> but
> > given the immediate priorities we are not going to be working on it in
> Q3.
> > Comes end of Q3 we will reevaluate. Likely scenario is we can work on it
> > late Q4 or Q12018.
> >
> > Cheers,
> >
> > Gurer
> >
> > On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> > > I think FAB sounds like the right approach. Waiting to hear back with
> > notes
> > > on AirBNB H2 discussion to see if they want to take this up.
> > >
> > > @Gurer, any idea when this will happen?
> > >
> > > On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >
> > > > One downside I see from FAB is that is does not do Business Role
> > mapping
> > > > to FAB role. I would prefer to create groups in IPA/LDAP/AD and have
> > > those
> > > > map to FAB roles instead of needing to manage that in FAB.
> > > >
> > > > B.
> > > >
> > > > > On 22 Jun 2017, at 09:36, Bolke de Bruin <bd...@gmail.com>
> wrote:
> > > > >
> > > > > Hi Guys,
> > > > >
> > > > > Thanks for putting the thinking in! It is about time that we get
> this
> > > > moving.
> > > > >
> > > > > The design looks pretty sound. One can argue about the different
> > roles
> > > > that are required, but that will be situation dependent I guess.
> > > > >
> > > > > Implementation wise I would argue together with Max that FAB is a
> > > better
> > > > or best fit. The ER model that is being described is pretty much a
> copy
> > > of
> > > > a normal security model. So a reimplementation of that is 1)
> > significant
> > > > duplication of effort and 2) bound to have bugs that have been solved
> > in
> > > > the other framework. Moreover, FAB does have integration out of the
> box
> > > > with some enterprisey systems like IPA, ActiveDirectory, and LDAP.
> > > > >
> > > > > So while you argue that using FAB would increase the scope of the
> > > > proposal significantly, but I think that is not true. Using FAB would
> > > allow
> > > > you to focus on what kind of out-of-the-box permission sets and roles
> > we
> > > > would need and maybe address some issues that FAB lacks (maybe how to
> > > deal
> > > > with non web access - ie. in DAGs, maybe Kerberos, probably how to
> deal
> > > > with API calls that are not CRUD). Implementation wise it probably
> > > > simplifies what we need to do. Maybe - using Max’s early POC as an
> > > example
> > > > - we can slowly move over?
> > > > >
> > > > > On a side note: Im planning to hire 2-3 ppl to work on Airflow
> coming
> > > > year. Improvement of Security, Enterprise Integration, Revamp UI are
> on
> > > the
> > > > todo list. However, this is not confirmed yet as business priorities
> > > might
> > > > change.
> > > > >
> > > > > Bolke.
> > > > >
> > > > >
> > > > >> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> > > > kalpeshdharwadkar@gmail.com> wrote:
> > > > >>
> > > > >> @Dan:
> > > > >>
> > > > >> Thanks for your feedback. I will remove the REFRESH_DAG
> permission.
> > > > >>
> > > > >> @Max:
> > > > >>
> > > > >> Thanks for your response.
> > > > >>
> > > > >> The scope of my proposal was just to add RBAC security feature to
> > > > Airflow
> > > > >> without replacing any existing frameworks.
> > > > >>
> > > > >> I understand that adopting FAB would serve Airflow better moving
> > > > forward,
> > > > >> however porting Airflow to using FAB significantly increases the
> > scope
> > > > of
> > > > >> the proposal and I don't have the time and expertise to carry out
> > the
> > > > tasks
> > > > >> in the extended scope.
> > > > >>
> > > > >> Hence, I'm curious to know if there's a plan for Airflow to
> migrate
> > to
> > > > FAB
> > > > >> this year?
> > > > >>
> > > > >> - Kalpesh
> > > > >>
> > > > >> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
> > > > >> maximebeauchemin@gmail.com> wrote:
> > > > >>
> > > > >>> It would be nice to go with a framework for this. I did some
> > > > >>> experimentation using FlaskAppBuilder to go in this direction. It
> > > > provides
> > > > >>> auth on different authentication backends out of the box (oauth,
> > > > openid,
> > > > >>> ldap, registration, ...), generates perms for each view that has
> an
> > > > >>> @has_access decorator, generates at set of perms for each ORM
> model
> > > > (show,
> > > > >>> edit, delete, add, ...) and enforces it in the CRUD views as well
> > as
> > > > in the
> > > > >>> generated REST api that you get for free as a byprdoduct of
> > deriving
> > > > FAB's
> > > > >>> models (essentially it's SqlAlchemy with a layer on top).
> > > > >>>
> > > > >>> I started a POC on FAB here a while ago:
> > > > >>> https://github.com/mistercrunch/airflow_webserver at the time my
> > > main
> > > > >>> motivation was the free/instantaneous REST api.
> > > > >>>
> > > > >>> I think FAB is a decent fit as the porting should be fairly
> > > > straightforward
> > > > >>> (moving the flask views over and deprecating Flask-Admin in favor
> > of
> > > > FAB's
> > > > >>> crud) though there was a few blockers. From memory I think FAB
> > didn't
> > > > like
> > > > >>> the compound PKs we use in some of the Airflow models. We'd have
> to
> > > > either
> > > > >>> write a db migration script on the Airflow side, or add support
> for
> > > > >>> compound keys to FAB (I recently became a maintainer of the
> > project,
> > > > so I
> > > > >>> could help with that)
> > > > >>>
> > > > >>> The only downside of FAB is that it's not as mature as something
> > like
> > > > >>> Django, but porting to Django would surely be much more work.
> > > > >>>
> > > > >>> Then there's the flask-security suite, but that looks like a bit
> > of a
> > > > >>> patchwork to me, I guess we can pick and choose which we want to
> > use.
> > > > >>>
> > > > >>> Max
> > > > >>>
> > > > >>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> > > > >>> dan.davydov@airbnb.com.invalid> wrote:
> > > > >>>
> > > > >>>> Looks good to me in general, thanks for putting this together!
> > > > >>>>
> > > > >>>> I think the ability to integrate with external RBAC systems like
> > > LDAP
> > > > is
> > > > >>>> important (i.e. the Airflow DB should not be decoupled with the
> > RBAC
> > > > >>>> database wherever possible).
> > > > >>>>
> > > > >>>> I wouldn't be too worried about the permissions about refreshing
> > > > DAGs, as
> > > > >>>> far as I know this functionality is no longer required with the
> > new
> > > > >>>> webservers which reload state periodically, and will certainly
> be
> > > > removed
> > > > >>>> when we have a better DAG consistency story.
> > > > >>>>
> > > > >>>> I think it would also be good to think about this
> > > > proposal/implementation
> > > > >>>> and how it applied in the API-driven world (e.g. when webserver
> > hits
> > > > APIs
> > > > >>>> like /clear on behalf of users instead of running commands
> against
> > > the
> > > > >>>> database directly).
> > > > >>>>
> > > > >>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
> > bdbruin@gmail.com
> > > >
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Will respond but im traveling at the moment. Give me a few
> days.
> > > > >>>>>
> > > > >>>>> Sent from my iPhone
> > > > >>>>>
> > > > >>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> > criccomini@apache.org>
> > > > >>>> wrote:
> > > > >>>>>>
> > > > >>>>>> Hey all,
> > > > >>>>>>
> > > > >>>>>> Checking in on this. We spent a good chunk of time thinking
> > about
> > > > >>> this,
> > > > >>>>> and
> > > > >>>>>> want to move forward with it, but want to make sure we're all
> on
> > > the
> > > > >>>> same
> > > > >>>>>> page.
> > > > >>>>>>
> > > > >>>>>> Max? Bolke? Dan? Jeremiah?
> > > > >>>>>>
> > > > >>>>>> Cheers,
> > > > >>>>>> Chris
> > > > >>>>>>
> > > > >>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh dharwadkar <
> > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hello everyone,
> > > > >>>>>>>
> > > > >>>>>>> As you all know, currently Airflow doesn’t have a built-in
> Role
> > > > >>> Based
> > > > >>>>>>> Access Control(RBAC) capability.  It does provide very
> limited
> > > > >>>>>>> authorization capability by providing admin, data_profiler,
> and
> > > > user
> > > > >>>>> roles.
> > > > >>>>>>> However, associating these roles to authenticated identities
> is
> > > not
> > > > >>> a
> > > > >>>>>>> simple effort.
> > > > >>>>>>>
> > > > >>>>>>> To address this issue, I have created a design proposal for
> > > > building
> > > > >>>>> RBAC
> > > > >>>>>>> into Airflow and simplifying user access management via the
> > > Airflow
> > > > >>>> UI.
> > > > >>>>>>>
> > > > >>>>>>> The design proposal is located at https://cwiki.apache.org/
> > > > >>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
> > > > >>>>>>>
> > > > >>>>>>> Any comments/questions/feedback are much appreciated.
> > > > >>>>>>>
> > > > >>>>>>> Thanks
> > > > >>>>>>> Kalpesh
> > > > >>>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >
> > > >
> > > >
> > >
> >
>
>
>
> --
>
> Joy Gao
> Software Engineer
> 350 Convention Way, Suite 200
> Redwood City, CA 94063
> Mobile:  669-224-9305
>
> Payments partner to the platform economy
>