You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Chris Riccomini <cr...@apache.org> on 2017/08/02 17:46:58 UTC
Re: Role Based Access Control for Airflow UI
Hey all,
Unfortunately, we're going to have to reschedule this meeting. A few
critical folks ended up having to back out at the last minute. I'll send an
update when I've coordinated a new time. Ideally, I'd like to do it later
this week, but we'll see.
Cheers,
Chris
On Tue, Jul 25, 2017 at 3:48 PM, Chris Riccomini <cr...@apache.org>
wrote:
> Invite sent for 8/2 from 3pm-5pm Pacific time! Here's the dial-in info:
>
> Join from PC, Mac, Linux, iOS or Android: https://wepay.zoom.us/j/
> 994482435
>
> Or iPhone one-tap (US Toll): +16465588656 <(646)%20558-8656>,,994482435#
> or +14086380968 <(408)%20638-0968>,,994482435#
>
> Or Telephone:
> Dial: +1 646 558 8656 <(646)%20558-8656> (US Toll) or +1 408 638 0968
> <(408)%20638-0968> (US Toll)
> +1 855 880 1246 <(855)%20880-1246> (US Toll Free)
> +1 877 369 0926 <(877)%20369-0926> (US Toll Free)
> Meeting ID: 994 482 435
> International numbers available: https://wepay.zoom.us/
> zoomconference?m=wHL1YsNLTvoAf8xdE8ki9yNEPVcus_eF
>
>
> On Tue, Jul 25, 2017 at 2:53 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
>> Great! I'll send an invite.
>>
>> On Tue, Jul 25, 2017 at 1:43 PM, Dan Davydov <
>> dan.davydov@airbnb.com.invalid> wrote:
>>
>>> Same as Alex, would be great to be able to remote in though I'm very
>>> interested.
>>>
>>> On Tue, Jul 25, 2017 at 1:37 PM, Alex Guziel <alex.guziel@airbnb.com
>>> .invalid
>>> > wrote:
>>>
>>> > Yeah, I could call in but I probably won't be able to come down that
>>> day.
>>> >
>>> > On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
>>> > maximebeauchemin@gmail.com> wrote:
>>> >
>>> > > Works for me! Dan said he might confcall in. Alex?
>>> > >
>>> > > Max
>>> > >
>>> > > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <
>>> criccomini@apache.org
>>> > >
>>> > > wrote:
>>> > >
>>> > > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room
>>> for
>>> > 12
>>> > > > people (and video conferencing) at WePay in this time slot. Should
>>> > allow
>>> > > > you to head home easily afterwards. :) That work for you guys?
>>> > > >
>>> > > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
>>> > > > maximebeauchemin@gmail.com> wrote:
>>> > > >
>>> > > > > The week of the 31st sound good. Wednesday?
>>> > > > >
>>> > > > > About React we may not need a frontend lib like it (or at least
>>> not
>>> > > just
>>> > > > > yet). We can talk about it at the meeting.
>>> > > > >
>>> > > > > Max
>>> > > > >
>>> > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <
>>> bdbruin@gmail.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > > We avoid React for the same reasons as the ASF and use Polymer
>>> 2
>>> > > > instead.
>>> > > > > > Would that work?
>>> > > > > >
>>> > > > > > Bolke.
>>> > > > > >
>>> > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <
>>> criccomini@apache.org
>>> > >
>>> > > > > wrote:
>>> > > > > > >
>>> > > > > > > Hey Max,
>>> > > > > > >
>>> > > > > > > Want to come down to WePay? We can set up a zoom for those
>>> that
>>> > > want
>>> > > > to
>>> > > > > > > join online, and record it as well to post for the community.
>>> > > > > > >
>>> > > > > > > Since Joy is just getting started, and it looks like there's
>>> > going
>>> > > to
>>> > > > > be
>>> > > > > > a
>>> > > > > > > K8s discussion next week, maybe we can shoot for the week
>>> after
>>> > > (the
>>> > > > > week
>>> > > > > > > of the 31st of July)? Care to float a few times that week?
>>> > > > > > >
>>> > > > > > > Cheers,
>>> > > > > > > Chris
>>> > > > > > >
>>> > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
>>> > > > > > > maximebeauchemin@gmail.com> wrote:
>>> > > > > > >
>>> > > > > > >> Sounds awesome, count me in!
>>> > > > > > >>
>>> > > > > > >> * check out the prototype in my fork, I went far enough to
>>> hit
>>> > > some
>>> > > > > > >> hurdles, try different workarounds. I hooked up the Airflow
>>> > > > Bootstrap
>>> > > > > > >> template too so that we feel at home in this new UI
>>> > > > > > >> * using a single `id` field is a requirement for FAB that
>>> > airflow
>>> > > > > > doesn't
>>> > > > > > >> respect (composite pks), either we add the feature to
>>> support
>>> > that
>>> > > > in
>>> > > > > > FAB,
>>> > > > > > >> or we align on the Airflow side and modify the models and
>>> add a
>>> > > > > > migration
>>> > > > > > >> script. This upgrade would require downtime and might be
>>> > annoying
>>> > > to
>>> > > > > the
>>> > > > > > >> Airflow community, but could help with db performance a bit
>>> > > (smaller
>>> > > > > > >> index)... I probably could be convinced either way but I'm
>>> > leaning
>>> > > > on
>>> > > > > > >> improving FAB
>>> > > > > > >> * I'm a maintainer for FAB so I can help get stuff through
>>> there
>>> > > > > > >> * React is in limbo at the ASF for licensing reasons, so no
>>> > React
>>> > > at
>>> > > > > > least
>>> > > > > > >> for now
>>> > > > > > >> * npm/webpack/ES6, javascript only in `.js` files
>>> > > > > > >> * I vote for eslint + eslint-config-airbnb as a set of
>>> linting
>>> > > rules
>>> > > > > > for JS
>>> > > > > > >> * Keep out of apache (for now), this new app ships as its
>>> own
>>> > pypi
>>> > > > > > package
>>> > > > > > >> `airflow-webserver`, have a period of overlap (maintaining
>>> 2 web
>>> > > > apps)
>>> > > > > > >> before ripping out `airflow/www` from the core package
>>> > > > > > >> * You need to get in touch with Marty Kausas, an intern at
>>> > Airbnb
>>> > > > > who's
>>> > > > > > >> been working on a Flask blueprint for improved, more
>>> > personalized
>>> > > > > views
>>> > > > > > on
>>> > > > > > >> DAGs that we were planning on merging into the main branch
>>> > > > eventually.
>>> > > > > > Some
>>> > > > > > >> of Marty's idea and code could be merged into this effort.
>>> > > > > > >>
>>> > > > > > >> These are ideas on how I would proceed personally on this
>>> but
>>> > > > > definitely
>>> > > > > > >> everything here is up for discussion.
>>> > > > > > >>
>>> > > > > > >> Let's meet physically at either WePay or Airbnb. Folks from
>>> the
>>> > > > > > community,
>>> > > > > > >> let us know on this thread if you want to be part of this
>>> > effort,
>>> > > > > we'll
>>> > > > > > be
>>> > > > > > >> happy to include you.
>>> > > > > > >>
>>> > > > > > >> Thanks,
>>> > > > > > >>
>>> > > > > > >> Max
>>> > > > > > >>
>>> > > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com>
>>> > wrote:
>>> > > > > > >>
>>> > > > > > >>> Hey everyone,
>>> > > > > > >>>
>>> > > > > > >>> I recently transferred to Data Infra team here at WePay to
>>> > focus
>>> > > on
>>> > > > > > >>> Airflow-related initiatives.
>>> > > > > > >>>
>>> > > > > > >>> Given the RBAC design is mostly hashed out, I'm happy to
>>> get
>>> > this
>>> > > > > > feature
>>> > > > > > >>> off the ground for Q3, starting with converting Airflow to
>>> Fab,
>>> > > if
>>> > > > > > there
>>> > > > > > >>> are no objections.
>>> > > > > > >>>
>>> > > > > > >>> Cheers,
>>> > > > > > >>> Joy
>>> > > > > > >>>
>>> > > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
>>> > > > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
>>> > > > > > >>>
>>> > > > > > >>>> Hey all,
>>> > > > > > >>>>
>>> > > > > > >>>> We talked about this internally. We would like to work on
>>> this
>>> > > > > feature
>>> > > > > > >>> but
>>> > > > > > >>>> given the immediate priorities we are not going to be
>>> working
>>> > on
>>> > > > it
>>> > > > > in
>>> > > > > > >>> Q3.
>>> > > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is we
>>> can
>>> > > work
>>> > > > > on
>>> > > > > > >> it
>>> > > > > > >>>> late Q4 or Q12018.
>>> > > > > > >>>>
>>> > > > > > >>>> Cheers,
>>> > > > > > >>>>
>>> > > > > > >>>> Gurer
>>> > > > > > >>>>
>>> > > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
>>> > > > > > >> criccomini@apache.org>
>>> > > > > > >>>> wrote:
>>> > > > > > >>>>
>>> > > > > > >>>>> I think FAB sounds like the right approach. Waiting to
>>> hear
>>> > > back
>>> > > > > with
>>> > > > > > >>>> notes
>>> > > > > > >>>>> on AirBNB H2 discussion to see if they want to take this
>>> up.
>>> > > > > > >>>>>
>>> > > > > > >>>>> @Gurer, any idea when this will happen?
>>> > > > > > >>>>>
>>> > > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
>>> > > > bdbruin@gmail.com
>>> > > > > >
>>> > > > > > >>>> wrote:
>>> > > > > > >>>>>
>>> > > > > > >>>>>> One downside I see from FAB is that is does not do
>>> Business
>>> > > Role
>>> > > > > > >>>> mapping
>>> > > > > > >>>>>> to FAB role. I would prefer to create groups in
>>> IPA/LDAP/AD
>>> > > and
>>> > > > > > >> have
>>> > > > > > >>>>> those
>>> > > > > > >>>>>> map to FAB roles instead of needing to manage that in
>>> FAB.
>>> > > > > > >>>>>>
>>> > > > > > >>>>>> B.
>>> > > > > > >>>>>>
>>> > > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <
>>> > bdbruin@gmail.com>
>>> > > > > > >>> wrote:
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> Hi Guys,
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> Thanks for putting the thinking in! It is about time
>>> that
>>> > we
>>> > > > get
>>> > > > > > >>> this
>>> > > > > > >>>>>> moving.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> The design looks pretty sound. One can argue about the
>>> > > > different
>>> > > > > > >>>> roles
>>> > > > > > >>>>>> that are required, but that will be situation dependent
>>> I
>>> > > guess.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> Implementation wise I would argue together with Max
>>> that
>>> > FAB
>>> > > > is a
>>> > > > > > >>>>> better
>>> > > > > > >>>>>> or best fit. The ER model that is being described is
>>> pretty
>>> > > > much a
>>> > > > > > >>> copy
>>> > > > > > >>>>> of
>>> > > > > > >>>>>> a normal security model. So a reimplementation of that
>>> is 1)
>>> > > > > > >>>> significant
>>> > > > > > >>>>>> duplication of effort and 2) bound to have bugs that
>>> have
>>> > been
>>> > > > > > >> solved
>>> > > > > > >>>> in
>>> > > > > > >>>>>> the other framework. Moreover, FAB does have
>>> integration out
>>> > > of
>>> > > > > the
>>> > > > > > >>> box
>>> > > > > > >>>>>> with some enterprisey systems like IPA,
>>> ActiveDirectory, and
>>> > > > LDAP.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> So while you argue that using FAB would increase the
>>> scope
>>> > of
>>> > > > the
>>> > > > > > >>>>>> proposal significantly, but I think that is not true.
>>> Using
>>> > > FAB
>>> > > > > > >> would
>>> > > > > > >>>>> allow
>>> > > > > > >>>>>> you to focus on what kind of out-of-the-box permission
>>> sets
>>> > > and
>>> > > > > > >> roles
>>> > > > > > >>>> we
>>> > > > > > >>>>>> would need and maybe address some issues that FAB lacks
>>> > (maybe
>>> > > > how
>>> > > > > > >> to
>>> > > > > > >>>>> deal
>>> > > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos,
>>> probably
>>> > > how
>>> > > > to
>>> > > > > > >>> deal
>>> > > > > > >>>>>> with API calls that are not CRUD). Implementation wise
>>> it
>>> > > > probably
>>> > > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s
>>> early POC
>>> > > as
>>> > > > an
>>> > > > > > >>>>> example
>>> > > > > > >>>>>> - we can slowly move over?
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on
>>> > > Airflow
>>> > > > > > >>> coming
>>> > > > > > >>>>>> year. Improvement of Security, Enterprise Integration,
>>> > Revamp
>>> > > UI
>>> > > > > > >> are
>>> > > > > > >>> on
>>> > > > > > >>>>> the
>>> > > > > > >>>>>> todo list. However, this is not confirmed yet as
>>> business
>>> > > > > > >> priorities
>>> > > > > > >>>>> might
>>> > > > > > >>>>>> change.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>> Bolke.
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
>>> > > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> @Dan:
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> Thanks for your feedback. I will remove the
>>> REFRESH_DAG
>>> > > > > > >>> permission.
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> @Max:
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> Thanks for your response.
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> The scope of my proposal was just to add RBAC security
>>> > > feature
>>> > > > > > >> to
>>> > > > > > >>>>>> Airflow
>>> > > > > > >>>>>>>> without replacing any existing frameworks.
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> I understand that adopting FAB would serve Airflow
>>> better
>>> > > > moving
>>> > > > > > >>>>>> forward,
>>> > > > > > >>>>>>>> however porting Airflow to using FAB significantly
>>> > increases
>>> > > > the
>>> > > > > > >>>> scope
>>> > > > > > >>>>>> of
>>> > > > > > >>>>>>>> the proposal and I don't have the time and expertise
>>> to
>>> > > carry
>>> > > > > > >> out
>>> > > > > > >>>> the
>>> > > > > > >>>>>> tasks
>>> > > > > > >>>>>>>> in the extended scope.
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for
>>> Airflow
>>> > to
>>> > > > > > >>> migrate
>>> > > > > > >>>> to
>>> > > > > > >>>>>> FAB
>>> > > > > > >>>>>>>> this year?
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> - Kalpesh
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
>>> > > > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
>>> > > > > > >>>>>>>>
>>> > > > > > >>>>>>>>> It would be nice to go with a framework for this. I
>>> did
>>> > > some
>>> > > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this
>>> > > > direction.
>>> > > > > > >> It
>>> > > > > > >>>>>> provides
>>> > > > > > >>>>>>>>> auth on different authentication backends out of the
>>> box
>>> > > > > > >> (oauth,
>>> > > > > > >>>>>> openid,
>>> > > > > > >>>>>>>>> ldap, registration, ...), generates perms for each
>>> view
>>> > > that
>>> > > > > > >> has
>>> > > > > > >>> an
>>> > > > > > >>>>>>>>> @has_access decorator, generates at set of perms for
>>> each
>>> > > ORM
>>> > > > > > >>> model
>>> > > > > > >>>>>> (show,
>>> > > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD
>>> views
>>> > > as
>>> > > > > > >> well
>>> > > > > > >>>> as
>>> > > > > > >>>>>> in the
>>> > > > > > >>>>>>>>> generated REST api that you get for free as a
>>> byprdoduct
>>> > of
>>> > > > > > >>>> deriving
>>> > > > > > >>>>>> FAB's
>>> > > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on
>>> top).
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> I started a POC on FAB here a while ago:
>>> > > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver
>>> at the
>>> > > > time
>>> > > > > > >> my
>>> > > > > > >>>>> main
>>> > > > > > >>>>>>>>> motivation was the free/instantaneous REST api.
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> I think FAB is a decent fit as the porting should be
>>> > fairly
>>> > > > > > >>>>>> straightforward
>>> > > > > > >>>>>>>>> (moving the flask views over and deprecating
>>> Flask-Admin
>>> > in
>>> > > > > > >> favor
>>> > > > > > >>>> of
>>> > > > > > >>>>>> FAB's
>>> > > > > > >>>>>>>>> crud) though there was a few blockers. From memory I
>>> > think
>>> > > > FAB
>>> > > > > > >>>> didn't
>>> > > > > > >>>>>> like
>>> > > > > > >>>>>>>>> the compound PKs we use in some of the Airflow
>>> models.
>>> > We'd
>>> > > > > > >> have
>>> > > > > > >>> to
>>> > > > > > >>>>>> either
>>> > > > > > >>>>>>>>> write a db migration script on the Airflow side, or
>>> add
>>> > > > support
>>> > > > > > >>> for
>>> > > > > > >>>>>>>>> compound keys to FAB (I recently became a maintainer
>>> of
>>> > the
>>> > > > > > >>>> project,
>>> > > > > > >>>>>> so I
>>> > > > > > >>>>>>>>> could help with that)
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> The only downside of FAB is that it's not as mature
>>> as
>>> > > > > > >> something
>>> > > > > > >>>> like
>>> > > > > > >>>>>>>>> Django, but porting to Django would surely be much
>>> more
>>> > > work.
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> Then there's the flask-security suite, but that looks
>>> > like
>>> > > a
>>> > > > > > >> bit
>>> > > > > > >>>> of a
>>> > > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose
>>> which we
>>> > > want
>>> > > > > > >> to
>>> > > > > > >>>> use.
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> Max
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
>>> > > > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>>>> Looks good to me in general, thanks for putting this
>>> > > > together!
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>> I think the ability to integrate with external RBAC
>>> > > systems
>>> > > > > > >> like
>>> > > > > > >>>>> LDAP
>>> > > > > > >>>>>> is
>>> > > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be
>>> decoupled
>>> > > with
>>> > > > > > >> the
>>> > > > > > >>>> RBAC
>>> > > > > > >>>>>>>>>> database wherever possible).
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>> I wouldn't be too worried about the permissions
>>> about
>>> > > > > > >> refreshing
>>> > > > > > >>>>>> DAGs, as
>>> > > > > > >>>>>>>>>> far as I know this functionality is no longer
>>> required
>>> > > with
>>> > > > > > >> the
>>> > > > > > >>>> new
>>> > > > > > >>>>>>>>>> webservers which reload state periodically, and will
>>> > > > certainly
>>> > > > > > >>> be
>>> > > > > > >>>>>> removed
>>> > > > > > >>>>>>>>>> when we have a better DAG consistency story.
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>> I think it would also be good to think about this
>>> > > > > > >>>>>> proposal/implementation
>>> > > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g.
>>> when
>>> > > > > > >> webserver
>>> > > > > > >>>> hits
>>> > > > > > >>>>>> APIs
>>> > > > > > >>>>>>>>>> like /clear on behalf of users instead of running
>>> > commands
>>> > > > > > >>> against
>>> > > > > > >>>>> the
>>> > > > > > >>>>>>>>>> database directly).
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
>>> > > > > > >>>> bdbruin@gmail.com
>>> > > > > > >>>>>>
>>> > > > > > >>>>>>>>>> wrote:
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give
>>> me a
>>> > > few
>>> > > > > > >>> days.
>>> > > > > > >>>>>>>>>>>
>>> > > > > > >>>>>>>>>>> Sent from my iPhone
>>> > > > > > >>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
>>> > > > > > >>>> criccomini@apache.org>
>>> > > > > > >>>>>>>>>> wrote:
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> Hey all,
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of time
>>> > > > thinking
>>> > > > > > >>>> about
>>> > > > > > >>>>>>>>> this,
>>> > > > > > >>>>>>>>>>> and
>>> > > > > > >>>>>>>>>>>> want to move forward with it, but want to make
>>> sure
>>> > > we're
>>> > > > > > >> all
>>> > > > > > >>> on
>>> > > > > > >>>>> the
>>> > > > > > >>>>>>>>>> same
>>> > > > > > >>>>>>>>>>>> page.
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> Cheers,
>>> > > > > > >>>>>>>>>>>> Chris
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh
>>> dharwadkar <
>>> > > > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>> > > > > > >>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> Hello everyone,
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have a
>>> > > > built-in
>>> > > > > > >>> Role
>>> > > > > > >>>>>>>>> Based
>>> > > > > > >>>>>>>>>>>>> Access Control(RBAC) capability. It does provide
>>> > very
>>> > > > > > >>> limited
>>> > > > > > >>>>>>>>>>>>> authorization capability by providing admin,
>>> > > > data_profiler,
>>> > > > > > >>> and
>>> > > > > > >>>>>> user
>>> > > > > > >>>>>>>>>>> roles.
>>> > > > > > >>>>>>>>>>>>> However, associating these roles to authenticated
>>> > > > > > >> identities
>>> > > > > > >>> is
>>> > > > > > >>>>> not
>>> > > > > > >>>>>>>>> a
>>> > > > > > >>>>>>>>>>>>> simple effort.
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> To address this issue, I have created a design
>>> > proposal
>>> > > > for
>>> > > > > > >>>>>> building
>>> > > > > > >>>>>>>>>>> RBAC
>>> > > > > > >>>>>>>>>>>>> into Airflow and simplifying user access
>>> management
>>> > via
>>> > > > the
>>> > > > > > >>>>> Airflow
>>> > > > > > >>>>>>>>>> UI.
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> The design proposal is located at
>>> > > > > > >> https://cwiki.apache.org/
>>> > > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Airflow+RBAC+proposal
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much
>>> appreciated.
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>>> Thanks
>>> > > > > > >>>>>>>>>>>>> Kalpesh
>>> > > > > > >>>>>>>>>>>>>
>>> > > > > > >>>>>>>>>>>
>>> > > > > > >>>>>>>>>>
>>> > > > > > >>>>>>>>>
>>> > > > > > >>>>>>>
>>> > > > > > >>>>>>
>>> > > > > > >>>>>>
>>> > > > > > >>>>>
>>> > > > > > >>>>
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>> --
>>> > > > > > >>>
>>> > > > > > >>> Joy Gao
>>> > > > > > >>> Software Engineer
>>> > > > > > >>> 350 Convention Way, Suite 200
>>> > > > > > >>> Redwood City, CA 94063
>>> > > > > > >>> Mobile: 669-224-9305
>>> > > > > > >>>
>>> > > > > > >>> Payments partner to the platform economy
>>> > > > > > >>>
>>> > > > > > >>
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: Role Based Access Control for Airflow UI
Posted by Joy Gao <jo...@wepay.com>.
Hi all,
We had a good discussion about next steps on implementing RBAC this
morning, here is the current agreed-upon approach:
1. I will be working on prototyping the FAB-based webserver on a forked
Apache-Airflow repo. This will reside in the directory
apache-airflow/airflow/www2/.
2. I will also create a fork of Flask-AppBuilder to fix any potential
limitations in current version of FAB
3. This FAB-based prototype will have its own models.py, views.py, and
config.py
1. The prototype will cover all ModelViews + the home page view only
2. The goal is feature parity: there will be no redesign of UI
3. The new models.py will only have the ORM mapping of each existing
model; while the new views will use the old models' methods to
prevent code
duplication
4. Once the prototype is ready I’ll create a PR against Apache-Airflow’s
master branch. This will be dependent on the FAB-related fixes getting
merged and released to FAB
5. Going forward, the remaining views will be done on the master branch,
this encourages all new features on the views to be compatible with both
versions of webservers going forward.
6. Eventually, once we reach complete feature parity on all the views
and add in the RBAC support, we can deprecate www/ in favor of www2/, and
point `airflow webserver` to the FAB-based webserver.
Some additional notes:
- The reason we want to develop the FAB webserver on the master branch
as soon as possible instead of creating a separate pypi package outside of
Apache is to avoid the situation where the new fork deviates so much from
the master branch that it becomes risky to merge it back in
- One of the current major blockers is compound PKs support in FAB.
Without it, we’d have to do create new Primary Keys and force a DB
migration, and that is not ideal so we want to avoid that.
- FAB currently does not do Business Role (group) to FAB role mapping,
that’s something we’ll need to discuss later to see how to implement it.
We attempted to record a video of this discussion, but apparently it came
out sounding like a skrillex dubstep mix. So we'll skip that :P
Let me know if you have any questions/concerns regarding this approach.
Regards,
Joy
On Fri, Aug 4, 2017 at 1:30 PM, Chris Riccomini <cr...@apache.org>
wrote:
>
> OK! I booked another meeting for Monday at 10am Pacific. Zoom link is in
> the invite but just in case:
>
> Join from PC, Mac, Linux, iOS or Android: https://wepay.zoom.us/j/
620652517
>
> Or iPhone one-tap (US Toll): +16465588656 <(646)%20558-8656>
<(646)%20558-8656>,,620652517#
> or +14086380968 <(408)%20638-0968> <(408)%20638-0968>,,620652517#
>
> Or Telephone:
> Dial: +1 646 558 8656 <(646)%20558-8656> <(646)%20558-8656> (US Toll)
or +1 408 638 0968 <(408)%20638-0968>
> <(408)%20638-0968> (US Toll)
> +1 855 880 1246 <(855)%20880-1246> <(855)%20880-1246> (US Toll Free)
> +1 877 369 0926 <(877)%20369-0926> <(877)%20369-0926> (US Toll Free)
> Meeting ID: 620 652 517
> International numbers available: https://wepay.zoom.us/
zoomconference?m=
> oRQvXKiEAD4tQlraBOLujkKBPZ_220mB
>
> On Wed, Aug 2, 2017 at 10:46 AM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > Hey all,
> >
> > Unfortunately, we're going to have to reschedule this meeting. A few
> > critical folks ended up having to back out at the last minute. I'll
send an
> > update when I've coordinated a new time. Ideally, I'd like to do it
later
> > this week, but we'll see.
> >
> > Cheers,
> > Chris
> >
> > On Tue, Jul 25, 2017 at 3:48 PM, Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> >> Invite sent for 8/2 from 3pm-5pm Pacific time! Here's the dial-in info:
> >>
> >> Join from PC, Mac, Linux, iOS or Android:
https://wepay.zoom.us/j/994482
> >> 435
> >>
> >> Or iPhone one-tap (US Toll): +16465588656 <(646)%20558-8656>
<(646)%20558-8656>,,994482435#
> >> or +14086380968 <(408)%20638-0968> <(408)%20638-0968>,,994482435#
> >>
> >> Or Telephone:
> >> Dial: +1 646 558 8656 <(646)%20558-8656> <(646)%20558-8656> (US
Toll) or +1 408 638 0968 <(408)%20638-0968>
> >> <(408)%20638-0968> (US Toll)
> >> +1 855 880 1246 <(855)%20880-1246> <(855)%20880-1246> (US Toll
Free)
> >> +1 877 369 0926 <(877)%20369-0926> <(877)%20369-0926> (US Toll
Free)
> >> Meeting ID: 994 482 435
> >> International numbers available: https://wepay.zoom.us/zoomconf
> >> erence?m=wHL1YsNLTvoAf8xdE8ki9yNEPVcus_eF
> >>
> >>
> >> On Tue, Jul 25, 2017 at 2:53 PM, Chris Riccomini <criccomini@apache.org
>
> >> wrote:
> >>
> >>> Great! I'll send an invite.
> >>>
> >>> On Tue, Jul 25, 2017 at 1:43 PM, Dan Davydov <
> >>> dan.davydov@airbnb.com.invalid> wrote:
> >>>
> >>>> Same as Alex, would be great to be able to remote in though I'm very
> >>>> interested.
> >>>>
> >>>> On Tue, Jul 25, 2017 at 1:37 PM, Alex Guziel <alex.guziel@airbnb.com
> >>>> .invalid
> >>>> > wrote:
> >>>>
> >>>> > Yeah, I could call in but I probably won't be able to come down
that
> >>>> day.
> >>>> >
> >>>> > On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
> >>>> > maximebeauchemin@gmail.com> wrote:
> >>>> >
> >>>> > > Works for me! Dan said he might confcall in. Alex?
> >>>> > >
> >>>> > > Max
> >>>> > >
> >>>> > > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <
> >>>> criccomini@apache.org
> >>>> > >
> >>>> > > wrote:
> >>>> > >
> >>>> > > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a
room
> >>>> for
> >>>> > 12
> >>>> > > > people (and video conferencing) at WePay in this time slot.
Should
> >>>> > allow
> >>>> > > > you to head home easily afterwards. :) That work for you guys?
> >>>> > > >
> >>>> > > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
> >>>> > > > maximebeauchemin@gmail.com> wrote:
> >>>> > > >
> >>>> > > > > The week of the 31st sound good. Wednesday?
> >>>> > > > >
> >>>> > > > > About React we may not need a frontend lib like it (or at
least
> >>>> not
> >>>> > > just
> >>>> > > > > yet). We can talk about it at the meeting.
> >>>> > > > >
> >>>> > > > > Max
> >>>> > > > >
> >>>> > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <
> >>>> bdbruin@gmail.com>
> >>>> > > > > wrote:
> >>>> > > > >
> >>>> > > > > > We avoid React for the same reasons as the ASF and use
> >>>> Polymer 2
> >>>> > > > instead.
> >>>> > > > > > Would that work?
> >>>> > > > > >
> >>>> > > > > > Bolke.
> >>>> > > > > >
> >>>> > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <
> >>>> criccomini@apache.org
> >>>> > >
> >>>> > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > Hey Max,
> >>>> > > > > > >
> >>>> > > > > > > Want to come down to WePay? We can set up a zoom for
those
> >>>> that
> >>>> > > want
> >>>> > > > to
> >>>> > > > > > > join online, and record it as well to post for the
> >>>> community.
> >>>> > > > > > >
> >>>> > > > > > > Since Joy is just getting started, and it looks like
there's
> >>>> > going
> >>>> > > to
> >>>> > > > > be
> >>>> > > > > > a
> >>>> > > > > > > K8s discussion next week, maybe we can shoot for the week
> >>>> after
> >>>> > > (the
> >>>> > > > > week
> >>>> > > > > > > of the 31st of July)? Care to float a few times that
week?
> >>>> > > > > > >
> >>>> > > > > > > Cheers,
> >>>> > > > > > > Chris
> >>>> > > > > > >
> >>>> > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
> >>>> > > > > > > maximebeauchemin@gmail.com> wrote:
> >>>> > > > > > >
> >>>> > > > > > >> Sounds awesome, count me in!
> >>>> > > > > > >>
> >>>> > > > > > >> * check out the prototype in my fork, I went far enough
to
> >>>> hit
> >>>> > > some
> >>>> > > > > > >> hurdles, try different workarounds. I hooked up the
Airflow
> >>>> > > > Bootstrap
> >>>> > > > > > >> template too so that we feel at home in this new UI
> >>>> > > > > > >> * using a single `id` field is a requirement for FAB
that
> >>>> > airflow
> >>>> > > > > > doesn't
> >>>> > > > > > >> respect (composite pks), either we add the feature to
> >>>> support
> >>>> > that
> >>>> > > > in
> >>>> > > > > > FAB,
> >>>> > > > > > >> or we align on the Airflow side and modify the models
and
> >>>> add a
> >>>> > > > > > migration
> >>>> > > > > > >> script. This upgrade would require downtime and might be
> >>>> > annoying
> >>>> > > to
> >>>> > > > > the
> >>>> > > > > > >> Airflow community, but could help with db performance a
bit
> >>>> > > (smaller
> >>>> > > > > > >> index)... I probably could be convinced either way but
I'm
> >>>> > leaning
> >>>> > > > on
> >>>> > > > > > >> improving FAB
> >>>> > > > > > >> * I'm a maintainer for FAB so I can help get stuff
through
> >>>> there
> >>>> > > > > > >> * React is in limbo at the ASF for licensing reasons,
so no
> >>>> > React
> >>>> > > at
> >>>> > > > > > least
> >>>> > > > > > >> for now
> >>>> > > > > > >> * npm/webpack/ES6, javascript only in `.js` files
> >>>> > > > > > >> * I vote for eslint + eslint-config-airbnb as a set of
> >>>> linting
> >>>> > > rules
> >>>> > > > > > for JS
> >>>> > > > > > >> * Keep out of apache (for now), this new app ships as
its
> >>>> own
> >>>> > pypi
> >>>> > > > > > package
> >>>> > > > > > >> `airflow-webserver`, have a period of overlap
(maintaining
> >>>> 2 web
> >>>> > > > apps)
> >>>> > > > > > >> before ripping out `airflow/www` from the core package
> >>>> > > > > > >> * You need to get in touch with Marty Kausas, an intern
at
> >>>> > Airbnb
> >>>> > > > > who's
> >>>> > > > > > >> been working on a Flask blueprint for improved, more
> >>>> > personalized
> >>>> > > > > views
> >>>> > > > > > on
> >>>> > > > > > >> DAGs that we were planning on merging into the main
branch
> >>>> > > > eventually.
> >>>> > > > > > Some
> >>>> > > > > > >> of Marty's idea and code could be merged into this
effort.
> >>>> > > > > > >>
> >>>> > > > > > >> These are ideas on how I would proceed personally on
this
> >>>> but
> >>>> > > > > definitely
> >>>> > > > > > >> everything here is up for discussion.
> >>>> > > > > > >>
> >>>> > > > > > >> Let's meet physically at either WePay or Airbnb. Folks
> >>>> from the
> >>>> > > > > > community,
> >>>> > > > > > >> let us know on this thread if you want to be part of
this
> >>>> > effort,
> >>>> > > > > we'll
> >>>> > > > > > be
> >>>> > > > > > >> happy to include you.
> >>>> > > > > > >>
> >>>> > > > > > >> Thanks,
> >>>> > > > > > >>
> >>>> > > > > > >> Max
> >>>> > > > > > >>
> >>>> > > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <joyg@wepay.com
>
> >>>> > wrote:
> >>>> > > > > > >>
> >>>> > > > > > >>> Hey everyone,
> >>>> > > > > > >>>
> >>>> > > > > > >>> I recently transferred to Data Infra team here at
WePay to
> >>>> > focus
> >>>> > > on
> >>>> > > > > > >>> Airflow-related initiatives.
> >>>> > > > > > >>>
> >>>> > > > > > >>> Given the RBAC design is mostly hashed out, I'm happy
to
> >>>> get
> >>>> > this
> >>>> > > > > > feature
> >>>> > > > > > >>> off the ground for Q3, starting with converting Airflow
> >>>> to Fab,
> >>>> > > if
> >>>> > > > > > there
> >>>> > > > > > >>> are no objections.
> >>>> > > > > > >>>
> >>>> > > > > > >>> Cheers,
> >>>> > > > > > >>> Joy
> >>>> > > > > > >>>
> >>>> > > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
> >>>> > > > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
> >>>> > > > > > >>>
> >>>> > > > > > >>>> Hey all,
> >>>> > > > > > >>>>
> >>>> > > > > > >>>> We talked about this internally. We would like to work
> >>>> on this
> >>>> > > > > feature
> >>>> > > > > > >>> but
> >>>> > > > > > >>>> given the immediate priorities we are not going to be
> >>>> working
> >>>> > on
> >>>> > > > it
> >>>> > > > > in
> >>>> > > > > > >>> Q3.
> >>>> > > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is
> >>>> we can
> >>>> > > work
> >>>> > > > > on
> >>>> > > > > > >> it
> >>>> > > > > > >>>> late Q4 or Q12018.
> >>>> > > > > > >>>>
> >>>> > > > > > >>>> Cheers,
> >>>> > > > > > >>>>
> >>>> > > > > > >>>> Gurer
> >>>> > > > > > >>>>
> >>>> > > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
> >>>> > > > > > >> criccomini@apache.org>
> >>>> > > > > > >>>> wrote:
> >>>> > > > > > >>>>
> >>>> > > > > > >>>>> I think FAB sounds like the right approach. Waiting
to
> >>>> hear
> >>>> > > back
> >>>> > > > > with
> >>>> > > > > > >>>> notes
> >>>> > > > > > >>>>> on AirBNB H2 discussion to see if they want to take
> >>>> this up.
> >>>> > > > > > >>>>>
> >>>> > > > > > >>>>> @Gurer, any idea when this will happen?
> >>>> > > > > > >>>>>
> >>>> > > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
> >>>> > > > bdbruin@gmail.com
> >>>> > > > > >
> >>>> > > > > > >>>> wrote:
> >>>> > > > > > >>>>>
> >>>> > > > > > >>>>>> One downside I see from FAB is that is does not do
> >>>> Business
> >>>> > > Role
> >>>> > > > > > >>>> mapping
> >>>> > > > > > >>>>>> to FAB role. I would prefer to create groups in
> >>>> IPA/LDAP/AD
> >>>> > > and
> >>>> > > > > > >> have
> >>>> > > > > > >>>>> those
> >>>> > > > > > >>>>>> map to FAB roles instead of needing to manage that
in
> >>>> FAB.
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>> B.
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <
> >>>> > bdbruin@gmail.com>
> >>>> > > > > > >>> wrote:
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> Hi Guys,
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> Thanks for putting the thinking in! It is about
time
> >>>> that
> >>>> > we
> >>>> > > > get
> >>>> > > > > > >>> this
> >>>> > > > > > >>>>>> moving.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> The design looks pretty sound. One can argue about
the
> >>>> > > > different
> >>>> > > > > > >>>> roles
> >>>> > > > > > >>>>>> that are required, but that will be situation
> >>>> dependent I
> >>>> > > guess.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> Implementation wise I would argue together with Max
> >>>> that
> >>>> > FAB
> >>>> > > > is a
> >>>> > > > > > >>>>> better
> >>>> > > > > > >>>>>> or best fit. The ER model that is being described is
> >>>> pretty
> >>>> > > > much a
> >>>> > > > > > >>> copy
> >>>> > > > > > >>>>> of
> >>>> > > > > > >>>>>> a normal security model. So a reimplementation of
that
> >>>> is 1)
> >>>> > > > > > >>>> significant
> >>>> > > > > > >>>>>> duplication of effort and 2) bound to have bugs that
> >>>> have
> >>>> > been
> >>>> > > > > > >> solved
> >>>> > > > > > >>>> in
> >>>> > > > > > >>>>>> the other framework. Moreover, FAB does have
> >>>> integration out
> >>>> > > of
> >>>> > > > > the
> >>>> > > > > > >>> box
> >>>> > > > > > >>>>>> with some enterprisey systems like IPA,
> >>>> ActiveDirectory, and
> >>>> > > > LDAP.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> So while you argue that using FAB would increase
the
> >>>> scope
> >>>> > of
> >>>> > > > the
> >>>> > > > > > >>>>>> proposal significantly, but I think that is not
true.
> >>>> Using
> >>>> > > FAB
> >>>> > > > > > >> would
> >>>> > > > > > >>>>> allow
> >>>> > > > > > >>>>>> you to focus on what kind of out-of-the-box
permission
> >>>> sets
> >>>> > > and
> >>>> > > > > > >> roles
> >>>> > > > > > >>>> we
> >>>> > > > > > >>>>>> would need and maybe address some issues that FAB
lacks
> >>>> > (maybe
> >>>> > > > how
> >>>> > > > > > >> to
> >>>> > > > > > >>>>> deal
> >>>> > > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos,
> >>>> probably
> >>>> > > how
> >>>> > > > to
> >>>> > > > > > >>> deal
> >>>> > > > > > >>>>>> with API calls that are not CRUD). Implementation
wise
> >>>> it
> >>>> > > > probably
> >>>> > > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s
> >>>> early POC
> >>>> > > as
> >>>> > > > an
> >>>> > > > > > >>>>> example
> >>>> > > > > > >>>>>> - we can slowly move over?
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to
work on
> >>>> > > Airflow
> >>>> > > > > > >>> coming
> >>>> > > > > > >>>>>> year. Improvement of Security, Enterprise
Integration,
> >>>> > Revamp
> >>>> > > UI
> >>>> > > > > > >> are
> >>>> > > > > > >>> on
> >>>> > > > > > >>>>> the
> >>>> > > > > > >>>>>> todo list. However, this is not confirmed yet as
> >>>> business
> >>>> > > > > > >> priorities
> >>>> > > > > > >>>>> might
> >>>> > > > > > >>>>>> change.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>> Bolke.
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
> >>>> > > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> @Dan:
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> Thanks for your feedback. I will remove the
> >>>> REFRESH_DAG
> >>>> > > > > > >>> permission.
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> @Max:
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> Thanks for your response.
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> The scope of my proposal was just to add RBAC
> >>>> security
> >>>> > > feature
> >>>> > > > > > >> to
> >>>> > > > > > >>>>>> Airflow
> >>>> > > > > > >>>>>>>> without replacing any existing frameworks.
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> I understand that adopting FAB would serve Airflow
> >>>> better
> >>>> > > > moving
> >>>> > > > > > >>>>>> forward,
> >>>> > > > > > >>>>>>>> however porting Airflow to using FAB significantly
> >>>> > increases
> >>>> > > > the
> >>>> > > > > > >>>> scope
> >>>> > > > > > >>>>>> of
> >>>> > > > > > >>>>>>>> the proposal and I don't have the time and
expertise
> >>>> to
> >>>> > > carry
> >>>> > > > > > >> out
> >>>> > > > > > >>>> the
> >>>> > > > > > >>>>>> tasks
> >>>> > > > > > >>>>>>>> in the extended scope.
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for
> >>>> Airflow
> >>>> > to
> >>>> > > > > > >>> migrate
> >>>> > > > > > >>>> to
> >>>> > > > > > >>>>>> FAB
> >>>> > > > > > >>>>>>>> this year?
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> - Kalpesh
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime
Beauchemin <
> >>>> > > > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
> >>>> > > > > > >>>>>>>>
> >>>> > > > > > >>>>>>>>> It would be nice to go with a framework for
this. I
> >>>> did
> >>>> > > some
> >>>> > > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in
this
> >>>> > > > direction.
> >>>> > > > > > >> It
> >>>> > > > > > >>>>>> provides
> >>>> > > > > > >>>>>>>>> auth on different authentication backends out of
> >>>> the box
> >>>> > > > > > >> (oauth,
> >>>> > > > > > >>>>>> openid,
> >>>> > > > > > >>>>>>>>> ldap, registration, ...), generates perms for
each
> >>>> view
> >>>> > > that
> >>>> > > > > > >> has
> >>>> > > > > > >>> an
> >>>> > > > > > >>>>>>>>> @has_access decorator, generates at set of perms
> >>>> for each
> >>>> > > ORM
> >>>> > > > > > >>> model
> >>>> > > > > > >>>>>> (show,
> >>>> > > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the
CRUD
> >>>> views
> >>>> > > as
> >>>> > > > > > >> well
> >>>> > > > > > >>>> as
> >>>> > > > > > >>>>>> in the
> >>>> > > > > > >>>>>>>>> generated REST api that you get for free as a
> >>>> byprdoduct
> >>>> > of
> >>>> > > > > > >>>> deriving
> >>>> > > > > > >>>>>> FAB's
> >>>> > > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer
on
> >>>> top).
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> I started a POC on FAB here a while ago:
> >>>> > > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver
> >>>> at the
> >>>> > > > time
> >>>> > > > > > >> my
> >>>> > > > > > >>>>> main
> >>>> > > > > > >>>>>>>>> motivation was the free/instantaneous REST api.
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> I think FAB is a decent fit as the porting
should be
> >>>> > fairly
> >>>> > > > > > >>>>>> straightforward
> >>>> > > > > > >>>>>>>>> (moving the flask views over and deprecating
> >>>> Flask-Admin
> >>>> > in
> >>>> > > > > > >> favor
> >>>> > > > > > >>>> of
> >>>> > > > > > >>>>>> FAB's
> >>>> > > > > > >>>>>>>>> crud) though there was a few blockers. From
memory I
> >>>> > think
> >>>> > > > FAB
> >>>> > > > > > >>>> didn't
> >>>> > > > > > >>>>>> like
> >>>> > > > > > >>>>>>>>> the compound PKs we use in some of the Airflow
> >>>> models.
> >>>> > We'd
> >>>> > > > > > >> have
> >>>> > > > > > >>> to
> >>>> > > > > > >>>>>> either
> >>>> > > > > > >>>>>>>>> write a db migration script on the Airflow side,
or
> >>>> add
> >>>> > > > support
> >>>> > > > > > >>> for
> >>>> > > > > > >>>>>>>>> compound keys to FAB (I recently became a
> >>>> maintainer of
> >>>> > the
> >>>> > > > > > >>>> project,
> >>>> > > > > > >>>>>> so I
> >>>> > > > > > >>>>>>>>> could help with that)
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> The only downside of FAB is that it's not as
mature
> >>>> as
> >>>> > > > > > >> something
> >>>> > > > > > >>>> like
> >>>> > > > > > >>>>>>>>> Django, but porting to Django would surely be
much
> >>>> more
> >>>> > > work.
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> Then there's the flask-security suite, but that
> >>>> looks
> >>>> > like
> >>>> > > a
> >>>> > > > > > >> bit
> >>>> > > > > > >>>> of a
> >>>> > > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose
> >>>> which we
> >>>> > > want
> >>>> > > > > > >> to
> >>>> > > > > > >>>> use.
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> Max
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
> >>>> > > > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>>>> Looks good to me in general, thanks for putting
> >>>> this
> >>>> > > > together!
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>> I think the ability to integrate with external
RBAC
> >>>> > > systems
> >>>> > > > > > >> like
> >>>> > > > > > >>>>> LDAP
> >>>> > > > > > >>>>>> is
> >>>> > > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be
> >>>> decoupled
> >>>> > > with
> >>>> > > > > > >> the
> >>>> > > > > > >>>> RBAC
> >>>> > > > > > >>>>>>>>>> database wherever possible).
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>> I wouldn't be too worried about the permissions
> >>>> about
> >>>> > > > > > >> refreshing
> >>>> > > > > > >>>>>> DAGs, as
> >>>> > > > > > >>>>>>>>>> far as I know this functionality is no longer
> >>>> required
> >>>> > > with
> >>>> > > > > > >> the
> >>>> > > > > > >>>> new
> >>>> > > > > > >>>>>>>>>> webservers which reload state periodically, and
> >>>> will
> >>>> > > > certainly
> >>>> > > > > > >>> be
> >>>> > > > > > >>>>>> removed
> >>>> > > > > > >>>>>>>>>> when we have a better DAG consistency story.
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>> I think it would also be good to think about
this
> >>>> > > > > > >>>>>> proposal/implementation
> >>>> > > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g.
> >>>> when
> >>>> > > > > > >> webserver
> >>>> > > > > > >>>> hits
> >>>> > > > > > >>>>>> APIs
> >>>> > > > > > >>>>>>>>>> like /clear on behalf of users instead of
running
> >>>> > commands
> >>>> > > > > > >>> against
> >>>> > > > > > >>>>> the
> >>>> > > > > > >>>>>>>>>> database directly).
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de
Bruin <
> >>>> > > > > > >>>> bdbruin@gmail.com
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>>>>>> wrote:
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>> Will respond but im traveling at the moment.
Give
> >>>> me a
> >>>> > > few
> >>>> > > > > > >>> days.
> >>>> > > > > > >>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>> Sent from my iPhone
> >>>> > > > > > >>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
> >>>> > > > > > >>>> criccomini@apache.org>
> >>>> > > > > > >>>>>>>>>> wrote:
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> Hey all,
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of
> >>>> time
> >>>> > > > thinking
> >>>> > > > > > >>>> about
> >>>> > > > > > >>>>>>>>> this,
> >>>> > > > > > >>>>>>>>>>> and
> >>>> > > > > > >>>>>>>>>>>> want to move forward with it, but want to make
> >>>> sure
> >>>> > > we're
> >>>> > > > > > >> all
> >>>> > > > > > >>> on
> >>>> > > > > > >>>>> the
> >>>> > > > > > >>>>>>>>>> same
> >>>> > > > > > >>>>>>>>>>>> page.
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> Cheers,
> >>>> > > > > > >>>>>>>>>>>> Chris
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh
> >>>> dharwadkar <
> >>>> > > > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
> >>>> > > > > > >>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> Hello everyone,
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t
have
> >>>> a
> >>>> > > > built-in
> >>>> > > > > > >>> Role
> >>>> > > > > > >>>>>>>>> Based
> >>>> > > > > > >>>>>>>>>>>>> Access Control(RBAC) capability. It does
> >>>> provide
> >>>> > very
> >>>> > > > > > >>> limited
> >>>> > > > > > >>>>>>>>>>>>> authorization capability by providing admin,
> >>>> > > > data_profiler,
> >>>> > > > > > >>> and
> >>>> > > > > > >>>>>> user
> >>>> > > > > > >>>>>>>>>>> roles.
> >>>> > > > > > >>>>>>>>>>>>> However, associating these roles to
> >>>> authenticated
> >>>> > > > > > >> identities
> >>>> > > > > > >>> is
> >>>> > > > > > >>>>> not
> >>>> > > > > > >>>>>>>>> a
> >>>> > > > > > >>>>>>>>>>>>> simple effort.
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> To address this issue, I have created a
design
> >>>> > proposal
> >>>> > > > for
> >>>> > > > > > >>>>>> building
> >>>> > > > > > >>>>>>>>>>> RBAC
> >>>> > > > > > >>>>>>>>>>>>> into Airflow and simplifying user access
> >>>> management
> >>>> > via
> >>>> > > > the
> >>>> > > > > > >>>>> Airflow
> >>>> > > > > > >>>>>>>>>> UI.
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> The design proposal is located at
> >>>> > > > > > >> https://cwiki.apache.org/
> >>>> > > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Air
> >>>> flow+RBAC+proposal
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much
> >>>> appreciated.
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>>> Thanks
> >>>> > > > > > >>>>>>>>>>>>> Kalpesh
> >>>> > > > > > >>>>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>>
> >>>> > > > > > >>>>>>>>>>
> >>>> > > > > > >>>>>>>>>
> >>>> > > > > > >>>>>>>
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>>
> >>>> > > > > > >>>>>
> >>>> > > > > > >>>>
> >>>> > > > > > >>>
> >>>> > > > > > >>>
> >>>> > > > > > >>>
> >>>> > > > > > >>> --
> >>>> > > > > > >>>
> >>>> > > > > > >>> Joy Gao
> >>>> > > > > > >>> Software Engineer
> >>>> > > > > > >>> 350 Convention Way, Suite 200
> >>>> > > > > > >>> Redwood City, CA 94063
> >>>> > > > > > >>> Mobile: 669-224-9305 <(669)%20224-9305>
> >>>> > > > > > >>>
> >>>> > > > > > >>> Payments partner to the platform economy
> >>>> > > > > > >>>
> >>>> > > > > > >>
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >
Re: Role Based Access Control for Airflow UI
Posted by Chris Riccomini <cr...@apache.org>.
OK! I booked another meeting for Monday at 10am Pacific. Zoom link is in
the invite but just in case:
Join from PC, Mac, Linux, iOS or Android: https://wepay.zoom.us/j/620652517
Or iPhone one-tap (US Toll): +16465588656 <(646)%20558-8656>,,620652517#
or +14086380968 <(408)%20638-0968>,,620652517#
Or Telephone:
Dial: +1 646 558 8656 <(646)%20558-8656> (US Toll) or +1 408 638 0968
<(408)%20638-0968> (US Toll)
+1 855 880 1246 <(855)%20880-1246> (US Toll Free)
+1 877 369 0926 <(877)%20369-0926> (US Toll Free)
Meeting ID: 620 652 517
International numbers available: https://wepay.zoom.us/zoomconference?m=
oRQvXKiEAD4tQlraBOLujkKBPZ_220mB
On Wed, Aug 2, 2017 at 10:46 AM, Chris Riccomini <cr...@apache.org>
wrote:
> Hey all,
>
> Unfortunately, we're going to have to reschedule this meeting. A few
> critical folks ended up having to back out at the last minute. I'll send an
> update when I've coordinated a new time. Ideally, I'd like to do it later
> this week, but we'll see.
>
> Cheers,
> Chris
>
> On Tue, Jul 25, 2017 at 3:48 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
>> Invite sent for 8/2 from 3pm-5pm Pacific time! Here's the dial-in info:
>>
>> Join from PC, Mac, Linux, iOS or Android: https://wepay.zoom.us/j/994482
>> 435
>>
>> Or iPhone one-tap (US Toll): +16465588656 <(646)%20558-8656>,,994482435#
>> or +14086380968 <(408)%20638-0968>,,994482435#
>>
>> Or Telephone:
>> Dial: +1 646 558 8656 <(646)%20558-8656> (US Toll) or +1 408 638 0968
>> <(408)%20638-0968> (US Toll)
>> +1 855 880 1246 <(855)%20880-1246> (US Toll Free)
>> +1 877 369 0926 <(877)%20369-0926> (US Toll Free)
>> Meeting ID: 994 482 435
>> International numbers available: https://wepay.zoom.us/zoomconf
>> erence?m=wHL1YsNLTvoAf8xdE8ki9yNEPVcus_eF
>>
>>
>> On Tue, Jul 25, 2017 at 2:53 PM, Chris Riccomini <cr...@apache.org>
>> wrote:
>>
>>> Great! I'll send an invite.
>>>
>>> On Tue, Jul 25, 2017 at 1:43 PM, Dan Davydov <
>>> dan.davydov@airbnb.com.invalid> wrote:
>>>
>>>> Same as Alex, would be great to be able to remote in though I'm very
>>>> interested.
>>>>
>>>> On Tue, Jul 25, 2017 at 1:37 PM, Alex Guziel <alex.guziel@airbnb.com
>>>> .invalid
>>>> > wrote:
>>>>
>>>> > Yeah, I could call in but I probably won't be able to come down that
>>>> day.
>>>> >
>>>> > On Tue, Jul 25, 2017 at 1:36 PM, Maxime Beauchemin <
>>>> > maximebeauchemin@gmail.com> wrote:
>>>> >
>>>> > > Works for me! Dan said he might confcall in. Alex?
>>>> > >
>>>> > > Max
>>>> > >
>>>> > > On Mon, Jul 24, 2017 at 11:25 AM, Chris Riccomini <
>>>> criccomini@apache.org
>>>> > >
>>>> > > wrote:
>>>> > >
>>>> > > > Wednesday 8/2 is perfect. Want to do it like 3-5? I booked a room
>>>> for
>>>> > 12
>>>> > > > people (and video conferencing) at WePay in this time slot. Should
>>>> > allow
>>>> > > > you to head home easily afterwards. :) That work for you guys?
>>>> > > >
>>>> > > > On Mon, Jul 24, 2017 at 11:01 AM, Maxime Beauchemin <
>>>> > > > maximebeauchemin@gmail.com> wrote:
>>>> > > >
>>>> > > > > The week of the 31st sound good. Wednesday?
>>>> > > > >
>>>> > > > > About React we may not need a frontend lib like it (or at least
>>>> not
>>>> > > just
>>>> > > > > yet). We can talk about it at the meeting.
>>>> > > > >
>>>> > > > > Max
>>>> > > > >
>>>> > > > > On Fri, Jul 21, 2017 at 12:47 AM, Bolke de Bruin <
>>>> bdbruin@gmail.com>
>>>> > > > > wrote:
>>>> > > > >
>>>> > > > > > We avoid React for the same reasons as the ASF and use
>>>> Polymer 2
>>>> > > > instead.
>>>> > > > > > Would that work?
>>>> > > > > >
>>>> > > > > > Bolke.
>>>> > > > > >
>>>> > > > > > > On 20 Jul 2017, at 19:35, Chris Riccomini <
>>>> criccomini@apache.org
>>>> > >
>>>> > > > > wrote:
>>>> > > > > > >
>>>> > > > > > > Hey Max,
>>>> > > > > > >
>>>> > > > > > > Want to come down to WePay? We can set up a zoom for those
>>>> that
>>>> > > want
>>>> > > > to
>>>> > > > > > > join online, and record it as well to post for the
>>>> community.
>>>> > > > > > >
>>>> > > > > > > Since Joy is just getting started, and it looks like there's
>>>> > going
>>>> > > to
>>>> > > > > be
>>>> > > > > > a
>>>> > > > > > > K8s discussion next week, maybe we can shoot for the week
>>>> after
>>>> > > (the
>>>> > > > > week
>>>> > > > > > > of the 31st of July)? Care to float a few times that week?
>>>> > > > > > >
>>>> > > > > > > Cheers,
>>>> > > > > > > Chris
>>>> > > > > > >
>>>> > > > > > > On Thu, Jul 20, 2017 at 9:31 AM, Maxime Beauchemin <
>>>> > > > > > > maximebeauchemin@gmail.com> wrote:
>>>> > > > > > >
>>>> > > > > > >> Sounds awesome, count me in!
>>>> > > > > > >>
>>>> > > > > > >> * check out the prototype in my fork, I went far enough to
>>>> hit
>>>> > > some
>>>> > > > > > >> hurdles, try different workarounds. I hooked up the Airflow
>>>> > > > Bootstrap
>>>> > > > > > >> template too so that we feel at home in this new UI
>>>> > > > > > >> * using a single `id` field is a requirement for FAB that
>>>> > airflow
>>>> > > > > > doesn't
>>>> > > > > > >> respect (composite pks), either we add the feature to
>>>> support
>>>> > that
>>>> > > > in
>>>> > > > > > FAB,
>>>> > > > > > >> or we align on the Airflow side and modify the models and
>>>> add a
>>>> > > > > > migration
>>>> > > > > > >> script. This upgrade would require downtime and might be
>>>> > annoying
>>>> > > to
>>>> > > > > the
>>>> > > > > > >> Airflow community, but could help with db performance a bit
>>>> > > (smaller
>>>> > > > > > >> index)... I probably could be convinced either way but I'm
>>>> > leaning
>>>> > > > on
>>>> > > > > > >> improving FAB
>>>> > > > > > >> * I'm a maintainer for FAB so I can help get stuff through
>>>> there
>>>> > > > > > >> * React is in limbo at the ASF for licensing reasons, so no
>>>> > React
>>>> > > at
>>>> > > > > > least
>>>> > > > > > >> for now
>>>> > > > > > >> * npm/webpack/ES6, javascript only in `.js` files
>>>> > > > > > >> * I vote for eslint + eslint-config-airbnb as a set of
>>>> linting
>>>> > > rules
>>>> > > > > > for JS
>>>> > > > > > >> * Keep out of apache (for now), this new app ships as its
>>>> own
>>>> > pypi
>>>> > > > > > package
>>>> > > > > > >> `airflow-webserver`, have a period of overlap (maintaining
>>>> 2 web
>>>> > > > apps)
>>>> > > > > > >> before ripping out `airflow/www` from the core package
>>>> > > > > > >> * You need to get in touch with Marty Kausas, an intern at
>>>> > Airbnb
>>>> > > > > who's
>>>> > > > > > >> been working on a Flask blueprint for improved, more
>>>> > personalized
>>>> > > > > views
>>>> > > > > > on
>>>> > > > > > >> DAGs that we were planning on merging into the main branch
>>>> > > > eventually.
>>>> > > > > > Some
>>>> > > > > > >> of Marty's idea and code could be merged into this effort.
>>>> > > > > > >>
>>>> > > > > > >> These are ideas on how I would proceed personally on this
>>>> but
>>>> > > > > definitely
>>>> > > > > > >> everything here is up for discussion.
>>>> > > > > > >>
>>>> > > > > > >> Let's meet physically at either WePay or Airbnb. Folks
>>>> from the
>>>> > > > > > community,
>>>> > > > > > >> let us know on this thread if you want to be part of this
>>>> > effort,
>>>> > > > > we'll
>>>> > > > > > be
>>>> > > > > > >> happy to include you.
>>>> > > > > > >>
>>>> > > > > > >> Thanks,
>>>> > > > > > >>
>>>> > > > > > >> Max
>>>> > > > > > >>
>>>> > > > > > >> On Wed, Jul 19, 2017 at 7:33 PM, Joy Gao <jo...@wepay.com>
>>>> > wrote:
>>>> > > > > > >>
>>>> > > > > > >>> Hey everyone,
>>>> > > > > > >>>
>>>> > > > > > >>> I recently transferred to Data Infra team here at WePay to
>>>> > focus
>>>> > > on
>>>> > > > > > >>> Airflow-related initiatives.
>>>> > > > > > >>>
>>>> > > > > > >>> Given the RBAC design is mostly hashed out, I'm happy to
>>>> get
>>>> > this
>>>> > > > > > feature
>>>> > > > > > >>> off the ground for Q3, starting with converting Airflow
>>>> to Fab,
>>>> > > if
>>>> > > > > > there
>>>> > > > > > >>> are no objections.
>>>> > > > > > >>>
>>>> > > > > > >>> Cheers,
>>>> > > > > > >>> Joy
>>>> > > > > > >>>
>>>> > > > > > >>> On Thu, Jun 29, 2017 at 7:32 AM, Gurer Kiratli <
>>>> > > > > > >>> gurer.kiratli@airbnb.com.invalid> wrote:
>>>> > > > > > >>>
>>>> > > > > > >>>> Hey all,
>>>> > > > > > >>>>
>>>> > > > > > >>>> We talked about this internally. We would like to work
>>>> on this
>>>> > > > > feature
>>>> > > > > > >>> but
>>>> > > > > > >>>> given the immediate priorities we are not going to be
>>>> working
>>>> > on
>>>> > > > it
>>>> > > > > in
>>>> > > > > > >>> Q3.
>>>> > > > > > >>>> Comes end of Q3 we will reevaluate. Likely scenario is
>>>> we can
>>>> > > work
>>>> > > > > on
>>>> > > > > > >> it
>>>> > > > > > >>>> late Q4 or Q12018.
>>>> > > > > > >>>>
>>>> > > > > > >>>> Cheers,
>>>> > > > > > >>>>
>>>> > > > > > >>>> Gurer
>>>> > > > > > >>>>
>>>> > > > > > >>>> On Tue, Jun 27, 2017 at 8:08 AM, Chris Riccomini <
>>>> > > > > > >> criccomini@apache.org>
>>>> > > > > > >>>> wrote:
>>>> > > > > > >>>>
>>>> > > > > > >>>>> I think FAB sounds like the right approach. Waiting to
>>>> hear
>>>> > > back
>>>> > > > > with
>>>> > > > > > >>>> notes
>>>> > > > > > >>>>> on AirBNB H2 discussion to see if they want to take
>>>> this up.
>>>> > > > > > >>>>>
>>>> > > > > > >>>>> @Gurer, any idea when this will happen?
>>>> > > > > > >>>>>
>>>> > > > > > >>>>> On Thu, Jun 22, 2017 at 1:00 AM, Bolke de Bruin <
>>>> > > > bdbruin@gmail.com
>>>> > > > > >
>>>> > > > > > >>>> wrote:
>>>> > > > > > >>>>>
>>>> > > > > > >>>>>> One downside I see from FAB is that is does not do
>>>> Business
>>>> > > Role
>>>> > > > > > >>>> mapping
>>>> > > > > > >>>>>> to FAB role. I would prefer to create groups in
>>>> IPA/LDAP/AD
>>>> > > and
>>>> > > > > > >> have
>>>> > > > > > >>>>> those
>>>> > > > > > >>>>>> map to FAB roles instead of needing to manage that in
>>>> FAB.
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>> B.
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>>> On 22 Jun 2017, at 09:36, Bolke de Bruin <
>>>> > bdbruin@gmail.com>
>>>> > > > > > >>> wrote:
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> Hi Guys,
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> Thanks for putting the thinking in! It is about time
>>>> that
>>>> > we
>>>> > > > get
>>>> > > > > > >>> this
>>>> > > > > > >>>>>> moving.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> The design looks pretty sound. One can argue about the
>>>> > > > different
>>>> > > > > > >>>> roles
>>>> > > > > > >>>>>> that are required, but that will be situation
>>>> dependent I
>>>> > > guess.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> Implementation wise I would argue together with Max
>>>> that
>>>> > FAB
>>>> > > > is a
>>>> > > > > > >>>>> better
>>>> > > > > > >>>>>> or best fit. The ER model that is being described is
>>>> pretty
>>>> > > > much a
>>>> > > > > > >>> copy
>>>> > > > > > >>>>> of
>>>> > > > > > >>>>>> a normal security model. So a reimplementation of that
>>>> is 1)
>>>> > > > > > >>>> significant
>>>> > > > > > >>>>>> duplication of effort and 2) bound to have bugs that
>>>> have
>>>> > been
>>>> > > > > > >> solved
>>>> > > > > > >>>> in
>>>> > > > > > >>>>>> the other framework. Moreover, FAB does have
>>>> integration out
>>>> > > of
>>>> > > > > the
>>>> > > > > > >>> box
>>>> > > > > > >>>>>> with some enterprisey systems like IPA,
>>>> ActiveDirectory, and
>>>> > > > LDAP.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> So while you argue that using FAB would increase the
>>>> scope
>>>> > of
>>>> > > > the
>>>> > > > > > >>>>>> proposal significantly, but I think that is not true.
>>>> Using
>>>> > > FAB
>>>> > > > > > >> would
>>>> > > > > > >>>>> allow
>>>> > > > > > >>>>>> you to focus on what kind of out-of-the-box permission
>>>> sets
>>>> > > and
>>>> > > > > > >> roles
>>>> > > > > > >>>> we
>>>> > > > > > >>>>>> would need and maybe address some issues that FAB lacks
>>>> > (maybe
>>>> > > > how
>>>> > > > > > >> to
>>>> > > > > > >>>>> deal
>>>> > > > > > >>>>>> with non web access - ie. in DAGs, maybe Kerberos,
>>>> probably
>>>> > > how
>>>> > > > to
>>>> > > > > > >>> deal
>>>> > > > > > >>>>>> with API calls that are not CRUD). Implementation wise
>>>> it
>>>> > > > probably
>>>> > > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s
>>>> early POC
>>>> > > as
>>>> > > > an
>>>> > > > > > >>>>> example
>>>> > > > > > >>>>>> - we can slowly move over?
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> On a side note: Im planning to hire 2-3 ppl to work on
>>>> > > Airflow
>>>> > > > > > >>> coming
>>>> > > > > > >>>>>> year. Improvement of Security, Enterprise Integration,
>>>> > Revamp
>>>> > > UI
>>>> > > > > > >> are
>>>> > > > > > >>> on
>>>> > > > > > >>>>> the
>>>> > > > > > >>>>>> todo list. However, this is not confirmed yet as
>>>> business
>>>> > > > > > >> priorities
>>>> > > > > > >>>>> might
>>>> > > > > > >>>>>> change.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>> Bolke.
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>>> On 15 Jun 2017, at 21:45, kalpesh dharwadkar <
>>>> > > > > > >>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> @Dan:
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> Thanks for your feedback. I will remove the
>>>> REFRESH_DAG
>>>> > > > > > >>> permission.
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> @Max:
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> Thanks for your response.
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> The scope of my proposal was just to add RBAC
>>>> security
>>>> > > feature
>>>> > > > > > >> to
>>>> > > > > > >>>>>> Airflow
>>>> > > > > > >>>>>>>> without replacing any existing frameworks.
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> I understand that adopting FAB would serve Airflow
>>>> better
>>>> > > > moving
>>>> > > > > > >>>>>> forward,
>>>> > > > > > >>>>>>>> however porting Airflow to using FAB significantly
>>>> > increases
>>>> > > > the
>>>> > > > > > >>>> scope
>>>> > > > > > >>>>>> of
>>>> > > > > > >>>>>>>> the proposal and I don't have the time and expertise
>>>> to
>>>> > > carry
>>>> > > > > > >> out
>>>> > > > > > >>>> the
>>>> > > > > > >>>>>> tasks
>>>> > > > > > >>>>>>>> in the extended scope.
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> Hence, I'm curious to know if there's a plan for
>>>> Airflow
>>>> > to
>>>> > > > > > >>> migrate
>>>> > > > > > >>>> to
>>>> > > > > > >>>>>> FAB
>>>> > > > > > >>>>>>>> this year?
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> - Kalpesh
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>> On Mon, Jun 12, 2017 at 6:16 PM, Maxime Beauchemin <
>>>> > > > > > >>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>> > > > > > >>>>>>>>
>>>> > > > > > >>>>>>>>> It would be nice to go with a framework for this. I
>>>> did
>>>> > > some
>>>> > > > > > >>>>>>>>> experimentation using FlaskAppBuilder to go in this
>>>> > > > direction.
>>>> > > > > > >> It
>>>> > > > > > >>>>>> provides
>>>> > > > > > >>>>>>>>> auth on different authentication backends out of
>>>> the box
>>>> > > > > > >> (oauth,
>>>> > > > > > >>>>>> openid,
>>>> > > > > > >>>>>>>>> ldap, registration, ...), generates perms for each
>>>> view
>>>> > > that
>>>> > > > > > >> has
>>>> > > > > > >>> an
>>>> > > > > > >>>>>>>>> @has_access decorator, generates at set of perms
>>>> for each
>>>> > > ORM
>>>> > > > > > >>> model
>>>> > > > > > >>>>>> (show,
>>>> > > > > > >>>>>>>>> edit, delete, add, ...) and enforces it in the CRUD
>>>> views
>>>> > > as
>>>> > > > > > >> well
>>>> > > > > > >>>> as
>>>> > > > > > >>>>>> in the
>>>> > > > > > >>>>>>>>> generated REST api that you get for free as a
>>>> byprdoduct
>>>> > of
>>>> > > > > > >>>> deriving
>>>> > > > > > >>>>>> FAB's
>>>> > > > > > >>>>>>>>> models (essentially it's SqlAlchemy with a layer on
>>>> top).
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> I started a POC on FAB here a while ago:
>>>> > > > > > >>>>>>>>> https://github.com/mistercrunch/airflow_webserver
>>>> at the
>>>> > > > time
>>>> > > > > > >> my
>>>> > > > > > >>>>> main
>>>> > > > > > >>>>>>>>> motivation was the free/instantaneous REST api.
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> I think FAB is a decent fit as the porting should be
>>>> > fairly
>>>> > > > > > >>>>>> straightforward
>>>> > > > > > >>>>>>>>> (moving the flask views over and deprecating
>>>> Flask-Admin
>>>> > in
>>>> > > > > > >> favor
>>>> > > > > > >>>> of
>>>> > > > > > >>>>>> FAB's
>>>> > > > > > >>>>>>>>> crud) though there was a few blockers. From memory I
>>>> > think
>>>> > > > FAB
>>>> > > > > > >>>> didn't
>>>> > > > > > >>>>>> like
>>>> > > > > > >>>>>>>>> the compound PKs we use in some of the Airflow
>>>> models.
>>>> > We'd
>>>> > > > > > >> have
>>>> > > > > > >>> to
>>>> > > > > > >>>>>> either
>>>> > > > > > >>>>>>>>> write a db migration script on the Airflow side, or
>>>> add
>>>> > > > support
>>>> > > > > > >>> for
>>>> > > > > > >>>>>>>>> compound keys to FAB (I recently became a
>>>> maintainer of
>>>> > the
>>>> > > > > > >>>> project,
>>>> > > > > > >>>>>> so I
>>>> > > > > > >>>>>>>>> could help with that)
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> The only downside of FAB is that it's not as mature
>>>> as
>>>> > > > > > >> something
>>>> > > > > > >>>> like
>>>> > > > > > >>>>>>>>> Django, but porting to Django would surely be much
>>>> more
>>>> > > work.
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> Then there's the flask-security suite, but that
>>>> looks
>>>> > like
>>>> > > a
>>>> > > > > > >> bit
>>>> > > > > > >>>> of a
>>>> > > > > > >>>>>>>>> patchwork to me, I guess we can pick and choose
>>>> which we
>>>> > > want
>>>> > > > > > >> to
>>>> > > > > > >>>> use.
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> Max
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>> On Mon, Jun 12, 2017 at 12:50 PM, Dan Davydov <
>>>> > > > > > >>>>>>>>> dan.davydov@airbnb.com.invalid> wrote:
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>>>> Looks good to me in general, thanks for putting
>>>> this
>>>> > > > together!
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>> I think the ability to integrate with external RBAC
>>>> > > systems
>>>> > > > > > >> like
>>>> > > > > > >>>>> LDAP
>>>> > > > > > >>>>>> is
>>>> > > > > > >>>>>>>>>> important (i.e. the Airflow DB should not be
>>>> decoupled
>>>> > > with
>>>> > > > > > >> the
>>>> > > > > > >>>> RBAC
>>>> > > > > > >>>>>>>>>> database wherever possible).
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>> I wouldn't be too worried about the permissions
>>>> about
>>>> > > > > > >> refreshing
>>>> > > > > > >>>>>> DAGs, as
>>>> > > > > > >>>>>>>>>> far as I know this functionality is no longer
>>>> required
>>>> > > with
>>>> > > > > > >> the
>>>> > > > > > >>>> new
>>>> > > > > > >>>>>>>>>> webservers which reload state periodically, and
>>>> will
>>>> > > > certainly
>>>> > > > > > >>> be
>>>> > > > > > >>>>>> removed
>>>> > > > > > >>>>>>>>>> when we have a better DAG consistency story.
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>> I think it would also be good to think about this
>>>> > > > > > >>>>>> proposal/implementation
>>>> > > > > > >>>>>>>>>> and how it applied in the API-driven world (e.g.
>>>> when
>>>> > > > > > >> webserver
>>>> > > > > > >>>> hits
>>>> > > > > > >>>>>> APIs
>>>> > > > > > >>>>>>>>>> like /clear on behalf of users instead of running
>>>> > commands
>>>> > > > > > >>> against
>>>> > > > > > >>>>> the
>>>> > > > > > >>>>>>>>>> database directly).
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>> On Mon, Jun 12, 2017 at 11:12 AM, Bolke de Bruin <
>>>> > > > > > >>>> bdbruin@gmail.com
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>>>>>> wrote:
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>>> Will respond but im traveling at the moment. Give
>>>> me a
>>>> > > few
>>>> > > > > > >>> days.
>>>> > > > > > >>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>> Sent from my iPhone
>>>> > > > > > >>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> On 12 Jun 2017, at 13:39, Chris Riccomini <
>>>> > > > > > >>>> criccomini@apache.org>
>>>> > > > > > >>>>>>>>>> wrote:
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> Hey all,
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> Checking in on this. We spent a good chunk of
>>>> time
>>>> > > > thinking
>>>> > > > > > >>>> about
>>>> > > > > > >>>>>>>>> this,
>>>> > > > > > >>>>>>>>>>> and
>>>> > > > > > >>>>>>>>>>>> want to move forward with it, but want to make
>>>> sure
>>>> > > we're
>>>> > > > > > >> all
>>>> > > > > > >>> on
>>>> > > > > > >>>>> the
>>>> > > > > > >>>>>>>>>> same
>>>> > > > > > >>>>>>>>>>>> page.
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> Max? Bolke? Dan? Jeremiah?
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> Cheers,
>>>> > > > > > >>>>>>>>>>>> Chris
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>> On Thu, Jun 8, 2017 at 1:49 PM, kalpesh
>>>> dharwadkar <
>>>> > > > > > >>>>>>>>>>>> kalpeshdharwadkar@gmail.com> wrote:
>>>> > > > > > >>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> Hello everyone,
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> As you all know, currently Airflow doesn’t have
>>>> a
>>>> > > > built-in
>>>> > > > > > >>> Role
>>>> > > > > > >>>>>>>>> Based
>>>> > > > > > >>>>>>>>>>>>> Access Control(RBAC) capability. It does
>>>> provide
>>>> > very
>>>> > > > > > >>> limited
>>>> > > > > > >>>>>>>>>>>>> authorization capability by providing admin,
>>>> > > > data_profiler,
>>>> > > > > > >>> and
>>>> > > > > > >>>>>> user
>>>> > > > > > >>>>>>>>>>> roles.
>>>> > > > > > >>>>>>>>>>>>> However, associating these roles to
>>>> authenticated
>>>> > > > > > >> identities
>>>> > > > > > >>> is
>>>> > > > > > >>>>> not
>>>> > > > > > >>>>>>>>> a
>>>> > > > > > >>>>>>>>>>>>> simple effort.
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> To address this issue, I have created a design
>>>> > proposal
>>>> > > > for
>>>> > > > > > >>>>>> building
>>>> > > > > > >>>>>>>>>>> RBAC
>>>> > > > > > >>>>>>>>>>>>> into Airflow and simplifying user access
>>>> management
>>>> > via
>>>> > > > the
>>>> > > > > > >>>>> Airflow
>>>> > > > > > >>>>>>>>>> UI.
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> The design proposal is located at
>>>> > > > > > >> https://cwiki.apache.org/
>>>> > > > > > >>>>>>>>>>>>> confluence/display/AIRFLOW/Air
>>>> flow+RBAC+proposal
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> Any comments/questions/feedback are much
>>>> appreciated.
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>>> Thanks
>>>> > > > > > >>>>>>>>>>>>> Kalpesh
>>>> > > > > > >>>>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>>
>>>> > > > > > >>>>>>>>>>
>>>> > > > > > >>>>>>>>>
>>>> > > > > > >>>>>>>
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>>
>>>> > > > > > >>>>>
>>>> > > > > > >>>>
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>> --
>>>> > > > > > >>>
>>>> > > > > > >>> Joy Gao
>>>> > > > > > >>> Software Engineer
>>>> > > > > > >>> 350 Convention Way, Suite 200
>>>> > > > > > >>> Redwood City, CA 94063
>>>> > > > > > >>> Mobile: 669-224-9305
>>>> > > > > > >>>
>>>> > > > > > >>> Payments partner to the platform economy
>>>> > > > > > >>>
>>>> > > > > > >>
>>>> > > > > >
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>