You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Tao Feng <fe...@gmail.com> on 2018/04/02 21:44:52 UTC
Re: DAG Level permissions (was Re: RBAC Update)
Hi everyone,
Thanks a lot for all the great discussions. To summarize in brief, here are
the few approaches we discussed so far:
1. One permission per DAG. The user has homogenous rights on the dag.
The concerns:
- not flexible to certain use cases(e.g the user has view only access on
certain dags and write access on certain other dags );
- difficult to implement as it breaks the existing FAB security model.
2. Extend current model(ab_permission_view_role) with an additional
optional column (dag_id).
The concerns:
- introduce a 3rd column to existing permission_view table.
- It requires us to handle db migration carefully from view only access
UI to dag level access UI.
3. Define a set of pre-defined dag-level permissions. Reuse the current
existing permission_view model and consider each individual dag as a new
view.
It seems that the 3rd approach is a preferable approach which makes the
security model easy to extend without introducing additional complexity. If
no other concern, I will work with Max to create an initial proposal / PR
based on 3rd approach for the work(https://issues.apache.
org/jira/browse/AIRFLOW-2267).
Thanks,
-Tao
On Sat, Mar 31, 2018 at 12:09 AM, Joy Gao <jo...@wepay.com> wrote:
> +1!
>
> I was originally tempted to re-use existing perms and views for dag-level
> access control since dag-level perm/view is a subset of view-level
> perm/view, but your proposal of defining new dag-level perms/views
> independent from view-level perms/views is interesting. This actually makes
> a lot of sense, since even in the existing models, views can also be menu
> options, so we are simply extending the concept of views to include dags as
> well.
>
> On Fri, Mar 30, 2018 at 5:24 PM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > I'd suggest something else that avoids having to add a 3rd column. I
> think
> > we can fit our use case into the existing structure nicely.
> >
> > My idea is to mimic what FAB does with its own Models.
> >
> > When you create a Model and ModelView in FAB (say DagRun for example), it
> > creates a new view_menu (DagRun) and matches it with existing permission
> > (can_read, can_delete, can_edit, can_show, ...) to create a new set of
> > "permission_views" which are combinations of permission and view_menu as
> in
> > "DagRun - can_read", "DagRun - can_edit", ... It's not a cartesian
> product
> > of all perms and view_menus, it's a predetermined list of model-specific
> > perms that get combined with DagRun here.
> >
> > Similarly, when Airflow would detect a new DAG (example "my_dag"), it
> > would create a new view_menu `my_dag`, and match it with permissions that
> > are identified as "combinable" with DAG. To avoid potential conflicts I'd
> > create new permissions that are DAG-specific like "dag_clear", "dag_run",
> > "dag_view".
> >
> > I'm arguing about the how to use the FAB model here specifically. I think
> > this allows for all the flexibility we need without changing the model. I
> > care less about what exactly the atomicity of the per DAG perms should
> look
> > like. As far as I'm concerned, per-DAG read and write is probably
> granular
> > enough
> >
> > Also note that:
> > * we need an "_all_dags" logical DAG, meaning we'd have extra
> > permission_views "_all_dags - dag_view", "_all_dags - dag_run",
> "all_dags -
> > dag_clear"
> > * we probably want to derive FAB's SecurityManger and have an
> > AirflowSecurityManager that has an extra method "can_dag_action(user,
> > dag_id, action)". The SecurityManger class is neat because people can
> > provide their own if they want and override methods. That means that you
> > can defer any of the security-related checks to another system if you
> want
> > to. Many companies have some sort of company RBAC system and that can be
> > used to integrate with it.
> >
> > Max
> >
> >
> >
> > On Fri, Mar 30, 2018 at 4:54 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> >> Hi all,
> >>
> >> I also agree that having view-only access to some dags while write
> access
> >> to other dags is useful, so I prefer option 2. Although option 2 is more
> >> difficult to manage, it is cleaner and more consistent with the current
> >> security model. (On the other hand, even though option 1 may be may be
> >> easier to manage, it might be trickier to implement: with one perm per
> >> dag, it breaks the existing FAB security model since the existing, more
> >> granular permissions will have to be grouped for each dag).
> >>
> >> What Brian suggested in the other thread makes sense to me...
> >>
> >> The current security model in FAB looks like the following:
> >>
> >> --- permissions
> >> user --- role --- permission_view ---|
> >> --- views
> >>
> >> FAB generates a few relationship tables to manage the mappings, one of
> >> them is *ab_permission_view_role*, which has all the
> >> role-to-permission_view mapping. We can add a dag_id column to this
> >> relationship that expresses dag-level granularity, so the security model
> >> becomes:
> >>
> >> ---
> >> permissions
> >> |
> >> user --- role --- dag_permission_view --- --- views
> >> |
> >> ---
> dags
> >>
> >> We can make the dag_id field an optional field in the relationship, and
> >> only lazily add new dag-level mappings for DAGs that specify 'access
> >> control'. This way we don't have to create new permissions or views. (We
> >> will still have to introduce new dag-level roles on top of the 5 generic
> >> roles (public/viewer/user/op/admin)).
> >>
> >> (I think this is similar to what Arthur suggested earlier, but not sure
> >> if I interpreted correctly).
> >>
> >> Joy
> >>
> >> On Thu, Mar 29, 2018 at 10:01 AM, Arthur Wiedmer <
> >> arthur.wiedmer@gmail.com> wrote:
> >>
> >>> (Creating a new thread)
> >>>
> >>> Hi Max,
> >>>
> >>> I was just wondering about this. There are definite use cases for
> people
> >>> having only view access to some DAGs, mostly for monitoring. I want to
> know
> >>> what the upstream DAGs are doing, but maybe I don't need clear/run
> access.
> >>>
> >>> I feel like the granular operation permissions will start to become
> >>> unwieldy fast. It grows as (# users * # DAGs * # operations) but with
> >>> hopefully a small constant factor in front: most people should only
> have a
> >>> small number of DAGs they care about. The Ops team has a need to have
> >>> access to all on the other hand.
> >>>
> >>> I was wondering we could get by with something slightly more complex
> but
> >>> easier on the size of that permissions table :
> >>> 1) One toggle on the user level for broad access (ALL:ALL,
> >>> RUN/CLEAR:ALL, VIEW:ALL) default NULL
> >>> 2) More granular permissions at the DAG level.
> >>>
> >>> So in order, check the user's broad level permission first, then DAG
> >>> level. For large amounts of DAGs, this should help shave a little bit
> from
> >>> that table.
> >>>
> >>> Best,
> >>> Arthur
> >>>
> >>>
> >>> On Thu, Mar 29, 2018 at 8:27 AM, Maxime Beauchemin <
> >>> maximebeauchemin@gmail.com> wrote:
> >>>
> >>>> Hijacking the thread further here, any thoughts on how to breakdown
> per
> >>>> DAG
> >>>> access?
> >>>>
> >>>> Tao & I are talking about introducing per-DAG permissions and one big
> >>>> question is whether we'll need to support different operation-types
> at a
> >>>> per-DAG level, which changes the way we need to model the perms.
> >>>>
> >>>> First [simpler] option is to introduce one perm per DAG. If you have
> >>>> access
> >>>> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have
> >>>> homogenous
> >>>> rights on the DAGs you have access to.
> >>>>
> >>>> Second option is to have a breakdown per DAG. Meaning for each DAG we
> >>>> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...).
> So
> >>>> one
> >>>> user could have modify on some DAGs, view on others, and other DAGs
> >>>> would
> >>>> be invisible. This could be broken down further ({dag_id}_can_clear,
> >>>> ...)
> >>>> but it gets hard to manage.
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> Max
> >>>>
> >>>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fe...@gmail.com>
> wrote:
> >>>>
> >>>> > Great work Joy. This is awesome! I am interested in helping out the
> >>>> per dag
> >>>> > level access. Just created a ticket to check(AIRFLOW-2267). Let me
> >>>> know if
> >>>> > you have any suggestions. I will share my proposal once I am ready.
> >>>> >
> >>>> > On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
> >>>> >
> >>>> > > Hey guys!
> >>>> > >
> >>>> > > The RBAC UI <https://github.com/apache/inc
> ubator-airflow/pull/3015>
> >>>> has
> >>>> > > been merged to master. I'm looking forward to early adopters'
> >>>> feedback
> >>>> > and
> >>>> > > bug reports. I also hope to have more folks helping out with the
> >>>> RBAC UI,
> >>>> > > especially with introducing DAG-Level access control, which is a
> >>>> feature
> >>>> > > that a lot of people have been asking. If you are interested in
> >>>> helping
> >>>> > out
> >>>> > > with this effort, let's talk more!
> >>>> > >
> >>>> > > This commit will be in the 1.10.0 release, and we are going to
> >>>> maintain
> >>>> > > both UIs simultaneously for a short period of time. Once RBAC UI
> is
> >>>> > stable
> >>>> > > and battle-tested, we will deprecate the old UI and eventually
> >>>> remove it
> >>>> > > from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to
> >>>> prevent
> >>>> > > two UIs from forking into separate paths, as that would become
> very
> >>>> > > difficult to maintain.
> >>>> > >
> >>>> > > Going forward while both UIs are up, if you are making a change to
> >>>> any
> >>>> > > files in airflow/www/ (old UI), where applicable, please also make
> >>>> the
> >>>> > > change to the airflow/www_rbac/ (new UI). If you rather not make
> >>>> changes
> >>>> > in
> >>>> > > both UIs, it is recommended that you only make the changes to the
> >>>> RBAC
> >>>> > UI,
> >>>> > > since that is the one we are maintaining in the long term.
> >>>> > >
> >>>> > > I'm excited that the RBAC UI will be able to bring additional
> >>>> security to
> >>>> > > Airflow, and with FAB framework in place we can look into
> >>>> leveraging it
> >>>> > for
> >>>> > > a unified set of APIs used by both UI and CLI.
> >>>> > >
> >>>> > > Joy
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
> >>>> > >
> >>>> > > > Hi folks,
> >>>> > > >
> >>>> > > > I have a PR <https://github.com/apache/
> >>>> incubator-airflow/pull/3015>
> >>>> > out
> >>>> > > > for the new UI. I've included instructions on how to test it out
> >>>> in the
> >>>> > > PR
> >>>> > > > description. Looking forward to your feedbacks.
> >>>> > > >
> >>>> > > > Cheers,
> >>>> > > > Joy
> >>>> > > >
> >>>> > > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
> >>>> > > >
> >>>> > > >> Thanks for the background info. Would be really awesome for you
> >>>> to
> >>>> > have
> >>>> > > >> PyPi access :D I'll make the change to have Airflow Webserver's
> >>>> FAB
> >>>> > > >> dependency pointing to my fork for the mean time.
> >>>> > > >>
> >>>> > > >> For folks who are interested in RBAC, I will be giving a
> >>>> talk/demo at
> >>>> > > the Airflow
> >>>> > > >> Meet-Up
> >>>> > > >> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> >>>> > > Meetup/events/244525050/>
> >>>> > > >> next Monday. Happy to chat afterwards about it as well :)
> >>>> > > >>
> >>>> > > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> >>>> > > >> maximebeauchemin@gmail.com> wrote:
> >>>> > > >>
> >>>> > > >>> A bit of related history here:
> >>>> > > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> >>>> > > >>>
> >>>> > > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> >>>> > > >>> maximebeauchemin@gmail.com> wrote:
> >>>> > > >>>
> >>>> > > >>> > Given I have merge rights on FAB I could probably do another
> >>>> round
> >>>> > of
> >>>> > > >>> > review and get your PRs through. I would really like to get
> >>>> the
> >>>> > main
> >>>> > > >>> > maintainer's input on things that touch the core
> >>>> (composite-key
> >>>> > > >>> support) as
> >>>> > > >>> > he might have concerns/intuitions that we can't know about.
> >>>> > > >>> >
> >>>> > > >>> > I do not have Pypi access though so I cannot push new
> releases
> >>>> > out. I
> >>>> > > >>> > could ask for that.
> >>>> > > >>> >
> >>>> > > >>> > I've threatened to fork the project before, that's always an
> >>>> > option.
> >>>> > > >>> I've
> >>>> > > >>> > noticed his involvement is sporadic and comes in bursts.
> >>>> > > >>> >
> >>>> > > >>> > In the meantime, you can have the dependency in Airflow
> >>>> Webserver
> >>>> > > >>> pointing
> >>>> > > >>> > straight to your fork.
> >>>> > > >>> >
> >>>> > > >>> > Max
> >>>> > > >>> >
> >>>> > > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com>
> >>>> wrote:
> >>>> > > >>> >
> >>>> > > >>> >> I just created a new webserver instance if you haven't
> >>>> gotten a
> >>>> > > >>> chance to
> >>>> > > >>> >> fiddle around with the new web UI and the RBAC
> configurations
> >>>> > > (thanks
> >>>> > > >>> >> Maxime for getting started with this earlier!):
> >>>> > > >>> >>
> >>>> > > >>> >> http://104.209.38.171:8080/
> >>>> > > >>> >>
> >>>> > > >>> >> Admin Account
> >>>> > > >>> >> username: admin
> >>>> > > >>> >> password: admin
> >>>> > > >>> >>
> >>>> > > >>> >> Read-Only Account
> >>>> > > >>> >> username: viewer
> >>>> > > >>> >> password: password
> >>>> > > >>> >>
> >>>> > > >>> >>
> >>>> > > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com>
> >>>> wrote:
> >>>> > > >>> >>
> >>>> > > >>> >> > Hi folks,
> >>>> > > >>> >> >
> >>>> > > >>> >> > Thanks for all the feedback regarding to the new Airflow
> >>>> > Webserver
> >>>> > > >>> UI
> >>>> > > >>> >> > <https://github.com/wepay/airflow-webserver/>! I've been
> >>>> > actively
> >>>> > > >>> >> > addressing all the bugs that were raised on Github. So I
> >>>> want to
> >>>> > > >>> take
> >>>> > > >>> >> this
> >>>> > > >>> >> > opportunity to discuss two issues coming up:
> >>>> > > >>> >> >
> >>>> > > >>> >> > The first issue is unaddressed PRs in FAB. If these PRs
> >>>> continue
> >>>> > > to
> >>>> > > >>> stay
> >>>> > > >>> >> > unaddressed, RBAC is blocked from making further
> progress.
> >>>> If
> >>>> > this
> >>>> > > >>> >> continue
> >>>> > > >>> >> > to be an issue, I'm inclined to fork FAB, even though
> it's
> >>>> not
> >>>> > > >>> >> idealistic.
> >>>> > > >>> >> >
> >>>> > > >>> >> >
> >>>> > > >>> >> > - PR/631 <https://github.com/dpgaspar/F
> >>>> > > lask-AppBuilder/pull/631>
> >>>> > > >>> >> Binary
> >>>> > > >>> >> > column support (merged, unreleased)
> >>>> > > >>> >> > <https://github.com/dpgaspar/F
> lask-AppBuilder/pull/631>
> >>>> > > >>> >> > - PR/639 <https://github.com/dpgaspar/F
> >>>> > > lask-AppBuilder/pull/639>
> >>>> > > >>> >> Composite
> >>>> > > >>> >> > primary key support (unmerged)
> >>>> > > >>> >> > - PR/655 <https://github.com/dpgaspar/F
> >>>> > > lask-AppBuilder/pull/655>
> >>>> > > >>> >> Form
> >>>> > > >>> >> > prefill support (unmerged)
> >>>> > > >>> >> >
> >>>> > > >>> >> >
> >>>> > > >>> >> > The second issue is an open question about the next step
> of
> >>>> > > Airflow
> >>>> > > >>> >> > Webserver itself. Here are the 3 potential directions we
> >>>> could
> >>>> > > >>> take, and
> >>>> > > >>> >> > I've added my thought on each.
> >>>> > > >>> >> >
> >>>> > > >>> >> > 1. Permanently keep Airflow Webserver as a separated
> >>>> package
> >>>> > from
> >>>> > > >>> >> Airflow,
> >>>> > > >>> >> > and treat it as another UI option. Keep `www` in Airflow.
> >>>> Allow
> >>>> > > >>> >> development
> >>>> > > >>> >> > on both UIs.
> >>>> > > >>> >> > *I'm not a fan of this. When there is an existing UI in
> >>>> Airflow,
> >>>> > > >>> most
> >>>> > > >>> >> > contributors would prefer to maintain the official
> version
> >>>> that
> >>>> > is
> >>>> > > >>> >> > installed out-of-the-box. **Having a second UI outside of
> >>>> > Airflow
> >>>> > > >>> will
> >>>> > > >>> >> > make maintaining it very difficult, leading to an
> eventual
> >>>> death
> >>>> > > of
> >>>> > > >>> the
> >>>> > > >>> >> new
> >>>> > > >>> >> > UI :(*
> >>>> > > >>> >> >
> >>>> > > >>> >> > 2. Permanently keep Airflow Webserver as a separated
> >>>> package
> >>>> > from
> >>>> > > >>> >> Airflow,
> >>>> > > >>> >> > but freeze all development on `www` and direct all
> future
> >>>> UI
> >>>> > > >>> >> development
> >>>> > > >>> >> > to Airflow Webserver, eventually removing `www`
> completely
> >>>> when
> >>>> > > >>> Airflow
> >>>> > > >>> >> > Webserver is stable.
> >>>> > > >>> >> > *I'm not a fan of this either. First of all, the views
> and
> >>>> > models
> >>>> > > >>> are
> >>>> > > >>> >> > tightly coupled in both old and new UI; until we have a
> >>>> > > full-fledged
> >>>> > > >>> >> REST
> >>>> > > >>> >> > API to build the UI (and cli) on top of it, separating
> >>>> them to a
> >>>> > > >>> >> separate
> >>>> > > >>> >> > package now will potentially cause dependency issues and
> >>>> add
> >>>> > > >>> >> complication
> >>>> > > >>> >> > to our release cycle. **Secondly, **majority of Airflow
> >>>> users
> >>>> > run
> >>>> > > >>> >> Airflow
> >>>> > > >>> >> > with the UI; it's one of Airflow's best features.
> >>>> Separating UI
> >>>> > > out
> >>>> > > >>> of
> >>>> > > >>> >> > Airflow core will complicate setup and configuration,
> while
> >>>> > making
> >>>> > > >>> >> Airflow
> >>>> > > >>> >> > core less complete.*
> >>>> > > >>> >> >
> >>>> > > >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`,
> >>>> freeze
> >>>> > all
> >>>> > > >>> >> > development on `www`, eventually removing `www`
> completely
> >>>> when
> >>>> > > >>> `www2`
> >>>> > > >>> >> is
> >>>> > > >>> >> > stable.
> >>>> > > >>> >> > *This makes the most sense to me. Airflow Webserver is
> >>>> developed
> >>>> > > >>> with
> >>>> > > >>> >> the
> >>>> > > >>> >> > goal of feature parity to the current UI, plus additional
> >>>> RBAC
> >>>> > > >>> >> capability,
> >>>> > > >>> >> > in hope to replace the old UI completely. Yes, this means
> >>>> there
> >>>> > > >>> will be
> >>>> > > >>> >> a
> >>>> > > >>> >> > short period of having to maintain two UIs, but once we
> >>>> freeze
> >>>> > > >>> >> development
> >>>> > > >>> >> > on www, it shouldn't be a concern for long.*
> >>>> > > >>> >> >
> >>>> > > >>> >> > I'd love to hear everyone's thoughts on this! I'm excited
> >>>> about
> >>>> > > >>> bringing
> >>>> > > >>> >> > RBAC to airflow and I hope it's something others will
> find
> >>>> > useful
> >>>> > > as
> >>>> > > >>> >> well!
> >>>> > > >>> >> >
> >>>> > > >>> >> > Cheers,
> >>>> > > >>> >> > Joy
> >>>> > > >>> >> >
> >>>> > > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <
> joyg@wepay.com>
> >>>> > wrote:
> >>>> > > >>> >> >
> >>>> > > >>> >> >> Thank you everyone for the active feedback so far, and
> >>>> thanks
> >>>> > for
> >>>> > > >>> >> setting
> >>>> > > >>> >> >> up the demo Maxime!
> >>>> > > >>> >> >>
> >>>> > > >>> >> >> Going to work on pruning through the issues in the
> >>>> upcoming
> >>>> > days.
> >>>> > > >>> >> >>
> >>>> > > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception
> >>>> message
> >>>> > so I
> >>>> > > >>> can
> >>>> > > >>> >> >> look into it? Otherwise I'll wait until it's down again
> =P
> >>>> > > >>> >> >>
> >>>> > > >>> >> >> Cheers,
> >>>> > > >>> >> >>
> >>>> > > >>> >> >> Joy
> >>>> > > >>> >> >>
> >>>> > > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >>>> > > >>> >> >> maximebeauchemin@gmail.com> wrote:
> >>>> > > >>> >> >>
> >>>> > > >>> >> >>> I just restarted it, not sure how long it will take to
> >>>> get in
> >>>> > a
> >>>> > > >>> bad
> >>>> > > >>> >> state
> >>>> > > >>> >> >>> again...
> >>>> > > >>> >> >>>
> >>>> > > >>> >> >>> Max
> >>>> > > >>> >> >>>
> >>>> > > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> >>>> > > >>> >> <fokko@driesprong.frl
> >>>> > > >>> >> >>> >
> >>>> > > >>> >> >>> wrote:
> >>>> > > >>> >> >>>
> >>>> > > >>> >> >>> > Good morning,
> >>>> > > >>> >> >>> >
> >>>> > > >>> >> >>> > The demo provided by Max is down, it throws a
> >>>> > > >>> SQLAlchemyexception
> >>>> > > >>> >> :'(
> >>>> > > >>> >> >>> >
> >>>> > > >>> >> >>> > Cheers, Fokko
> >>>> > > >>> >> >>> >
> >>>> > > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> >>>> > > >>> criccomini@apache.org>:
> >>>> > > >>> >> >>> >
> >>>> > > >>> >> >>> > > @bolke, open issues on the Github repo, please.
> >>>> > > >>> >> >>> > >
> >>>> > > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> >>>> > > >>> >> bdbruin@gmail.com>
> >>>> > > >>> >> >>> > > wrote:
> >>>> > > >>> >> >>> > >
> >>>> > > >>> >> >>> > > > Chris,
> >>>> > > >>> >> >>> > > >
> >>>> > > >>> >> >>> > > > Do you want us to report bugs somewhere (I have
> >>>> > > encountered
> >>>> > > >>> a
> >>>> > > >>> >> >>> few)? Or
> >>>> > > >>> >> >>> > > > just generic user experiences posted here?
> >>>> > > >>> >> >>> > > >
> >>>> > > >>> >> >>> > > > Cheers
> >>>> > > >>> >> >>> > > > Bolke
> >>>> > > >>> >> >>> > > >
> >>>> > > >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> >>>> > > >>> >> criccomini@apache.org
> >>>> > > >>> >> >>> >
> >>>> > > >>> >> >>> > > wrote:
> >>>> > > >>> >> >>> > > > >
> >>>> > > >>> >> >>> > > > > Hey all,
> >>>> > > >>> >> >>> > > > >
> >>>> > > >>> >> >>> > > > > I know the weekend is coming up, and for those
> >>>> of us
> >>>> > in
> >>>> > > >>> the
> >>>> > > >>> >> US,
> >>>> > > >>> >> >>> next
> >>>> > > >>> >> >>> > > week
> >>>> > > >>> >> >>> > > > > is a bit of a slow holiday week. Would love to
> >>>> get
> >>>> > some
> >>>> > > >>> >> feedback
> >>>> > > >>> >> >>> from
> >>>> > > >>> >> >>> > > > > everyone on this. The goal would ideally to be
> to
> >>>> > > >>> converge on
> >>>> > > >>> >> >>> this
> >>>> > > >>> >> >>> > and
> >>>> > > >>> >> >>> > > > > eventually replace the existing Airflow UI with
> >>>> this
> >>>> > > one.
> >>>> > > >>> >> >>> > > > >
> >>>> > > >>> >> >>> > > > > Cheers,
> >>>> > > >>> >> >>> > > > > Chris
> >>>> > > >>> >> >>> > > > >
> >>>> > > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> >>>> > > joyg@wepay.com>
> >>>> > > >>> >> wrote:
> >>>> > > >>> >> >>> > > > >
> >>>> > > >>> >> >>> > > > >> Hi guys.
> >>>> > > >>> >> >>> > > > >>
> >>>> > > >>> >> >>> > > > >> I've been working on moving airflow from
> >>>> Flask-Admin
> >>>> > to
> >>>> > > >>> >> >>> > > Flask-AppBuilder
> >>>> > > >>> >> >>> > > > >> for RBAC
> >>>> > > >>> >> >>> > > > >> <https://cwiki.apache.org/
> >>>> > confluence/display/AIRFLOW/
> >>>> > > >>> >> >>> > > > Airflow+RBAC+proposal
> >>>> > > >>> >> >>> > > > >>> ,
> >>>> > > >>> >> >>> > > > >> check it out at
> https://github.com/wepay/airfl
> >>>> > > >>> ow-webserver.
> >>>> > > >>> >> >>> > > > >>
> >>>> > > >>> >> >>> > > > >> It's still a work-in-progress, but most
> >>>> features you
> >>>> > > see
> >>>> > > >>> in
> >>>> > > >>> >> the
> >>>> > > >>> >> >>> > > > webserver
> >>>> > > >>> >> >>> > > > >> UI today is available there. For those who are
> >>>> > > >>> interested in
> >>>> > > >>> >> >>> RBAC,
> >>>> > > >>> >> >>> > I'd
> >>>> > > >>> >> >>> > > > love
> >>>> > > >>> >> >>> > > > >> to get some early feedback in terms of the
> >>>> following:
> >>>> > > >>> >> >>> > > > >>
> >>>> > > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any
> bugs/regressions)
> >>>> > > >>> >> >>> > > > >> - Setup issues
> >>>> > > >>> >> >>> > > > >> - Ease of integration with third party auth
> >>>> (i.e.
> >>>> > LDAP,
> >>>> > > >>> AD,
> >>>> > > >>> >> >>> OAuth,
> >>>> > > >>> >> >>> > > etc.)
> >>>> > > >>> >> >>> > > > >> - Any other thoughts/concerns
> >>>> > > >>> >> >>> > > > >>
> >>>> > > >>> >> >>> > > > >> Thanks a lot!
> >>>> > > >>> >> >>> > > > >>
> >>>> > > >>> >> >>> > > > >> Cheers,
> >>>> > > >>> >> >>> > > > >> Joy
> >>>> > > >>> >> >>> > > > >>
> >>>> > > >>> >> >>> > > >
> >>>> > > >>> >> >>> > > >
> >>>> > > >>> >> >>> > >
> >>>> > > >>> >> >>> >
> >>>> > > >>> >> >>>
> >>>> > > >>> >> >>
> >>>> > > >>> >> >>
> >>>> > > >>> >> >
> >>>> > > >>> >>
> >>>> > > >>> >
> >>>> > > >>> >
> >>>> > > >>>
> >>>> > > >>
> >>>> > > >>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >
>
Re: DAG Level permissions (was Re: RBAC Update)
Posted by Martin Gauthier <ga...@gmail.com>.
Nice work Feng. We were waiting for this from our end.
Looking forward to contributing for the upcoming feature for DAG group
access control.
Martin
>
>
>
>
Re: DAG Level permissions (was Re: RBAC Update)
Posted by Naik Kaxil <k....@reply.com>.
Awesome work Feng. Well done.
On 17/07/2018, 13:27, "James Meickle" <jm...@quantopian.com.INVALID> wrote:
Really excited for this one - we have a lot of internal access controls and
this will help us implement them properly. It's going to be great being
able to give everyone access to see the overall state of DAG progress
without seeing its parameters or logs!
On Tue, Jul 17, 2018 at 12:48 AM, Ruiqin Yang <yr...@gmail.com> wrote:
> Congratulations! Extraordinary work! Thank you very much! This has been a
> highly desired feature for us for quite a while.
>
> Cheers,
> Kevin Yang
>
> Tao Feng <fe...@gmail.com>于2018年7月16日 周一下午9:30写道：
>
> > Hi,
> >
> > Just want to give an update that Airflow DAG level access just checked in
> > today(https://github.com/apache/incubator-airflow/pull/3197). Thanks a
> lot
> > for Max and Joy's review which helps me improving the pr. I create the
> > following three tickets as a follow up:
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2694 # Allow parsing
> access
> > control in DAG file
> > https://issues.apache.org/jira/browse/AIRFLOW-2693 # Implement sync_perm
> > model view endpoint
> > https://issues.apache.org/jira/browse/AIRFLOW-2695 # Support assign
> groups
> > of dag permission to a role
> >
> > I will start working on them in Q3.
> >
> > Thanks a lot,
> > -Tao
> >
> > On Sun, Apr 8, 2018 at 11:26 AM, Tao Feng <fe...@gmail.com> wrote:
> >
> > > Hi Max, Joy and Everyone,
> > >
> > > Based on the discussion, I put up a work in progress pr (
> > > https://github.com/apache/incubator-airflow/pull/3197/) with a doc(
> > > https://docs.google.com/document/d/1qs26lE9kAuCY0Qa0ga-8
> > > 0EQ7d7m4s-590lhjtMBjmxw/edit#) for DAG level access. I would like to
> get
> > > some early feedbacks and see if I miss anything or am in the wrong
> > > direction as it touches the core part.
> > >
> > > In the meantime, I will still continue improving the pr for couples of
> > > todos.
> > >
> > > Looking forward to your feedback.
> > >
> > > Thanks,
> > > -Tao
> > >
> > > On Mon, Apr 2, 2018 at 2:44 PM, Tao Feng <fe...@gmail.com> wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> Thanks a lot for all the great discussions. To summarize in brief,
> here
> > >> are the few approaches we discussed so far:
> > >>
> > >> 1. One permission per DAG. The user has homogenous rights on the dag.
> > >> The concerns:
> > >>
> > >> - not flexible to certain use cases(e.g the user has view only
> access
> > >> on certain dags and write access on certain other dags );
> > >> - difficult to implement as it breaks the existing FAB security
> > model.
> > >>
> > >> 2. Extend current model(ab_permission_view_role) with an additional
> > >> optional column (dag_id).
> > >> The concerns:
> > >>
> > >> - introduce a 3rd column to existing permission_view table.
> > >> - It requires us to handle db migration carefully from view only
> > >> access UI to dag level access UI.
> > >>
> > >> 3. Define a set of pre-defined dag-level permissions. Reuse the
> current
> > >> existing permission_view model and consider each individual dag as a
> new
> > >> view.
> > >>
> > >> It seems that the 3rd approach is a preferable approach which makes
> the
> > >> security model easy to extend without introducing additional
> > complexity. If
> > >> no other concern, I will work with Max to create an initial proposal /
> > PR
> > >> based on 3rd approach for the work(https://issues.apache.org
> > >> /jira/browse/AIRFLOW-2267).
> > >>
> > >> Thanks,
> > >> -Tao
> > >>
> > >> On Sat, Mar 31, 2018 at 12:09 AM, Joy Gao <jo...@wepay.com> wrote:
> > >>
> > >>> +1!
> > >>>
> > >>> I was originally tempted to re-use existing perms and views for
> > dag-level
> > >>> access control since dag-level perm/view is a subset of view-level
> > >>> perm/view, but your proposal of defining new dag-level perms/views
> > >>> independent from view-level perms/views is interesting. This actually
> > >>> makes
> > >>> a lot of sense, since even in the existing models, views can also be
> > menu
> > >>> options, so we are simply extending the concept of views to include
> > dags
> > >>> as
> > >>> well.
> > >>>
> > >>> On Fri, Mar 30, 2018 at 5:24 PM Maxime Beauchemin <
> > >>> maximebeauchemin@gmail.com> wrote:
> > >>>
> > >>> > I'd suggest something else that avoids having to add a 3rd column.
> I
> > >>> think
> > >>> > we can fit our use case into the existing structure nicely.
> > >>> >
> > >>> > My idea is to mimic what FAB does with its own Models.
> > >>> >
> > >>> > When you create a Model and ModelView in FAB (say DagRun for
> > example),
> > >>> it
> > >>> > creates a new view_menu (DagRun) and matches it with existing
> > >>> permission
> > >>> > (can_read, can_delete, can_edit, can_show, ...) to create a new set
> > of
> > >>> > "permission_views" which are combinations of permission and
> view_menu
> > >>> as in
> > >>> > "DagRun - can_read", "DagRun - can_edit", ... It's not a cartesian
> > >>> product
> > >>> > of all perms and view_menus, it's a predetermined list of
> > >>> model-specific
> > >>> > perms that get combined with DagRun here.
> > >>> >
> > >>> > Similarly, when Airflow would detect a new DAG (example "my_dag"),
> it
> > >>> > would create a new view_menu `my_dag`, and match it with
> permissions
> > >>> that
> > >>> > are identified as "combinable" with DAG. To avoid potential
> conflicts
> > >>> I'd
> > >>> > create new permissions that are DAG-specific like "dag_clear",
> > >>> "dag_run",
> > >>> > "dag_view".
> > >>> >
> > >>> > I'm arguing about the how to use the FAB model here specifically. I
> > >>> think
> > >>> > this allows for all the flexibility we need without changing the
> > >>> model. I
> > >>> > care less about what exactly the atomicity of the per DAG perms
> > should
> > >>> look
> > >>> > like. As far as I'm concerned, per-DAG read and write is probably
> > >>> granular
> > >>> > enough
> > >>> >
> > >>> > Also note that:
> > >>> > * we need an "_all_dags" logical DAG, meaning we'd have extra
> > >>> > permission_views "_all_dags - dag_view", "_all_dags - dag_run",
> > >>> "all_dags -
> > >>> > dag_clear"
> > >>> > * we probably want to derive FAB's SecurityManger and have an
> > >>> > AirflowSecurityManager that has an extra method
> "can_dag_action(user,
> > >>> > dag_id, action)". The SecurityManger class is neat because people
> can
> > >>> > provide their own if they want and override methods. That means
> that
> > >>> you
> > >>> > can defer any of the security-related checks to another system if
> you
> > >>> want
> > >>> > to. Many companies have some sort of company RBAC system and that
> can
> > >>> be
> > >>> > used to integrate with it.
> > >>> >
> > >>> > Max
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Fri, Mar 30, 2018 at 4:54 PM, Joy Gao <jo...@wepay.com> wrote:
> > >>> >
> > >>> >> Hi all,
> > >>> >>
> > >>> >> I also agree that having view-only access to some dags while write
> > >>> access
> > >>> >> to other dags is useful, so I prefer option 2. Although option 2
> is
> > >>> more
> > >>> >> difficult to manage, it is cleaner and more consistent with the
> > >>> current
> > >>> >> security model. (On the other hand, even though option 1 may be
> may
> > be
> > >>> >> easier to manage, it might be trickier to implement: with one perm
> > per
> > >>> >> dag, it breaks the existing FAB security model since the existing,
> > >>> more
> > >>> >> granular permissions will have to be grouped for each dag).
> > >>> >>
> > >>> >> What Brian suggested in the other thread makes sense to me...
> > >>> >>
> > >>> >> The current security model in FAB looks like the following:
> > >>> >>
> > >>> >> ---
> > >>> permissions
> > >>> >> user --- role --- permission_view ---|
> > >>> >> --- views
> > >>> >>
> > >>> >> FAB generates a few relationship tables to manage the mappings,
> one
> > of
> > >>> >> them is *ab_permission_view_role*, which has all the
> > >>> >> role-to-permission_view mapping. We can add a dag_id column to
> this
> > >>> >> relationship that expresses dag-level granularity, so the security
> > >>> model
> > >>> >> becomes:
> > >>> >>
> > >>> >>
> > ---
> > >>> >> permissions
> > >>> >> |
> > >>> >> user --- role --- dag_permission_view --- --- views
> > >>> >> |
> > >>> >>
> ---
> > >>> dags
> > >>> >>
> > >>> >> We can make the dag_id field an optional field in the
> relationship,
> > >>> and
> > >>> >> only lazily add new dag-level mappings for DAGs that specify
> 'access
> > >>> >> control'. This way we don't have to create new permissions or
> views.
> > >>> (We
> > >>> >> will still have to introduce new dag-level roles on top of the 5
> > >>> generic
> > >>> >> roles (public/viewer/user/op/admin)).
> > >>> >>
> > >>> >> (I think this is similar to what Arthur suggested earlier, but not
> > >>> sure
> > >>> >> if I interpreted correctly).
> > >>> >>
> > >>> >> Joy
> > >>> >>
> > >>> >> On Thu, Mar 29, 2018 at 10:01 AM, Arthur Wiedmer <
> > >>> >> arthur.wiedmer@gmail.com> wrote:
> > >>> >>
> > >>> >>> (Creating a new thread)
> > >>> >>>
> > >>> >>> Hi Max,
> > >>> >>>
> > >>> >>> I was just wondering about this. There are definite use cases for
> > >>> people
> > >>> >>> having only view access to some DAGs, mostly for monitoring. I
> want
> > >>> to know
> > >>> >>> what the upstream DAGs are doing, but maybe I don't need
> clear/run
> > >>> access.
> > >>> >>>
> > >>> >>> I feel like the granular operation permissions will start to
> become
> > >>> >>> unwieldy fast. It grows as (# users * # DAGs * # operations) but
> > with
> > >>> >>> hopefully a small constant factor in front: most people should
> only
> > >>> have a
> > >>> >>> small number of DAGs they care about. The Ops team has a need to
> > have
> > >>> >>> access to all on the other hand.
> > >>> >>>
> > >>> >>> I was wondering we could get by with something slightly more
> > complex
> > >>> but
> > >>> >>> easier on the size of that permissions table :
> > >>> >>> 1) One toggle on the user level for broad access (ALL:ALL,
> > >>> >>> RUN/CLEAR:ALL, VIEW:ALL) default NULL
> > >>> >>> 2) More granular permissions at the DAG level.
> > >>> >>>
> > >>> >>> So in order, check the user's broad level permission first, then
> > DAG
> > >>> >>> level. For large amounts of DAGs, this should help shave a little
> > >>> bit from
> > >>> >>> that table.
> > >>> >>>
> > >>> >>> Best,
> > >>> >>> Arthur
> > >>> >>>
> > >>> >>>
> > >>> >>> On Thu, Mar 29, 2018 at 8:27 AM, Maxime Beauchemin <
> > >>> >>> maximebeauchemin@gmail.com> wrote:
> > >>> >>>
> > >>> >>>> Hijacking the thread further here, any thoughts on how to
> > breakdown
> > >>> per
> > >>> >>>> DAG
> > >>> >>>> access?
> > >>> >>>>
> > >>> >>>> Tao & I are talking about introducing per-DAG permissions and
> one
> > >>> big
> > >>> >>>> question is whether we'll need to support different
> > operation-types
> > >>> at a
> > >>> >>>> per-DAG level, which changes the way we need to model the perms.
> > >>> >>>>
> > >>> >>>> First [simpler] option is to introduce one perm per DAG. If you
> > have
> > >>> >>>> access
> > >>> >>>> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have
> > >>> >>>> homogenous
> > >>> >>>> rights on the DAGs you have access to.
> > >>> >>>>
> > >>> >>>> Second option is to have a breakdown per DAG. Meaning for each
> DAG
> > >>> we
> > >>> >>>> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify,
> > >>> ...). So
> > >>> >>>> one
> > >>> >>>> user could have modify on some DAGs, view on others, and other
> > DAGs
> > >>> >>>> would
> > >>> >>>> be invisible. This could be broken down further
> > ({dag_id}_can_clear,
> > >>> >>>> ...)
> > >>> >>>> but it gets hard to manage.
> > >>> >>>>
> > >>> >>>> Thoughts?
> > >>> >>>>
> > >>> >>>> Max
> > >>> >>>>
> > >>> >>>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fengtao04@gmail.com
> >
> > >>> wrote:
> > >>> >>>>
> > >>> >>>> > Great work Joy. This is awesome! I am interested in helping
> out
> > >>> the
> > >>> >>>> per dag
> > >>> >>>> > level access. Just created a ticket to check(AIRFLOW-2267).
> Let
> > >>> me
> > >>> >>>> know if
> > >>> >>>> > you have any suggestions. I will share my proposal once I am
> > >>> ready.
> > >>> >>>> >
> > >>> >>>> > On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com>
> > wrote:
> > >>> >>>> >
> > >>> >>>> > > Hey guys!
> > >>> >>>> > >
> > >>> >>>> > > The RBAC UI <https://github.com/apache/inc
> > >>> ubator-airflow/pull/3015>
> > >>> >>>> has
> > >>> >>>> > > been merged to master. I'm looking forward to early
> adopters'
> > >>> >>>> feedback
> > >>> >>>> > and
> > >>> >>>> > > bug reports. I also hope to have more folks helping out with
> > the
> > >>> >>>> RBAC UI,
> > >>> >>>> > > especially with introducing DAG-Level access control, which
> > is a
> > >>> >>>> feature
> > >>> >>>> > > that a lot of people have been asking. If you are interested
> > in
> > >>> >>>> helping
> > >>> >>>> > out
> > >>> >>>> > > with this effort, let's talk more!
> > >>> >>>> > >
> > >>> >>>> > > This commit will be in the 1.10.0 release, and we are going
> to
> > >>> >>>> maintain
> > >>> >>>> > > both UIs simultaneously for a short period of time. Once
> RBAC
> > >>> UI is
> > >>> >>>> > stable
> > >>> >>>> > > and battle-tested, we will deprecate the old UI and
> eventually
> > >>> >>>> remove it
> > >>> >>>> > > from the repo (around Airflow 2.0.0 or 2.1.0 release). This
> is
> > >>> to
> > >>> >>>> prevent
> > >>> >>>> > > two UIs from forking into separate paths, as that would
> become
> > >>> very
> > >>> >>>> > > difficult to maintain.
> > >>> >>>> > >
> > >>> >>>> > > Going forward while both UIs are up, if you are making a
> > change
> > >>> to
> > >>> >>>> any
> > >>> >>>> > > files in airflow/www/ (old UI), where applicable, please
> also
> > >>> make
> > >>> >>>> the
> > >>> >>>> > > change to the airflow/www_rbac/ (new UI). If you rather not
> > make
> > >>> >>>> changes
> > >>> >>>> > in
> > >>> >>>> > > both UIs, it is recommended that you only make the changes
> to
> > >>> the
> > >>> >>>> RBAC
> > >>> >>>> > UI,
> > >>> >>>> > > since that is the one we are maintaining in the long term.
> > >>> >>>> > >
> > >>> >>>> > > I'm excited that the RBAC UI will be able to bring
> additional
> > >>> >>>> security to
> > >>> >>>> > > Airflow, and with FAB framework in place we can look into
> > >>> >>>> leveraging it
> > >>> >>>> > for
> > >>> >>>> > > a unified set of APIs used by both UI and CLI.
> > >>> >>>> > >
> > >>> >>>> > > Joy
> > >>> >>>> > >
> > >>> >>>> > >
> > >>> >>>> > >
> > >>> >>>> > > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com>
> > >>> wrote:
> > >>> >>>> > >
> > >>> >>>> > > > Hi folks,
> > >>> >>>> > > >
> > >>> >>>> > > > I have a PR <https://github.com/apache/
> > >>> >>>> incubator-airflow/pull/3015>
> > >>> >>>> > out
> > >>> >>>> > > > for the new UI. I've included instructions on how to test
> it
> > >>> out
> > >>> >>>> in the
> > >>> >>>> > > PR
> > >>> >>>> > > > description. Looking forward to your feedbacks.
> > >>> >>>> > > >
> > >>> >>>> > > > Cheers,
> > >>> >>>> > > > Joy
> > >>> >>>> > > >
> > >>> >>>> > > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com>
> > >>> wrote:
> > >>> >>>> > > >
> > >>> >>>> > > >> Thanks for the background info. Would be really awesome
> for
> > >>> you
> > >>> >>>> to
> > >>> >>>> > have
> > >>> >>>> > > >> PyPi access :D I'll make the change to have Airflow
> > >>> Webserver's
> > >>> >>>> FAB
> > >>> >>>> > > >> dependency pointing to my fork for the mean time.
> > >>> >>>> > > >>
> > >>> >>>> > > >> For folks who are interested in RBAC, I will be giving a
> > >>> >>>> talk/demo at
> > >>> >>>> > > the Airflow
> > >>> >>>> > > >> Meet-Up
> > >>> >>>> > > >> <
> > https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> > >>> >>>> > > Meetup/events/244525050/>
> > >>> >>>> > > >> next Monday. Happy to chat afterwards about it as well :)
> > >>> >>>> > > >>
> > >>> >>>> > > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> > >>> >>>> > > >> maximebeauchemin@gmail.com> wrote:
> > >>> >>>> > > >>
> > >>> >>>> > > >>> A bit of related history here:
> > >>> >>>> > > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> > >>> >>>> > > >>>
> > >>> >>>> > > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> > >>> >>>> > > >>> maximebeauchemin@gmail.com> wrote:
> > >>> >>>> > > >>>
> > >>> >>>> > > >>> > Given I have merge rights on FAB I could probably do
> > >>> another
> > >>> >>>> round
> > >>> >>>> > of
> > >>> >>>> > > >>> > review and get your PRs through. I would really like
> to
> > >>> get
> > >>> >>>> the
> > >>> >>>> > main
> > >>> >>>> > > >>> > maintainer's input on things that touch the core
> > >>> >>>> (composite-key
> > >>> >>>> > > >>> support) as
> > >>> >>>> > > >>> > he might have concerns/intuitions that we can't know
> > >>> about.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > I do not have Pypi access though so I cannot push new
> > >>> releases
> > >>> >>>> > out. I
> > >>> >>>> > > >>> > could ask for that.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > I've threatened to fork the project before, that's
> > always
> > >>> an
> > >>> >>>> > option.
> > >>> >>>> > > >>> I've
> > >>> >>>> > > >>> > noticed his involvement is sporadic and comes in
> bursts.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > In the meantime, you can have the dependency in
> Airflow
> > >>> >>>> Webserver
> > >>> >>>> > > >>> pointing
> > >>> >>>> > > >>> > straight to your fork.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > Max
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <
> > joyg@wepay.com>
> > >>> >>>> wrote:
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> >> I just created a new webserver instance if you
> haven't
> > >>> >>>> gotten a
> > >>> >>>> > > >>> chance to
> > >>> >>>> > > >>> >> fiddle around with the new web UI and the RBAC
> > >>> configurations
> > >>> >>>> > > (thanks
> > >>> >>>> > > >>> >> Maxime for getting started with this earlier!):
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> http://104.209.38.171:8080/
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> Admin Account
> > >>> >>>> > > >>> >> username: admin
> > >>> >>>> > > >>> >> password: admin
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> Read-Only Account
> > >>> >>>> > > >>> >> username: viewer
> > >>> >>>> > > >>> >> password: password
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <
> > joyg@wepay.com
> > >>> >
> > >>> >>>> wrote:
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> > Hi folks,
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > Thanks for all the feedback regarding to the new
> > >>> Airflow
> > >>> >>>> > Webserver
> > >>> >>>> > > >>> UI
> > >>> >>>> > > >>> >> > <https://github.com/wepay/airflow-webserver/>!
> I've
> > >>> been
> > >>> >>>> > actively
> > >>> >>>> > > >>> >> > addressing all the bugs that were raised on Github.
> > So
> > >>> I
> > >>> >>>> want to
> > >>> >>>> > > >>> take
> > >>> >>>> > > >>> >> this
> > >>> >>>> > > >>> >> > opportunity to discuss two issues coming up:
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > The first issue is unaddressed PRs in FAB. If these
> > PRs
> > >>> >>>> continue
> > >>> >>>> > > to
> > >>> >>>> > > >>> stay
> > >>> >>>> > > >>> >> > unaddressed, RBAC is blocked from making further
> > >>> progress.
> > >>> >>>> If
> > >>> >>>> > this
> > >>> >>>> > > >>> >> continue
> > >>> >>>> > > >>> >> > to be an issue, I'm inclined to fork FAB, even
> though
> > >>> it's
> > >>> >>>> not
> > >>> >>>> > > >>> >> idealistic.
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > - PR/631 <https://github.com/dpgaspar/F
> > >>> >>>> > > lask-AppBuilder/pull/631>
> > >>> >>>> > > >>> >> Binary
> > >>> >>>> > > >>> >> > column support (merged, unreleased)
> > >>> >>>> > > >>> >> > <https://github.com/dpgaspar/F
> > >>> lask-AppBuilder/pull/631>
> > >>> >>>> > > >>> >> > - PR/639 <https://github.com/dpgaspar/F
> > >>> >>>> > > lask-AppBuilder/pull/639>
> > >>> >>>> > > >>> >> Composite
> > >>> >>>> > > >>> >> > primary key support (unmerged)
> > >>> >>>> > > >>> >> > - PR/655 <https://github.com/dpgaspar/F
> > >>> >>>> > > lask-AppBuilder/pull/655>
> > >>> >>>> > > >>> >> Form
> > >>> >>>> > > >>> >> > prefill support (unmerged)
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > The second issue is an open question about the next
> > >>> step of
> > >>> >>>> > > Airflow
> > >>> >>>> > > >>> >> > Webserver itself. Here are the 3 potential
> directions
> > >>> we
> > >>> >>>> could
> > >>> >>>> > > >>> take, and
> > >>> >>>> > > >>> >> > I've added my thought on each.
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > 1. Permanently keep Airflow Webserver as a
> separated
> > >>> >>>> package
> > >>> >>>> > from
> > >>> >>>> > > >>> >> Airflow,
> > >>> >>>> > > >>> >> > and treat it as another UI option. Keep `www` in
> > >>> Airflow.
> > >>> >>>> Allow
> > >>> >>>> > > >>> >> development
> > >>> >>>> > > >>> >> > on both UIs.
> > >>> >>>> > > >>> >> > *I'm not a fan of this. When there is an existing
> UI
> > in
> > >>> >>>> Airflow,
> > >>> >>>> > > >>> most
> > >>> >>>> > > >>> >> > contributors would prefer to maintain the official
> > >>> version
> > >>> >>>> that
> > >>> >>>> > is
> > >>> >>>> > > >>> >> > installed out-of-the-box. **Having a second UI
> > outside
> > >>> of
> > >>> >>>> > Airflow
> > >>> >>>> > > >>> will
> > >>> >>>> > > >>> >> > make maintaining it very difficult, leading to an
> > >>> eventual
> > >>> >>>> death
> > >>> >>>> > > of
> > >>> >>>> > > >>> the
> > >>> >>>> > > >>> >> new
> > >>> >>>> > > >>> >> > UI :(*
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > 2. Permanently keep Airflow Webserver as a
> separated
> > >>> >>>> package
> > >>> >>>> > from
> > >>> >>>> > > >>> >> Airflow,
> > >>> >>>> > > >>> >> > but freeze all development on `www` and direct all
> > >>> future
> > >>> >>>> UI
> > >>> >>>> > > >>> >> development
> > >>> >>>> > > >>> >> > to Airflow Webserver, eventually removing `www`
> > >>> completely
> > >>> >>>> when
> > >>> >>>> > > >>> Airflow
> > >>> >>>> > > >>> >> > Webserver is stable.
> > >>> >>>> > > >>> >> > *I'm not a fan of this either. First of all, the
> > views
> > >>> and
> > >>> >>>> > models
> > >>> >>>> > > >>> are
> > >>> >>>> > > >>> >> > tightly coupled in both old and new UI; until we
> > have a
> > >>> >>>> > > full-fledged
> > >>> >>>> > > >>> >> REST
> > >>> >>>> > > >>> >> > API to build the UI (and cli) on top of it,
> > separating
> > >>> >>>> them to a
> > >>> >>>> > > >>> >> separate
> > >>> >>>> > > >>> >> > package now will potentially cause dependency
> issues
> > >>> and
> > >>> >>>> add
> > >>> >>>> > > >>> >> complication
> > >>> >>>> > > >>> >> > to our release cycle. **Secondly, **majority of
> > Airflow
> > >>> >>>> users
> > >>> >>>> > run
> > >>> >>>> > > >>> >> Airflow
> > >>> >>>> > > >>> >> > with the UI; it's one of Airflow's best features.
> > >>> >>>> Separating UI
> > >>> >>>> > > out
> > >>> >>>> > > >>> of
> > >>> >>>> > > >>> >> > Airflow core will complicate setup and
> configuration,
> > >>> while
> > >>> >>>> > making
> > >>> >>>> > > >>> >> Airflow
> > >>> >>>> > > >>> >> > core less complete.*
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > 3. Merge Airflow Webserver back into Airflow as
> > `www2`,
> > >>> >>>> freeze
> > >>> >>>> > all
> > >>> >>>> > > >>> >> > development on `www`, eventually removing `www`
> > >>> completely
> > >>> >>>> when
> > >>> >>>> > > >>> `www2`
> > >>> >>>> > > >>> >> is
> > >>> >>>> > > >>> >> > stable.
> > >>> >>>> > > >>> >> > *This makes the most sense to me. Airflow Webserver
> > is
> > >>> >>>> developed
> > >>> >>>> > > >>> with
> > >>> >>>> > > >>> >> the
> > >>> >>>> > > >>> >> > goal of feature parity to the current UI, plus
> > >>> additional
> > >>> >>>> RBAC
> > >>> >>>> > > >>> >> capability,
> > >>> >>>> > > >>> >> > in hope to replace the old UI completely. Yes, this
> > >>> means
> > >>> >>>> there
> > >>> >>>> > > >>> will be
> > >>> >>>> > > >>> >> a
> > >>> >>>> > > >>> >> > short period of having to maintain two UIs, but
> once
> > we
> > >>> >>>> freeze
> > >>> >>>> > > >>> >> development
> > >>> >>>> > > >>> >> > on www, it shouldn't be a concern for long.*
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > I'd love to hear everyone's thoughts on this! I'm
> > >>> excited
> > >>> >>>> about
> > >>> >>>> > > >>> bringing
> > >>> >>>> > > >>> >> > RBAC to airflow and I hope it's something others
> will
> > >>> find
> > >>> >>>> > useful
> > >>> >>>> > > as
> > >>> >>>> > > >>> >> well!
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > Cheers,
> > >>> >>>> > > >>> >> > Joy
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <
> > >>> joyg@wepay.com>
> > >>> >>>> > wrote:
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> >> Thank you everyone for the active feedback so far,
> > and
> > >>> >>>> thanks
> > >>> >>>> > for
> > >>> >>>> > > >>> >> setting
> > >>> >>>> > > >>> >> >> up the demo Maxime!
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Going to work on pruning through the issues in the
> > >>> >>>> upcoming
> > >>> >>>> > days.
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy
> Exception
> > >>> >>>> message
> > >>> >>>> > so I
> > >>> >>>> > > >>> can
> > >>> >>>> > > >>> >> >> look into it? Otherwise I'll wait until it's down
> > >>> again =P
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Cheers,
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Joy
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime
> Beauchemin <
> > >>> >>>> > > >>> >> >> maximebeauchemin@gmail.com> wrote:
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >>> I just restarted it, not sure how long it will
> take
> > >>> to
> > >>> >>>> get in
> > >>> >>>> > a
> > >>> >>>> > > >>> bad
> > >>> >>>> > > >>> >> state
> > >>> >>>> > > >>> >> >>> again...
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>> Max
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong,
> Fokko
> > >>> >>>> > > >>> >> <fokko@driesprong.frl
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> wrote:
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>> > Good morning,
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > The demo provided by Max is down, it throws a
> > >>> >>>> > > >>> SQLAlchemyexception
> > >>> >>>> > > >>> >> :'(
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > Cheers, Fokko
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> > >>> >>>> > > >>> criccomini@apache.org>:
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > > @bolke, open issues on the Github repo,
> please.
> > >>> >>>> > > >>> >> >>> > >
> > >>> >>>> > > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de
> > Bruin
> > >>> <
> > >>> >>>> > > >>> >> bdbruin@gmail.com>
> > >>> >>>> > > >>> >> >>> > > wrote:
> > >>> >>>> > > >>> >> >>> > >
> > >>> >>>> > > >>> >> >>> > > > Chris,
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > > Do you want us to report bugs somewhere (I
> > have
> > >>> >>>> > > encountered
> > >>> >>>> > > >>> a
> > >>> >>>> > > >>> >> >>> few)? Or
> > >>> >>>> > > >>> >> >>> > > > just generic user experiences posted here?
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > > Cheers
> > >>> >>>> > > >>> >> >>> > > > Bolke
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > > >
Kaxil Naik
Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.naik@reply.com
www.reply.com
On 18 Nov 2017, at 00:47, Chris
> Riccomini <
> > >>> >>>> > > >>> >> criccomini@apache.org
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > > wrote:
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > Hey all,
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > I know the weekend is coming up, and for
> > >>> those
> > >>> >>>> of us
> > >>> >>>> > in
> > >>> >>>> > > >>> the
> > >>> >>>> > > >>> >> US,
> > >>> >>>> > > >>> >> >>> next
> > >>> >>>> > > >>> >> >>> > > week
> > >>> >>>> > > >>> >> >>> > > > > is a bit of a slow holiday week. Would
> love
> > >>> to
> > >>> >>>> get
> > >>> >>>> > some
> > >>> >>>> > > >>> >> feedback
> > >>> >>>> > > >>> >> >>> from
> > >>> >>>> > > >>> >> >>> > > > > everyone on this. The goal would ideally
> to
> > >>> be to
> > >>> >>>> > > >>> converge on
> > >>> >>>> > > >>> >> >>> this
> > >>> >>>> > > >>> >> >>> > and
> > >>> >>>> > > >>> >> >>> > > > > eventually replace the existing Airflow
> UI
> > >>> with
> > >>> >>>> this
> > >>> >>>> > > one.
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > Cheers,
> > >>> >>>> > > >>> >> >>> > > > > Chris
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao
> <
> > >>> >>>> > > joyg@wepay.com>
> > >>> >>>> > > >>> >> wrote:
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > >> Hi guys.
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> I've been working on moving airflow from
> > >>> >>>> Flask-Admin
> > >>> >>>> > to
> > >>> >>>> > > >>> >> >>> > > Flask-AppBuilder
> > >>> >>>> > > >>> >> >>> > > > >> for RBAC
> > >>> >>>> > > >>> >> >>> > > > >> <https://cwiki.apache.org/
> > >>> >>>> > confluence/display/AIRFLOW/
> > >>> >>>> > > >>> >> >>> > > > Airflow+RBAC+proposal
> > >>> >>>> > > >>> >> >>> > > > >>> ,
> > >>> >>>> > > >>> >> >>> > > > >> check it out at
> > >>> https://github.com/wepay/airfl
> > >>> >>>> > > >>> ow-webserver.
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> It's still a work-in-progress, but most
> > >>> >>>> features you
> > >>> >>>> > > see
> > >>> >>>> > > >>> in
> > >>> >>>> > > >>> >> the
> > >>> >>>> > > >>> >> >>> > > > webserver
> > >>> >>>> > > >>> >> >>> > > > >> UI today is available there. For those
> who
> > >>> are
> > >>> >>>> > > >>> interested in
> > >>> >>>> > > >>> >> >>> RBAC,
> > >>> >>>> > > >>> >> >>> > I'd
> > >>> >>>> > > >>> >> >>> > > > love
> > >>> >>>> > > >>> >> >>> > > > >> to get some early feedback in terms of
> the
> > >>> >>>> following:
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any
> > >>> bugs/regressions)
> > >>> >>>> > > >>> >> >>> > > > >> - Setup issues
> > >>> >>>> > > >>> >> >>> > > > >> - Ease of integration with third party
> > auth
> > >>> >>>> (i.e.
> > >>> >>>> > LDAP,
> > >>> >>>> > > >>> AD,
> > >>> >>>> > > >>> >> >>> OAuth,
> > >>> >>>> > > >>> >> >>> > > etc.)
> > >>> >>>> > > >>> >> >>> > > > >> - Any other thoughts/concerns
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> Thanks a lot!
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> Cheers,
> > >>> >>>> > > >>> >> >>> > > > >> Joy
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > >
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>>
> > >>> >>>> > > >>
> > >>> >>>> > > >>
> > >>> >>>> > > >
> > >>> >>>> > >
> > >>> >>>> >
> > >>> >>>>
> > >>> >>>
> > >>> >>>
> > >>> >>
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: DAG Level permissions (was Re: RBAC Update)
Posted by James Meickle <jm...@quantopian.com.INVALID>.
Really excited for this one - we have a lot of internal access controls and
this will help us implement them properly. It's going to be great being
able to give everyone access to see the overall state of DAG progress
without seeing its parameters or logs!
On Tue, Jul 17, 2018 at 12:48 AM, Ruiqin Yang <yr...@gmail.com> wrote:
> Congratulations! Extraordinary work! Thank you very much! This has been a
> highly desired feature for us for quite a while.
>
> Cheers,
> Kevin Yang
>
> Tao Feng <fe...@gmail.com>于2018年7月16日 周一下午9:30写道：
>
> > Hi,
> >
> > Just want to give an update that Airflow DAG level access just checked in
> > today(https://github.com/apache/incubator-airflow/pull/3197). Thanks a
> lot
> > for Max and Joy's review which helps me improving the pr. I create the
> > following three tickets as a follow up:
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2694 # Allow parsing
> access
> > control in DAG file
> > https://issues.apache.org/jira/browse/AIRFLOW-2693 # Implement sync_perm
> > model view endpoint
> > https://issues.apache.org/jira/browse/AIRFLOW-2695 # Support assign
> groups
> > of dag permission to a role
> >
> > I will start working on them in Q3.
> >
> > Thanks a lot,
> > -Tao
> >
> > On Sun, Apr 8, 2018 at 11:26 AM, Tao Feng <fe...@gmail.com> wrote:
> >
> > > Hi Max, Joy and Everyone,
> > >
> > > Based on the discussion, I put up a work in progress pr (
> > > https://github.com/apache/incubator-airflow/pull/3197/) with a doc(
> > > https://docs.google.com/document/d/1qs26lE9kAuCY0Qa0ga-8
> > > 0EQ7d7m4s-590lhjtMBjmxw/edit#) for DAG level access. I would like to
> get
> > > some early feedbacks and see if I miss anything or am in the wrong
> > > direction as it touches the core part.
> > >
> > > In the meantime, I will still continue improving the pr for couples of
> > > todos.
> > >
> > > Looking forward to your feedback.
> > >
> > > Thanks,
> > > -Tao
> > >
> > > On Mon, Apr 2, 2018 at 2:44 PM, Tao Feng <fe...@gmail.com> wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> Thanks a lot for all the great discussions. To summarize in brief,
> here
> > >> are the few approaches we discussed so far:
> > >>
> > >> 1. One permission per DAG. The user has homogenous rights on the dag.
> > >> The concerns:
> > >>
> > >> - not flexible to certain use cases(e.g the user has view only
> access
> > >> on certain dags and write access on certain other dags );
> > >> - difficult to implement as it breaks the existing FAB security
> > model.
> > >>
> > >> 2. Extend current model(ab_permission_view_role) with an additional
> > >> optional column (dag_id).
> > >> The concerns:
> > >>
> > >> - introduce a 3rd column to existing permission_view table.
> > >> - It requires us to handle db migration carefully from view only
> > >> access UI to dag level access UI.
> > >>
> > >> 3. Define a set of pre-defined dag-level permissions. Reuse the
> current
> > >> existing permission_view model and consider each individual dag as a
> new
> > >> view.
> > >>
> > >> It seems that the 3rd approach is a preferable approach which makes
> the
> > >> security model easy to extend without introducing additional
> > complexity. If
> > >> no other concern, I will work with Max to create an initial proposal /
> > PR
> > >> based on 3rd approach for the work(https://issues.apache.org
> > >> /jira/browse/AIRFLOW-2267).
> > >>
> > >> Thanks,
> > >> -Tao
> > >>
> > >> On Sat, Mar 31, 2018 at 12:09 AM, Joy Gao <jo...@wepay.com> wrote:
> > >>
> > >>> +1!
> > >>>
> > >>> I was originally tempted to re-use existing perms and views for
> > dag-level
> > >>> access control since dag-level perm/view is a subset of view-level
> > >>> perm/view, but your proposal of defining new dag-level perms/views
> > >>> independent from view-level perms/views is interesting. This actually
> > >>> makes
> > >>> a lot of sense, since even in the existing models, views can also be
> > menu
> > >>> options, so we are simply extending the concept of views to include
> > dags
> > >>> as
> > >>> well.
> > >>>
> > >>> On Fri, Mar 30, 2018 at 5:24 PM Maxime Beauchemin <
> > >>> maximebeauchemin@gmail.com> wrote:
> > >>>
> > >>> > I'd suggest something else that avoids having to add a 3rd column.
> I
> > >>> think
> > >>> > we can fit our use case into the existing structure nicely.
> > >>> >
> > >>> > My idea is to mimic what FAB does with its own Models.
> > >>> >
> > >>> > When you create a Model and ModelView in FAB (say DagRun for
> > example),
> > >>> it
> > >>> > creates a new view_menu (DagRun) and matches it with existing
> > >>> permission
> > >>> > (can_read, can_delete, can_edit, can_show, ...) to create a new set
> > of
> > >>> > "permission_views" which are combinations of permission and
> view_menu
> > >>> as in
> > >>> > "DagRun - can_read", "DagRun - can_edit", ... It's not a cartesian
> > >>> product
> > >>> > of all perms and view_menus, it's a predetermined list of
> > >>> model-specific
> > >>> > perms that get combined with DagRun here.
> > >>> >
> > >>> > Similarly, when Airflow would detect a new DAG (example "my_dag"),
> it
> > >>> > would create a new view_menu `my_dag`, and match it with
> permissions
> > >>> that
> > >>> > are identified as "combinable" with DAG. To avoid potential
> conflicts
> > >>> I'd
> > >>> > create new permissions that are DAG-specific like "dag_clear",
> > >>> "dag_run",
> > >>> > "dag_view".
> > >>> >
> > >>> > I'm arguing about the how to use the FAB model here specifically. I
> > >>> think
> > >>> > this allows for all the flexibility we need without changing the
> > >>> model. I
> > >>> > care less about what exactly the atomicity of the per DAG perms
> > should
> > >>> look
> > >>> > like. As far as I'm concerned, per-DAG read and write is probably
> > >>> granular
> > >>> > enough
> > >>> >
> > >>> > Also note that:
> > >>> > * we need an "_all_dags" logical DAG, meaning we'd have extra
> > >>> > permission_views "_all_dags - dag_view", "_all_dags - dag_run",
> > >>> "all_dags -
> > >>> > dag_clear"
> > >>> > * we probably want to derive FAB's SecurityManger and have an
> > >>> > AirflowSecurityManager that has an extra method
> "can_dag_action(user,
> > >>> > dag_id, action)". The SecurityManger class is neat because people
> can
> > >>> > provide their own if they want and override methods. That means
> that
> > >>> you
> > >>> > can defer any of the security-related checks to another system if
> you
> > >>> want
> > >>> > to. Many companies have some sort of company RBAC system and that
> can
> > >>> be
> > >>> > used to integrate with it.
> > >>> >
> > >>> > Max
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Fri, Mar 30, 2018 at 4:54 PM, Joy Gao <jo...@wepay.com> wrote:
> > >>> >
> > >>> >> Hi all,
> > >>> >>
> > >>> >> I also agree that having view-only access to some dags while write
> > >>> access
> > >>> >> to other dags is useful, so I prefer option 2. Although option 2
> is
> > >>> more
> > >>> >> difficult to manage, it is cleaner and more consistent with the
> > >>> current
> > >>> >> security model. (On the other hand, even though option 1 may be
> may
> > be
> > >>> >> easier to manage, it might be trickier to implement: with one perm
> > per
> > >>> >> dag, it breaks the existing FAB security model since the existing,
> > >>> more
> > >>> >> granular permissions will have to be grouped for each dag).
> > >>> >>
> > >>> >> What Brian suggested in the other thread makes sense to me...
> > >>> >>
> > >>> >> The current security model in FAB looks like the following:
> > >>> >>
> > >>> >> ---
> > >>> permissions
> > >>> >> user --- role --- permission_view ---|
> > >>> >> --- views
> > >>> >>
> > >>> >> FAB generates a few relationship tables to manage the mappings,
> one
> > of
> > >>> >> them is *ab_permission_view_role*, which has all the
> > >>> >> role-to-permission_view mapping. We can add a dag_id column to
> this
> > >>> >> relationship that expresses dag-level granularity, so the security
> > >>> model
> > >>> >> becomes:
> > >>> >>
> > >>> >>
> > ---
> > >>> >> permissions
> > >>> >> |
> > >>> >> user --- role --- dag_permission_view --- --- views
> > >>> >> |
> > >>> >>
> ---
> > >>> dags
> > >>> >>
> > >>> >> We can make the dag_id field an optional field in the
> relationship,
> > >>> and
> > >>> >> only lazily add new dag-level mappings for DAGs that specify
> 'access
> > >>> >> control'. This way we don't have to create new permissions or
> views.
> > >>> (We
> > >>> >> will still have to introduce new dag-level roles on top of the 5
> > >>> generic
> > >>> >> roles (public/viewer/user/op/admin)).
> > >>> >>
> > >>> >> (I think this is similar to what Arthur suggested earlier, but not
> > >>> sure
> > >>> >> if I interpreted correctly).
> > >>> >>
> > >>> >> Joy
> > >>> >>
> > >>> >> On Thu, Mar 29, 2018 at 10:01 AM, Arthur Wiedmer <
> > >>> >> arthur.wiedmer@gmail.com> wrote:
> > >>> >>
> > >>> >>> (Creating a new thread)
> > >>> >>>
> > >>> >>> Hi Max,
> > >>> >>>
> > >>> >>> I was just wondering about this. There are definite use cases for
> > >>> people
> > >>> >>> having only view access to some DAGs, mostly for monitoring. I
> want
> > >>> to know
> > >>> >>> what the upstream DAGs are doing, but maybe I don't need
> clear/run
> > >>> access.
> > >>> >>>
> > >>> >>> I feel like the granular operation permissions will start to
> become
> > >>> >>> unwieldy fast. It grows as (# users * # DAGs * # operations) but
> > with
> > >>> >>> hopefully a small constant factor in front: most people should
> only
> > >>> have a
> > >>> >>> small number of DAGs they care about. The Ops team has a need to
> > have
> > >>> >>> access to all on the other hand.
> > >>> >>>
> > >>> >>> I was wondering we could get by with something slightly more
> > complex
> > >>> but
> > >>> >>> easier on the size of that permissions table :
> > >>> >>> 1) One toggle on the user level for broad access (ALL:ALL,
> > >>> >>> RUN/CLEAR:ALL, VIEW:ALL) default NULL
> > >>> >>> 2) More granular permissions at the DAG level.
> > >>> >>>
> > >>> >>> So in order, check the user's broad level permission first, then
> > DAG
> > >>> >>> level. For large amounts of DAGs, this should help shave a little
> > >>> bit from
> > >>> >>> that table.
> > >>> >>>
> > >>> >>> Best,
> > >>> >>> Arthur
> > >>> >>>
> > >>> >>>
> > >>> >>> On Thu, Mar 29, 2018 at 8:27 AM, Maxime Beauchemin <
> > >>> >>> maximebeauchemin@gmail.com> wrote:
> > >>> >>>
> > >>> >>>> Hijacking the thread further here, any thoughts on how to
> > breakdown
> > >>> per
> > >>> >>>> DAG
> > >>> >>>> access?
> > >>> >>>>
> > >>> >>>> Tao & I are talking about introducing per-DAG permissions and
> one
> > >>> big
> > >>> >>>> question is whether we'll need to support different
> > operation-types
> > >>> at a
> > >>> >>>> per-DAG level, which changes the way we need to model the perms.
> > >>> >>>>
> > >>> >>>> First [simpler] option is to introduce one perm per DAG. If you
> > have
> > >>> >>>> access
> > >>> >>>> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have
> > >>> >>>> homogenous
> > >>> >>>> rights on the DAGs you have access to.
> > >>> >>>>
> > >>> >>>> Second option is to have a breakdown per DAG. Meaning for each
> DAG
> > >>> we
> > >>> >>>> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify,
> > >>> ...). So
> > >>> >>>> one
> > >>> >>>> user could have modify on some DAGs, view on others, and other
> > DAGs
> > >>> >>>> would
> > >>> >>>> be invisible. This could be broken down further
> > ({dag_id}_can_clear,
> > >>> >>>> ...)
> > >>> >>>> but it gets hard to manage.
> > >>> >>>>
> > >>> >>>> Thoughts?
> > >>> >>>>
> > >>> >>>> Max
> > >>> >>>>
> > >>> >>>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fengtao04@gmail.com
> >
> > >>> wrote:
> > >>> >>>>
> > >>> >>>> > Great work Joy. This is awesome! I am interested in helping
> out
> > >>> the
> > >>> >>>> per dag
> > >>> >>>> > level access. Just created a ticket to check(AIRFLOW-2267).
> Let
> > >>> me
> > >>> >>>> know if
> > >>> >>>> > you have any suggestions. I will share my proposal once I am
> > >>> ready.
> > >>> >>>> >
> > >>> >>>> > On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com>
> > wrote:
> > >>> >>>> >
> > >>> >>>> > > Hey guys!
> > >>> >>>> > >
> > >>> >>>> > > The RBAC UI <https://github.com/apache/inc
> > >>> ubator-airflow/pull/3015>
> > >>> >>>> has
> > >>> >>>> > > been merged to master. I'm looking forward to early
> adopters'
> > >>> >>>> feedback
> > >>> >>>> > and
> > >>> >>>> > > bug reports. I also hope to have more folks helping out with
> > the
> > >>> >>>> RBAC UI,
> > >>> >>>> > > especially with introducing DAG-Level access control, which
> > is a
> > >>> >>>> feature
> > >>> >>>> > > that a lot of people have been asking. If you are interested
> > in
> > >>> >>>> helping
> > >>> >>>> > out
> > >>> >>>> > > with this effort, let's talk more!
> > >>> >>>> > >
> > >>> >>>> > > This commit will be in the 1.10.0 release, and we are going
> to
> > >>> >>>> maintain
> > >>> >>>> > > both UIs simultaneously for a short period of time. Once
> RBAC
> > >>> UI is
> > >>> >>>> > stable
> > >>> >>>> > > and battle-tested, we will deprecate the old UI and
> eventually
> > >>> >>>> remove it
> > >>> >>>> > > from the repo (around Airflow 2.0.0 or 2.1.0 release). This
> is
> > >>> to
> > >>> >>>> prevent
> > >>> >>>> > > two UIs from forking into separate paths, as that would
> become
> > >>> very
> > >>> >>>> > > difficult to maintain.
> > >>> >>>> > >
> > >>> >>>> > > Going forward while both UIs are up, if you are making a
> > change
> > >>> to
> > >>> >>>> any
> > >>> >>>> > > files in airflow/www/ (old UI), where applicable, please
> also
> > >>> make
> > >>> >>>> the
> > >>> >>>> > > change to the airflow/www_rbac/ (new UI). If you rather not
> > make
> > >>> >>>> changes
> > >>> >>>> > in
> > >>> >>>> > > both UIs, it is recommended that you only make the changes
> to
> > >>> the
> > >>> >>>> RBAC
> > >>> >>>> > UI,
> > >>> >>>> > > since that is the one we are maintaining in the long term.
> > >>> >>>> > >
> > >>> >>>> > > I'm excited that the RBAC UI will be able to bring
> additional
> > >>> >>>> security to
> > >>> >>>> > > Airflow, and with FAB framework in place we can look into
> > >>> >>>> leveraging it
> > >>> >>>> > for
> > >>> >>>> > > a unified set of APIs used by both UI and CLI.
> > >>> >>>> > >
> > >>> >>>> > > Joy
> > >>> >>>> > >
> > >>> >>>> > >
> > >>> >>>> > >
> > >>> >>>> > > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com>
> > >>> wrote:
> > >>> >>>> > >
> > >>> >>>> > > > Hi folks,
> > >>> >>>> > > >
> > >>> >>>> > > > I have a PR <https://github.com/apache/
> > >>> >>>> incubator-airflow/pull/3015>
> > >>> >>>> > out
> > >>> >>>> > > > for the new UI. I've included instructions on how to test
> it
> > >>> out
> > >>> >>>> in the
> > >>> >>>> > > PR
> > >>> >>>> > > > description. Looking forward to your feedbacks.
> > >>> >>>> > > >
> > >>> >>>> > > > Cheers,
> > >>> >>>> > > > Joy
> > >>> >>>> > > >
> > >>> >>>> > > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com>
> > >>> wrote:
> > >>> >>>> > > >
> > >>> >>>> > > >> Thanks for the background info. Would be really awesome
> for
> > >>> you
> > >>> >>>> to
> > >>> >>>> > have
> > >>> >>>> > > >> PyPi access :D I'll make the change to have Airflow
> > >>> Webserver's
> > >>> >>>> FAB
> > >>> >>>> > > >> dependency pointing to my fork for the mean time.
> > >>> >>>> > > >>
> > >>> >>>> > > >> For folks who are interested in RBAC, I will be giving a
> > >>> >>>> talk/demo at
> > >>> >>>> > > the Airflow
> > >>> >>>> > > >> Meet-Up
> > >>> >>>> > > >> <
> > https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> > >>> >>>> > > Meetup/events/244525050/>
> > >>> >>>> > > >> next Monday. Happy to chat afterwards about it as well :)
> > >>> >>>> > > >>
> > >>> >>>> > > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> > >>> >>>> > > >> maximebeauchemin@gmail.com> wrote:
> > >>> >>>> > > >>
> > >>> >>>> > > >>> A bit of related history here:
> > >>> >>>> > > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> > >>> >>>> > > >>>
> > >>> >>>> > > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> > >>> >>>> > > >>> maximebeauchemin@gmail.com> wrote:
> > >>> >>>> > > >>>
> > >>> >>>> > > >>> > Given I have merge rights on FAB I could probably do
> > >>> another
> > >>> >>>> round
> > >>> >>>> > of
> > >>> >>>> > > >>> > review and get your PRs through. I would really like
> to
> > >>> get
> > >>> >>>> the
> > >>> >>>> > main
> > >>> >>>> > > >>> > maintainer's input on things that touch the core
> > >>> >>>> (composite-key
> > >>> >>>> > > >>> support) as
> > >>> >>>> > > >>> > he might have concerns/intuitions that we can't know
> > >>> about.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > I do not have Pypi access though so I cannot push new
> > >>> releases
> > >>> >>>> > out. I
> > >>> >>>> > > >>> > could ask for that.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > I've threatened to fork the project before, that's
> > always
> > >>> an
> > >>> >>>> > option.
> > >>> >>>> > > >>> I've
> > >>> >>>> > > >>> > noticed his involvement is sporadic and comes in
> bursts.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > In the meantime, you can have the dependency in
> Airflow
> > >>> >>>> Webserver
> > >>> >>>> > > >>> pointing
> > >>> >>>> > > >>> > straight to your fork.
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > Max
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <
> > joyg@wepay.com>
> > >>> >>>> wrote:
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> >> I just created a new webserver instance if you
> haven't
> > >>> >>>> gotten a
> > >>> >>>> > > >>> chance to
> > >>> >>>> > > >>> >> fiddle around with the new web UI and the RBAC
> > >>> configurations
> > >>> >>>> > > (thanks
> > >>> >>>> > > >>> >> Maxime for getting started with this earlier!):
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> http://104.209.38.171:8080/
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> Admin Account
> > >>> >>>> > > >>> >> username: admin
> > >>> >>>> > > >>> >> password: admin
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> Read-Only Account
> > >>> >>>> > > >>> >> username: viewer
> > >>> >>>> > > >>> >> password: password
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <
> > joyg@wepay.com
> > >>> >
> > >>> >>>> wrote:
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >> > Hi folks,
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > Thanks for all the feedback regarding to the new
> > >>> Airflow
> > >>> >>>> > Webserver
> > >>> >>>> > > >>> UI
> > >>> >>>> > > >>> >> > <https://github.com/wepay/airflow-webserver/>!
> I've
> > >>> been
> > >>> >>>> > actively
> > >>> >>>> > > >>> >> > addressing all the bugs that were raised on Github.
> > So
> > >>> I
> > >>> >>>> want to
> > >>> >>>> > > >>> take
> > >>> >>>> > > >>> >> this
> > >>> >>>> > > >>> >> > opportunity to discuss two issues coming up:
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > The first issue is unaddressed PRs in FAB. If these
> > PRs
> > >>> >>>> continue
> > >>> >>>> > > to
> > >>> >>>> > > >>> stay
> > >>> >>>> > > >>> >> > unaddressed, RBAC is blocked from making further
> > >>> progress.
> > >>> >>>> If
> > >>> >>>> > this
> > >>> >>>> > > >>> >> continue
> > >>> >>>> > > >>> >> > to be an issue, I'm inclined to fork FAB, even
> though
> > >>> it's
> > >>> >>>> not
> > >>> >>>> > > >>> >> idealistic.
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > - PR/631 <https://github.com/dpgaspar/F
> > >>> >>>> > > lask-AppBuilder/pull/631>
> > >>> >>>> > > >>> >> Binary
> > >>> >>>> > > >>> >> > column support (merged, unreleased)
> > >>> >>>> > > >>> >> > <https://github.com/dpgaspar/F
> > >>> lask-AppBuilder/pull/631>
> > >>> >>>> > > >>> >> > - PR/639 <https://github.com/dpgaspar/F
> > >>> >>>> > > lask-AppBuilder/pull/639>
> > >>> >>>> > > >>> >> Composite
> > >>> >>>> > > >>> >> > primary key support (unmerged)
> > >>> >>>> > > >>> >> > - PR/655 <https://github.com/dpgaspar/F
> > >>> >>>> > > lask-AppBuilder/pull/655>
> > >>> >>>> > > >>> >> Form
> > >>> >>>> > > >>> >> > prefill support (unmerged)
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > The second issue is an open question about the next
> > >>> step of
> > >>> >>>> > > Airflow
> > >>> >>>> > > >>> >> > Webserver itself. Here are the 3 potential
> directions
> > >>> we
> > >>> >>>> could
> > >>> >>>> > > >>> take, and
> > >>> >>>> > > >>> >> > I've added my thought on each.
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > 1. Permanently keep Airflow Webserver as a
> separated
> > >>> >>>> package
> > >>> >>>> > from
> > >>> >>>> > > >>> >> Airflow,
> > >>> >>>> > > >>> >> > and treat it as another UI option. Keep `www` in
> > >>> Airflow.
> > >>> >>>> Allow
> > >>> >>>> > > >>> >> development
> > >>> >>>> > > >>> >> > on both UIs.
> > >>> >>>> > > >>> >> > *I'm not a fan of this. When there is an existing
> UI
> > in
> > >>> >>>> Airflow,
> > >>> >>>> > > >>> most
> > >>> >>>> > > >>> >> > contributors would prefer to maintain the official
> > >>> version
> > >>> >>>> that
> > >>> >>>> > is
> > >>> >>>> > > >>> >> > installed out-of-the-box. **Having a second UI
> > outside
> > >>> of
> > >>> >>>> > Airflow
> > >>> >>>> > > >>> will
> > >>> >>>> > > >>> >> > make maintaining it very difficult, leading to an
> > >>> eventual
> > >>> >>>> death
> > >>> >>>> > > of
> > >>> >>>> > > >>> the
> > >>> >>>> > > >>> >> new
> > >>> >>>> > > >>> >> > UI :(*
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > 2. Permanently keep Airflow Webserver as a
> separated
> > >>> >>>> package
> > >>> >>>> > from
> > >>> >>>> > > >>> >> Airflow,
> > >>> >>>> > > >>> >> > but freeze all development on `www` and direct all
> > >>> future
> > >>> >>>> UI
> > >>> >>>> > > >>> >> development
> > >>> >>>> > > >>> >> > to Airflow Webserver, eventually removing `www`
> > >>> completely
> > >>> >>>> when
> > >>> >>>> > > >>> Airflow
> > >>> >>>> > > >>> >> > Webserver is stable.
> > >>> >>>> > > >>> >> > *I'm not a fan of this either. First of all, the
> > views
> > >>> and
> > >>> >>>> > models
> > >>> >>>> > > >>> are
> > >>> >>>> > > >>> >> > tightly coupled in both old and new UI; until we
> > have a
> > >>> >>>> > > full-fledged
> > >>> >>>> > > >>> >> REST
> > >>> >>>> > > >>> >> > API to build the UI (and cli) on top of it,
> > separating
> > >>> >>>> them to a
> > >>> >>>> > > >>> >> separate
> > >>> >>>> > > >>> >> > package now will potentially cause dependency
> issues
> > >>> and
> > >>> >>>> add
> > >>> >>>> > > >>> >> complication
> > >>> >>>> > > >>> >> > to our release cycle. **Secondly, **majority of
> > Airflow
> > >>> >>>> users
> > >>> >>>> > run
> > >>> >>>> > > >>> >> Airflow
> > >>> >>>> > > >>> >> > with the UI; it's one of Airflow's best features.
> > >>> >>>> Separating UI
> > >>> >>>> > > out
> > >>> >>>> > > >>> of
> > >>> >>>> > > >>> >> > Airflow core will complicate setup and
> configuration,
> > >>> while
> > >>> >>>> > making
> > >>> >>>> > > >>> >> Airflow
> > >>> >>>> > > >>> >> > core less complete.*
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > 3. Merge Airflow Webserver back into Airflow as
> > `www2`,
> > >>> >>>> freeze
> > >>> >>>> > all
> > >>> >>>> > > >>> >> > development on `www`, eventually removing `www`
> > >>> completely
> > >>> >>>> when
> > >>> >>>> > > >>> `www2`
> > >>> >>>> > > >>> >> is
> > >>> >>>> > > >>> >> > stable.
> > >>> >>>> > > >>> >> > *This makes the most sense to me. Airflow Webserver
> > is
> > >>> >>>> developed
> > >>> >>>> > > >>> with
> > >>> >>>> > > >>> >> the
> > >>> >>>> > > >>> >> > goal of feature parity to the current UI, plus
> > >>> additional
> > >>> >>>> RBAC
> > >>> >>>> > > >>> >> capability,
> > >>> >>>> > > >>> >> > in hope to replace the old UI completely. Yes, this
> > >>> means
> > >>> >>>> there
> > >>> >>>> > > >>> will be
> > >>> >>>> > > >>> >> a
> > >>> >>>> > > >>> >> > short period of having to maintain two UIs, but
> once
> > we
> > >>> >>>> freeze
> > >>> >>>> > > >>> >> development
> > >>> >>>> > > >>> >> > on www, it shouldn't be a concern for long.*
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > I'd love to hear everyone's thoughts on this! I'm
> > >>> excited
> > >>> >>>> about
> > >>> >>>> > > >>> bringing
> > >>> >>>> > > >>> >> > RBAC to airflow and I hope it's something others
> will
> > >>> find
> > >>> >>>> > useful
> > >>> >>>> > > as
> > >>> >>>> > > >>> >> well!
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > Cheers,
> > >>> >>>> > > >>> >> > Joy
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <
> > >>> joyg@wepay.com>
> > >>> >>>> > wrote:
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >> >> Thank you everyone for the active feedback so far,
> > and
> > >>> >>>> thanks
> > >>> >>>> > for
> > >>> >>>> > > >>> >> setting
> > >>> >>>> > > >>> >> >> up the demo Maxime!
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Going to work on pruning through the issues in the
> > >>> >>>> upcoming
> > >>> >>>> > days.
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy
> Exception
> > >>> >>>> message
> > >>> >>>> > so I
> > >>> >>>> > > >>> can
> > >>> >>>> > > >>> >> >> look into it? Otherwise I'll wait until it's down
> > >>> again =P
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Cheers,
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> Joy
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime
> Beauchemin <
> > >>> >>>> > > >>> >> >> maximebeauchemin@gmail.com> wrote:
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >>> I just restarted it, not sure how long it will
> take
> > >>> to
> > >>> >>>> get in
> > >>> >>>> > a
> > >>> >>>> > > >>> bad
> > >>> >>>> > > >>> >> state
> > >>> >>>> > > >>> >> >>> again...
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>> Max
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong,
> Fokko
> > >>> >>>> > > >>> >> <fokko@driesprong.frl
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> wrote:
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>> > Good morning,
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > The demo provided by Max is down, it throws a
> > >>> >>>> > > >>> SQLAlchemyexception
> > >>> >>>> > > >>> >> :'(
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > Cheers, Fokko
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> > >>> >>>> > > >>> criccomini@apache.org>:
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > > @bolke, open issues on the Github repo,
> please.
> > >>> >>>> > > >>> >> >>> > >
> > >>> >>>> > > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de
> > Bruin
> > >>> <
> > >>> >>>> > > >>> >> bdbruin@gmail.com>
> > >>> >>>> > > >>> >> >>> > > wrote:
> > >>> >>>> > > >>> >> >>> > >
> > >>> >>>> > > >>> >> >>> > > > Chris,
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > > Do you want us to report bugs somewhere (I
> > have
> > >>> >>>> > > encountered
> > >>> >>>> > > >>> a
> > >>> >>>> > > >>> >> >>> few)? Or
> > >>> >>>> > > >>> >> >>> > > > just generic user experiences posted here?
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > > Cheers
> > >>> >>>> > > >>> >> >>> > > > Bolke
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris
> Riccomini <
> > >>> >>>> > > >>> >> criccomini@apache.org
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>> > > wrote:
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > Hey all,
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > I know the weekend is coming up, and for
> > >>> those
> > >>> >>>> of us
> > >>> >>>> > in
> > >>> >>>> > > >>> the
> > >>> >>>> > > >>> >> US,
> > >>> >>>> > > >>> >> >>> next
> > >>> >>>> > > >>> >> >>> > > week
> > >>> >>>> > > >>> >> >>> > > > > is a bit of a slow holiday week. Would
> love
> > >>> to
> > >>> >>>> get
> > >>> >>>> > some
> > >>> >>>> > > >>> >> feedback
> > >>> >>>> > > >>> >> >>> from
> > >>> >>>> > > >>> >> >>> > > > > everyone on this. The goal would ideally
> to
> > >>> be to
> > >>> >>>> > > >>> converge on
> > >>> >>>> > > >>> >> >>> this
> > >>> >>>> > > >>> >> >>> > and
> > >>> >>>> > > >>> >> >>> > > > > eventually replace the existing Airflow
> UI
> > >>> with
> > >>> >>>> this
> > >>> >>>> > > one.
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > Cheers,
> > >>> >>>> > > >>> >> >>> > > > > Chris
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao
> <
> > >>> >>>> > > joyg@wepay.com>
> > >>> >>>> > > >>> >> wrote:
> > >>> >>>> > > >>> >> >>> > > > >
> > >>> >>>> > > >>> >> >>> > > > >> Hi guys.
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> I've been working on moving airflow from
> > >>> >>>> Flask-Admin
> > >>> >>>> > to
> > >>> >>>> > > >>> >> >>> > > Flask-AppBuilder
> > >>> >>>> > > >>> >> >>> > > > >> for RBAC
> > >>> >>>> > > >>> >> >>> > > > >> <https://cwiki.apache.org/
> > >>> >>>> > confluence/display/AIRFLOW/
> > >>> >>>> > > >>> >> >>> > > > Airflow+RBAC+proposal
> > >>> >>>> > > >>> >> >>> > > > >>> ,
> > >>> >>>> > > >>> >> >>> > > > >> check it out at
> > >>> https://github.com/wepay/airfl
> > >>> >>>> > > >>> ow-webserver.
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> It's still a work-in-progress, but most
> > >>> >>>> features you
> > >>> >>>> > > see
> > >>> >>>> > > >>> in
> > >>> >>>> > > >>> >> the
> > >>> >>>> > > >>> >> >>> > > > webserver
> > >>> >>>> > > >>> >> >>> > > > >> UI today is available there. For those
> who
> > >>> are
> > >>> >>>> > > >>> interested in
> > >>> >>>> > > >>> >> >>> RBAC,
> > >>> >>>> > > >>> >> >>> > I'd
> > >>> >>>> > > >>> >> >>> > > > love
> > >>> >>>> > > >>> >> >>> > > > >> to get some early feedback in terms of
> the
> > >>> >>>> following:
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any
> > >>> bugs/regressions)
> > >>> >>>> > > >>> >> >>> > > > >> - Setup issues
> > >>> >>>> > > >>> >> >>> > > > >> - Ease of integration with third party
> > auth
> > >>> >>>> (i.e.
> > >>> >>>> > LDAP,
> > >>> >>>> > > >>> AD,
> > >>> >>>> > > >>> >> >>> OAuth,
> > >>> >>>> > > >>> >> >>> > > etc.)
> > >>> >>>> > > >>> >> >>> > > > >> - Any other thoughts/concerns
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> Thanks a lot!
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > > >> Cheers,
> > >>> >>>> > > >>> >> >>> > > > >> Joy
> > >>> >>>> > > >>> >> >>> > > > >>
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > > >
> > >>> >>>> > > >>> >> >>> > >
> > >>> >>>> > > >>> >> >>> >
> > >>> >>>> > > >>> >> >>>
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >>
> > >>> >>>> > > >>> >> >
> > >>> >>>> > > >>> >>
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>> >
> > >>> >>>> > > >>>
> > >>> >>>> > > >>
> > >>> >>>> > > >>
> > >>> >>>> > > >
> > >>> >>>> > >
> > >>> >>>> >
> > >>> >>>>
> > >>> >>>
> > >>> >>>
> > >>> >>
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: DAG Level permissions (was Re: RBAC Update)
Posted by Ruiqin Yang <yr...@gmail.com>.
Congratulations! Extraordinary work! Thank you very much! This has been a
highly desired feature for us for quite a while.
Cheers,
Kevin Yang
Tao Feng <fe...@gmail.com>于2018年7月16日 周一下午9:30写道：
> Hi,
>
> Just want to give an update that Airflow DAG level access just checked in
> today(https://github.com/apache/incubator-airflow/pull/3197). Thanks a lot
> for Max and Joy's review which helps me improving the pr. I create the
> following three tickets as a follow up:
>
> https://issues.apache.org/jira/browse/AIRFLOW-2694 # Allow parsing access
> control in DAG file
> https://issues.apache.org/jira/browse/AIRFLOW-2693 # Implement sync_perm
> model view endpoint
> https://issues.apache.org/jira/browse/AIRFLOW-2695 # Support assign groups
> of dag permission to a role
>
> I will start working on them in Q3.
>
> Thanks a lot,
> -Tao
>
> On Sun, Apr 8, 2018 at 11:26 AM, Tao Feng <fe...@gmail.com> wrote:
>
> > Hi Max, Joy and Everyone,
> >
> > Based on the discussion, I put up a work in progress pr (
> > https://github.com/apache/incubator-airflow/pull/3197/) with a doc(
> > https://docs.google.com/document/d/1qs26lE9kAuCY0Qa0ga-8
> > 0EQ7d7m4s-590lhjtMBjmxw/edit#) for DAG level access. I would like to get
> > some early feedbacks and see if I miss anything or am in the wrong
> > direction as it touches the core part.
> >
> > In the meantime, I will still continue improving the pr for couples of
> > todos.
> >
> > Looking forward to your feedback.
> >
> > Thanks,
> > -Tao
> >
> > On Mon, Apr 2, 2018 at 2:44 PM, Tao Feng <fe...@gmail.com> wrote:
> >
> >> Hi everyone,
> >>
> >> Thanks a lot for all the great discussions. To summarize in brief, here
> >> are the few approaches we discussed so far:
> >>
> >> 1. One permission per DAG. The user has homogenous rights on the dag.
> >> The concerns:
> >>
> >> - not flexible to certain use cases(e.g the user has view only access
> >> on certain dags and write access on certain other dags );
> >> - difficult to implement as it breaks the existing FAB security
> model.
> >>
> >> 2. Extend current model(ab_permission_view_role) with an additional
> >> optional column (dag_id).
> >> The concerns:
> >>
> >> - introduce a 3rd column to existing permission_view table.
> >> - It requires us to handle db migration carefully from view only
> >> access UI to dag level access UI.
> >>
> >> 3. Define a set of pre-defined dag-level permissions. Reuse the current
> >> existing permission_view model and consider each individual dag as a new
> >> view.
> >>
> >> It seems that the 3rd approach is a preferable approach which makes the
> >> security model easy to extend without introducing additional
> complexity. If
> >> no other concern, I will work with Max to create an initial proposal /
> PR
> >> based on 3rd approach for the work(https://issues.apache.org
> >> /jira/browse/AIRFLOW-2267).
> >>
> >> Thanks,
> >> -Tao
> >>
> >> On Sat, Mar 31, 2018 at 12:09 AM, Joy Gao <jo...@wepay.com> wrote:
> >>
> >>> +1!
> >>>
> >>> I was originally tempted to re-use existing perms and views for
> dag-level
> >>> access control since dag-level perm/view is a subset of view-level
> >>> perm/view, but your proposal of defining new dag-level perms/views
> >>> independent from view-level perms/views is interesting. This actually
> >>> makes
> >>> a lot of sense, since even in the existing models, views can also be
> menu
> >>> options, so we are simply extending the concept of views to include
> dags
> >>> as
> >>> well.
> >>>
> >>> On Fri, Mar 30, 2018 at 5:24 PM Maxime Beauchemin <
> >>> maximebeauchemin@gmail.com> wrote:
> >>>
> >>> > I'd suggest something else that avoids having to add a 3rd column. I
> >>> think
> >>> > we can fit our use case into the existing structure nicely.
> >>> >
> >>> > My idea is to mimic what FAB does with its own Models.
> >>> >
> >>> > When you create a Model and ModelView in FAB (say DagRun for
> example),
> >>> it
> >>> > creates a new view_menu (DagRun) and matches it with existing
> >>> permission
> >>> > (can_read, can_delete, can_edit, can_show, ...) to create a new set
> of
> >>> > "permission_views" which are combinations of permission and view_menu
> >>> as in
> >>> > "DagRun - can_read", "DagRun - can_edit", ... It's not a cartesian
> >>> product
> >>> > of all perms and view_menus, it's a predetermined list of
> >>> model-specific
> >>> > perms that get combined with DagRun here.
> >>> >
> >>> > Similarly, when Airflow would detect a new DAG (example "my_dag"), it
> >>> > would create a new view_menu `my_dag`, and match it with permissions
> >>> that
> >>> > are identified as "combinable" with DAG. To avoid potential conflicts
> >>> I'd
> >>> > create new permissions that are DAG-specific like "dag_clear",
> >>> "dag_run",
> >>> > "dag_view".
> >>> >
> >>> > I'm arguing about the how to use the FAB model here specifically. I
> >>> think
> >>> > this allows for all the flexibility we need without changing the
> >>> model. I
> >>> > care less about what exactly the atomicity of the per DAG perms
> should
> >>> look
> >>> > like. As far as I'm concerned, per-DAG read and write is probably
> >>> granular
> >>> > enough
> >>> >
> >>> > Also note that:
> >>> > * we need an "_all_dags" logical DAG, meaning we'd have extra
> >>> > permission_views "_all_dags - dag_view", "_all_dags - dag_run",
> >>> "all_dags -
> >>> > dag_clear"
> >>> > * we probably want to derive FAB's SecurityManger and have an
> >>> > AirflowSecurityManager that has an extra method "can_dag_action(user,
> >>> > dag_id, action)". The SecurityManger class is neat because people can
> >>> > provide their own if they want and override methods. That means that
> >>> you
> >>> > can defer any of the security-related checks to another system if you
> >>> want
> >>> > to. Many companies have some sort of company RBAC system and that can
> >>> be
> >>> > used to integrate with it.
> >>> >
> >>> > Max
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Mar 30, 2018 at 4:54 PM, Joy Gao <jo...@wepay.com> wrote:
> >>> >
> >>> >> Hi all,
> >>> >>
> >>> >> I also agree that having view-only access to some dags while write
> >>> access
> >>> >> to other dags is useful, so I prefer option 2. Although option 2 is
> >>> more
> >>> >> difficult to manage, it is cleaner and more consistent with the
> >>> current
> >>> >> security model. (On the other hand, even though option 1 may be may
> be
> >>> >> easier to manage, it might be trickier to implement: with one perm
> per
> >>> >> dag, it breaks the existing FAB security model since the existing,
> >>> more
> >>> >> granular permissions will have to be grouped for each dag).
> >>> >>
> >>> >> What Brian suggested in the other thread makes sense to me...
> >>> >>
> >>> >> The current security model in FAB looks like the following:
> >>> >>
> >>> >> ---
> >>> permissions
> >>> >> user --- role --- permission_view ---|
> >>> >> --- views
> >>> >>
> >>> >> FAB generates a few relationship tables to manage the mappings, one
> of
> >>> >> them is *ab_permission_view_role*, which has all the
> >>> >> role-to-permission_view mapping. We can add a dag_id column to this
> >>> >> relationship that expresses dag-level granularity, so the security
> >>> model
> >>> >> becomes:
> >>> >>
> >>> >>
> ---
> >>> >> permissions
> >>> >> |
> >>> >> user --- role --- dag_permission_view --- --- views
> >>> >> |
> >>> >> ---
> >>> dags
> >>> >>
> >>> >> We can make the dag_id field an optional field in the relationship,
> >>> and
> >>> >> only lazily add new dag-level mappings for DAGs that specify 'access
> >>> >> control'. This way we don't have to create new permissions or views.
> >>> (We
> >>> >> will still have to introduce new dag-level roles on top of the 5
> >>> generic
> >>> >> roles (public/viewer/user/op/admin)).
> >>> >>
> >>> >> (I think this is similar to what Arthur suggested earlier, but not
> >>> sure
> >>> >> if I interpreted correctly).
> >>> >>
> >>> >> Joy
> >>> >>
> >>> >> On Thu, Mar 29, 2018 at 10:01 AM, Arthur Wiedmer <
> >>> >> arthur.wiedmer@gmail.com> wrote:
> >>> >>
> >>> >>> (Creating a new thread)
> >>> >>>
> >>> >>> Hi Max,
> >>> >>>
> >>> >>> I was just wondering about this. There are definite use cases for
> >>> people
> >>> >>> having only view access to some DAGs, mostly for monitoring. I want
> >>> to know
> >>> >>> what the upstream DAGs are doing, but maybe I don't need clear/run
> >>> access.
> >>> >>>
> >>> >>> I feel like the granular operation permissions will start to become
> >>> >>> unwieldy fast. It grows as (# users * # DAGs * # operations) but
> with
> >>> >>> hopefully a small constant factor in front: most people should only
> >>> have a
> >>> >>> small number of DAGs they care about. The Ops team has a need to
> have
> >>> >>> access to all on the other hand.
> >>> >>>
> >>> >>> I was wondering we could get by with something slightly more
> complex
> >>> but
> >>> >>> easier on the size of that permissions table :
> >>> >>> 1) One toggle on the user level for broad access (ALL:ALL,
> >>> >>> RUN/CLEAR:ALL, VIEW:ALL) default NULL
> >>> >>> 2) More granular permissions at the DAG level.
> >>> >>>
> >>> >>> So in order, check the user's broad level permission first, then
> DAG
> >>> >>> level. For large amounts of DAGs, this should help shave a little
> >>> bit from
> >>> >>> that table.
> >>> >>>
> >>> >>> Best,
> >>> >>> Arthur
> >>> >>>
> >>> >>>
> >>> >>> On Thu, Mar 29, 2018 at 8:27 AM, Maxime Beauchemin <
> >>> >>> maximebeauchemin@gmail.com> wrote:
> >>> >>>
> >>> >>>> Hijacking the thread further here, any thoughts on how to
> breakdown
> >>> per
> >>> >>>> DAG
> >>> >>>> access?
> >>> >>>>
> >>> >>>> Tao & I are talking about introducing per-DAG permissions and one
> >>> big
> >>> >>>> question is whether we'll need to support different
> operation-types
> >>> at a
> >>> >>>> per-DAG level, which changes the way we need to model the perms.
> >>> >>>>
> >>> >>>> First [simpler] option is to introduce one perm per DAG. If you
> have
> >>> >>>> access
> >>> >>>> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have
> >>> >>>> homogenous
> >>> >>>> rights on the DAGs you have access to.
> >>> >>>>
> >>> >>>> Second option is to have a breakdown per DAG. Meaning for each DAG
> >>> we
> >>> >>>> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify,
> >>> ...). So
> >>> >>>> one
> >>> >>>> user could have modify on some DAGs, view on others, and other
> DAGs
> >>> >>>> would
> >>> >>>> be invisible. This could be broken down further
> ({dag_id}_can_clear,
> >>> >>>> ...)
> >>> >>>> but it gets hard to manage.
> >>> >>>>
> >>> >>>> Thoughts?
> >>> >>>>
> >>> >>>> Max
> >>> >>>>
> >>> >>>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fe...@gmail.com>
> >>> wrote:
> >>> >>>>
> >>> >>>> > Great work Joy. This is awesome! I am interested in helping out
> >>> the
> >>> >>>> per dag
> >>> >>>> > level access. Just created a ticket to check(AIRFLOW-2267). Let
> >>> me
> >>> >>>> know if
> >>> >>>> > you have any suggestions. I will share my proposal once I am
> >>> ready.
> >>> >>>> >
> >>> >>>> > On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com>
> wrote:
> >>> >>>> >
> >>> >>>> > > Hey guys!
> >>> >>>> > >
> >>> >>>> > > The RBAC UI <https://github.com/apache/inc
> >>> ubator-airflow/pull/3015>
> >>> >>>> has
> >>> >>>> > > been merged to master. I'm looking forward to early adopters'
> >>> >>>> feedback
> >>> >>>> > and
> >>> >>>> > > bug reports. I also hope to have more folks helping out with
> the
> >>> >>>> RBAC UI,
> >>> >>>> > > especially with introducing DAG-Level access control, which
> is a
> >>> >>>> feature
> >>> >>>> > > that a lot of people have been asking. If you are interested
> in
> >>> >>>> helping
> >>> >>>> > out
> >>> >>>> > > with this effort, let's talk more!
> >>> >>>> > >
> >>> >>>> > > This commit will be in the 1.10.0 release, and we are going to
> >>> >>>> maintain
> >>> >>>> > > both UIs simultaneously for a short period of time. Once RBAC
> >>> UI is
> >>> >>>> > stable
> >>> >>>> > > and battle-tested, we will deprecate the old UI and eventually
> >>> >>>> remove it
> >>> >>>> > > from the repo (around Airflow 2.0.0 or 2.1.0 release). This is
> >>> to
> >>> >>>> prevent
> >>> >>>> > > two UIs from forking into separate paths, as that would become
> >>> very
> >>> >>>> > > difficult to maintain.
> >>> >>>> > >
> >>> >>>> > > Going forward while both UIs are up, if you are making a
> change
> >>> to
> >>> >>>> any
> >>> >>>> > > files in airflow/www/ (old UI), where applicable, please also
> >>> make
> >>> >>>> the
> >>> >>>> > > change to the airflow/www_rbac/ (new UI). If you rather not
> make
> >>> >>>> changes
> >>> >>>> > in
> >>> >>>> > > both UIs, it is recommended that you only make the changes to
> >>> the
> >>> >>>> RBAC
> >>> >>>> > UI,
> >>> >>>> > > since that is the one we are maintaining in the long term.
> >>> >>>> > >
> >>> >>>> > > I'm excited that the RBAC UI will be able to bring additional
> >>> >>>> security to
> >>> >>>> > > Airflow, and with FAB framework in place we can look into
> >>> >>>> leveraging it
> >>> >>>> > for
> >>> >>>> > > a unified set of APIs used by both UI and CLI.
> >>> >>>> > >
> >>> >>>> > > Joy
> >>> >>>> > >
> >>> >>>> > >
> >>> >>>> > >
> >>> >>>> > > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com>
> >>> wrote:
> >>> >>>> > >
> >>> >>>> > > > Hi folks,
> >>> >>>> > > >
> >>> >>>> > > > I have a PR <https://github.com/apache/
> >>> >>>> incubator-airflow/pull/3015>
> >>> >>>> > out
> >>> >>>> > > > for the new UI. I've included instructions on how to test it
> >>> out
> >>> >>>> in the
> >>> >>>> > > PR
> >>> >>>> > > > description. Looking forward to your feedbacks.
> >>> >>>> > > >
> >>> >>>> > > > Cheers,
> >>> >>>> > > > Joy
> >>> >>>> > > >
> >>> >>>> > > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com>
> >>> wrote:
> >>> >>>> > > >
> >>> >>>> > > >> Thanks for the background info. Would be really awesome for
> >>> you
> >>> >>>> to
> >>> >>>> > have
> >>> >>>> > > >> PyPi access :D I'll make the change to have Airflow
> >>> Webserver's
> >>> >>>> FAB
> >>> >>>> > > >> dependency pointing to my fork for the mean time.
> >>> >>>> > > >>
> >>> >>>> > > >> For folks who are interested in RBAC, I will be giving a
> >>> >>>> talk/demo at
> >>> >>>> > > the Airflow
> >>> >>>> > > >> Meet-Up
> >>> >>>> > > >> <
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> >>> >>>> > > Meetup/events/244525050/>
> >>> >>>> > > >> next Monday. Happy to chat afterwards about it as well :)
> >>> >>>> > > >>
> >>> >>>> > > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> >>> >>>> > > >> maximebeauchemin@gmail.com> wrote:
> >>> >>>> > > >>
> >>> >>>> > > >>> A bit of related history here:
> >>> >>>> > > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> >>> >>>> > > >>>
> >>> >>>> > > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> >>> >>>> > > >>> maximebeauchemin@gmail.com> wrote:
> >>> >>>> > > >>>
> >>> >>>> > > >>> > Given I have merge rights on FAB I could probably do
> >>> another
> >>> >>>> round
> >>> >>>> > of
> >>> >>>> > > >>> > review and get your PRs through. I would really like to
> >>> get
> >>> >>>> the
> >>> >>>> > main
> >>> >>>> > > >>> > maintainer's input on things that touch the core
> >>> >>>> (composite-key
> >>> >>>> > > >>> support) as
> >>> >>>> > > >>> > he might have concerns/intuitions that we can't know
> >>> about.
> >>> >>>> > > >>> >
> >>> >>>> > > >>> > I do not have Pypi access though so I cannot push new
> >>> releases
> >>> >>>> > out. I
> >>> >>>> > > >>> > could ask for that.
> >>> >>>> > > >>> >
> >>> >>>> > > >>> > I've threatened to fork the project before, that's
> always
> >>> an
> >>> >>>> > option.
> >>> >>>> > > >>> I've
> >>> >>>> > > >>> > noticed his involvement is sporadic and comes in bursts.
> >>> >>>> > > >>> >
> >>> >>>> > > >>> > In the meantime, you can have the dependency in Airflow
> >>> >>>> Webserver
> >>> >>>> > > >>> pointing
> >>> >>>> > > >>> > straight to your fork.
> >>> >>>> > > >>> >
> >>> >>>> > > >>> > Max
> >>> >>>> > > >>> >
> >>> >>>> > > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <
> joyg@wepay.com>
> >>> >>>> wrote:
> >>> >>>> > > >>> >
> >>> >>>> > > >>> >> I just created a new webserver instance if you haven't
> >>> >>>> gotten a
> >>> >>>> > > >>> chance to
> >>> >>>> > > >>> >> fiddle around with the new web UI and the RBAC
> >>> configurations
> >>> >>>> > > (thanks
> >>> >>>> > > >>> >> Maxime for getting started with this earlier!):
> >>> >>>> > > >>> >>
> >>> >>>> > > >>> >> http://104.209.38.171:8080/
> >>> >>>> > > >>> >>
> >>> >>>> > > >>> >> Admin Account
> >>> >>>> > > >>> >> username: admin
> >>> >>>> > > >>> >> password: admin
> >>> >>>> > > >>> >>
> >>> >>>> > > >>> >> Read-Only Account
> >>> >>>> > > >>> >> username: viewer
> >>> >>>> > > >>> >> password: password
> >>> >>>> > > >>> >>
> >>> >>>> > > >>> >>
> >>> >>>> > > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <
> joyg@wepay.com
> >>> >
> >>> >>>> wrote:
> >>> >>>> > > >>> >>
> >>> >>>> > > >>> >> > Hi folks,
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > Thanks for all the feedback regarding to the new
> >>> Airflow
> >>> >>>> > Webserver
> >>> >>>> > > >>> UI
> >>> >>>> > > >>> >> > <https://github.com/wepay/airflow-webserver/>! I've
> >>> been
> >>> >>>> > actively
> >>> >>>> > > >>> >> > addressing all the bugs that were raised on Github.
> So
> >>> I
> >>> >>>> want to
> >>> >>>> > > >>> take
> >>> >>>> > > >>> >> this
> >>> >>>> > > >>> >> > opportunity to discuss two issues coming up:
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > The first issue is unaddressed PRs in FAB. If these
> PRs
> >>> >>>> continue
> >>> >>>> > > to
> >>> >>>> > > >>> stay
> >>> >>>> > > >>> >> > unaddressed, RBAC is blocked from making further
> >>> progress.
> >>> >>>> If
> >>> >>>> > this
> >>> >>>> > > >>> >> continue
> >>> >>>> > > >>> >> > to be an issue, I'm inclined to fork FAB, even though
> >>> it's
> >>> >>>> not
> >>> >>>> > > >>> >> idealistic.
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > - PR/631 <https://github.com/dpgaspar/F
> >>> >>>> > > lask-AppBuilder/pull/631>
> >>> >>>> > > >>> >> Binary
> >>> >>>> > > >>> >> > column support (merged, unreleased)
> >>> >>>> > > >>> >> > <https://github.com/dpgaspar/F
> >>> lask-AppBuilder/pull/631>
> >>> >>>> > > >>> >> > - PR/639 <https://github.com/dpgaspar/F
> >>> >>>> > > lask-AppBuilder/pull/639>
> >>> >>>> > > >>> >> Composite
> >>> >>>> > > >>> >> > primary key support (unmerged)
> >>> >>>> > > >>> >> > - PR/655 <https://github.com/dpgaspar/F
> >>> >>>> > > lask-AppBuilder/pull/655>
> >>> >>>> > > >>> >> Form
> >>> >>>> > > >>> >> > prefill support (unmerged)
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > The second issue is an open question about the next
> >>> step of
> >>> >>>> > > Airflow
> >>> >>>> > > >>> >> > Webserver itself. Here are the 3 potential directions
> >>> we
> >>> >>>> could
> >>> >>>> > > >>> take, and
> >>> >>>> > > >>> >> > I've added my thought on each.
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > 1. Permanently keep Airflow Webserver as a separated
> >>> >>>> package
> >>> >>>> > from
> >>> >>>> > > >>> >> Airflow,
> >>> >>>> > > >>> >> > and treat it as another UI option. Keep `www` in
> >>> Airflow.
> >>> >>>> Allow
> >>> >>>> > > >>> >> development
> >>> >>>> > > >>> >> > on both UIs.
> >>> >>>> > > >>> >> > *I'm not a fan of this. When there is an existing UI
> in
> >>> >>>> Airflow,
> >>> >>>> > > >>> most
> >>> >>>> > > >>> >> > contributors would prefer to maintain the official
> >>> version
> >>> >>>> that
> >>> >>>> > is
> >>> >>>> > > >>> >> > installed out-of-the-box. **Having a second UI
> outside
> >>> of
> >>> >>>> > Airflow
> >>> >>>> > > >>> will
> >>> >>>> > > >>> >> > make maintaining it very difficult, leading to an
> >>> eventual
> >>> >>>> death
> >>> >>>> > > of
> >>> >>>> > > >>> the
> >>> >>>> > > >>> >> new
> >>> >>>> > > >>> >> > UI :(*
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > 2. Permanently keep Airflow Webserver as a separated
> >>> >>>> package
> >>> >>>> > from
> >>> >>>> > > >>> >> Airflow,
> >>> >>>> > > >>> >> > but freeze all development on `www` and direct all
> >>> future
> >>> >>>> UI
> >>> >>>> > > >>> >> development
> >>> >>>> > > >>> >> > to Airflow Webserver, eventually removing `www`
> >>> completely
> >>> >>>> when
> >>> >>>> > > >>> Airflow
> >>> >>>> > > >>> >> > Webserver is stable.
> >>> >>>> > > >>> >> > *I'm not a fan of this either. First of all, the
> views
> >>> and
> >>> >>>> > models
> >>> >>>> > > >>> are
> >>> >>>> > > >>> >> > tightly coupled in both old and new UI; until we
> have a
> >>> >>>> > > full-fledged
> >>> >>>> > > >>> >> REST
> >>> >>>> > > >>> >> > API to build the UI (and cli) on top of it,
> separating
> >>> >>>> them to a
> >>> >>>> > > >>> >> separate
> >>> >>>> > > >>> >> > package now will potentially cause dependency issues
> >>> and
> >>> >>>> add
> >>> >>>> > > >>> >> complication
> >>> >>>> > > >>> >> > to our release cycle. **Secondly, **majority of
> Airflow
> >>> >>>> users
> >>> >>>> > run
> >>> >>>> > > >>> >> Airflow
> >>> >>>> > > >>> >> > with the UI; it's one of Airflow's best features.
> >>> >>>> Separating UI
> >>> >>>> > > out
> >>> >>>> > > >>> of
> >>> >>>> > > >>> >> > Airflow core will complicate setup and configuration,
> >>> while
> >>> >>>> > making
> >>> >>>> > > >>> >> Airflow
> >>> >>>> > > >>> >> > core less complete.*
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > 3. Merge Airflow Webserver back into Airflow as
> `www2`,
> >>> >>>> freeze
> >>> >>>> > all
> >>> >>>> > > >>> >> > development on `www`, eventually removing `www`
> >>> completely
> >>> >>>> when
> >>> >>>> > > >>> `www2`
> >>> >>>> > > >>> >> is
> >>> >>>> > > >>> >> > stable.
> >>> >>>> > > >>> >> > *This makes the most sense to me. Airflow Webserver
> is
> >>> >>>> developed
> >>> >>>> > > >>> with
> >>> >>>> > > >>> >> the
> >>> >>>> > > >>> >> > goal of feature parity to the current UI, plus
> >>> additional
> >>> >>>> RBAC
> >>> >>>> > > >>> >> capability,
> >>> >>>> > > >>> >> > in hope to replace the old UI completely. Yes, this
> >>> means
> >>> >>>> there
> >>> >>>> > > >>> will be
> >>> >>>> > > >>> >> a
> >>> >>>> > > >>> >> > short period of having to maintain two UIs, but once
> we
> >>> >>>> freeze
> >>> >>>> > > >>> >> development
> >>> >>>> > > >>> >> > on www, it shouldn't be a concern for long.*
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > I'd love to hear everyone's thoughts on this! I'm
> >>> excited
> >>> >>>> about
> >>> >>>> > > >>> bringing
> >>> >>>> > > >>> >> > RBAC to airflow and I hope it's something others will
> >>> find
> >>> >>>> > useful
> >>> >>>> > > as
> >>> >>>> > > >>> >> well!
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > Cheers,
> >>> >>>> > > >>> >> > Joy
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <
> >>> joyg@wepay.com>
> >>> >>>> > wrote:
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >> >> Thank you everyone for the active feedback so far,
> and
> >>> >>>> thanks
> >>> >>>> > for
> >>> >>>> > > >>> >> setting
> >>> >>>> > > >>> >> >> up the demo Maxime!
> >>> >>>> > > >>> >> >>
> >>> >>>> > > >>> >> >> Going to work on pruning through the issues in the
> >>> >>>> upcoming
> >>> >>>> > days.
> >>> >>>> > > >>> >> >>
> >>> >>>> > > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception
> >>> >>>> message
> >>> >>>> > so I
> >>> >>>> > > >>> can
> >>> >>>> > > >>> >> >> look into it? Otherwise I'll wait until it's down
> >>> again =P
> >>> >>>> > > >>> >> >>
> >>> >>>> > > >>> >> >> Cheers,
> >>> >>>> > > >>> >> >>
> >>> >>>> > > >>> >> >> Joy
> >>> >>>> > > >>> >> >>
> >>> >>>> > > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >>> >>>> > > >>> >> >> maximebeauchemin@gmail.com> wrote:
> >>> >>>> > > >>> >> >>
> >>> >>>> > > >>> >> >>> I just restarted it, not sure how long it will take
> >>> to
> >>> >>>> get in
> >>> >>>> > a
> >>> >>>> > > >>> bad
> >>> >>>> > > >>> >> state
> >>> >>>> > > >>> >> >>> again...
> >>> >>>> > > >>> >> >>>
> >>> >>>> > > >>> >> >>> Max
> >>> >>>> > > >>> >> >>>
> >>> >>>> > > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> >>> >>>> > > >>> >> <fokko@driesprong.frl
> >>> >>>> > > >>> >> >>> >
> >>> >>>> > > >>> >> >>> wrote:
> >>> >>>> > > >>> >> >>>
> >>> >>>> > > >>> >> >>> > Good morning,
> >>> >>>> > > >>> >> >>> >
> >>> >>>> > > >>> >> >>> > The demo provided by Max is down, it throws a
> >>> >>>> > > >>> SQLAlchemyexception
> >>> >>>> > > >>> >> :'(
> >>> >>>> > > >>> >> >>> >
> >>> >>>> > > >>> >> >>> > Cheers, Fokko
> >>> >>>> > > >>> >> >>> >
> >>> >>>> > > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> >>> >>>> > > >>> criccomini@apache.org>:
> >>> >>>> > > >>> >> >>> >
> >>> >>>> > > >>> >> >>> > > @bolke, open issues on the Github repo, please.
> >>> >>>> > > >>> >> >>> > >
> >>> >>>> > > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de
> Bruin
> >>> <
> >>> >>>> > > >>> >> bdbruin@gmail.com>
> >>> >>>> > > >>> >> >>> > > wrote:
> >>> >>>> > > >>> >> >>> > >
> >>> >>>> > > >>> >> >>> > > > Chris,
> >>> >>>> > > >>> >> >>> > > >
> >>> >>>> > > >>> >> >>> > > > Do you want us to report bugs somewhere (I
> have
> >>> >>>> > > encountered
> >>> >>>> > > >>> a
> >>> >>>> > > >>> >> >>> few)? Or
> >>> >>>> > > >>> >> >>> > > > just generic user experiences posted here?
> >>> >>>> > > >>> >> >>> > > >
> >>> >>>> > > >>> >> >>> > > > Cheers
> >>> >>>> > > >>> >> >>> > > > Bolke
> >>> >>>> > > >>> >> >>> > > >
> >>> >>>> > > >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> >>> >>>> > > >>> >> criccomini@apache.org
> >>> >>>> > > >>> >> >>> >
> >>> >>>> > > >>> >> >>> > > wrote:
> >>> >>>> > > >>> >> >>> > > > >
> >>> >>>> > > >>> >> >>> > > > > Hey all,
> >>> >>>> > > >>> >> >>> > > > >
> >>> >>>> > > >>> >> >>> > > > > I know the weekend is coming up, and for
> >>> those
> >>> >>>> of us
> >>> >>>> > in
> >>> >>>> > > >>> the
> >>> >>>> > > >>> >> US,
> >>> >>>> > > >>> >> >>> next
> >>> >>>> > > >>> >> >>> > > week
> >>> >>>> > > >>> >> >>> > > > > is a bit of a slow holiday week. Would love
> >>> to
> >>> >>>> get
> >>> >>>> > some
> >>> >>>> > > >>> >> feedback
> >>> >>>> > > >>> >> >>> from
> >>> >>>> > > >>> >> >>> > > > > everyone on this. The goal would ideally to
> >>> be to
> >>> >>>> > > >>> converge on
> >>> >>>> > > >>> >> >>> this
> >>> >>>> > > >>> >> >>> > and
> >>> >>>> > > >>> >> >>> > > > > eventually replace the existing Airflow UI
> >>> with
> >>> >>>> this
> >>> >>>> > > one.
> >>> >>>> > > >>> >> >>> > > > >
> >>> >>>> > > >>> >> >>> > > > > Cheers,
> >>> >>>> > > >>> >> >>> > > > > Chris
> >>> >>>> > > >>> >> >>> > > > >
> >>> >>>> > > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> >>> >>>> > > joyg@wepay.com>
> >>> >>>> > > >>> >> wrote:
> >>> >>>> > > >>> >> >>> > > > >
> >>> >>>> > > >>> >> >>> > > > >> Hi guys.
> >>> >>>> > > >>> >> >>> > > > >>
> >>> >>>> > > >>> >> >>> > > > >> I've been working on moving airflow from
> >>> >>>> Flask-Admin
> >>> >>>> > to
> >>> >>>> > > >>> >> >>> > > Flask-AppBuilder
> >>> >>>> > > >>> >> >>> > > > >> for RBAC
> >>> >>>> > > >>> >> >>> > > > >> <https://cwiki.apache.org/
> >>> >>>> > confluence/display/AIRFLOW/
> >>> >>>> > > >>> >> >>> > > > Airflow+RBAC+proposal
> >>> >>>> > > >>> >> >>> > > > >>> ,
> >>> >>>> > > >>> >> >>> > > > >> check it out at
> >>> https://github.com/wepay/airfl
> >>> >>>> > > >>> ow-webserver.
> >>> >>>> > > >>> >> >>> > > > >>
> >>> >>>> > > >>> >> >>> > > > >> It's still a work-in-progress, but most
> >>> >>>> features you
> >>> >>>> > > see
> >>> >>>> > > >>> in
> >>> >>>> > > >>> >> the
> >>> >>>> > > >>> >> >>> > > > webserver
> >>> >>>> > > >>> >> >>> > > > >> UI today is available there. For those who
> >>> are
> >>> >>>> > > >>> interested in
> >>> >>>> > > >>> >> >>> RBAC,
> >>> >>>> > > >>> >> >>> > I'd
> >>> >>>> > > >>> >> >>> > > > love
> >>> >>>> > > >>> >> >>> > > > >> to get some early feedback in terms of the
> >>> >>>> following:
> >>> >>>> > > >>> >> >>> > > > >>
> >>> >>>> > > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any
> >>> bugs/regressions)
> >>> >>>> > > >>> >> >>> > > > >> - Setup issues
> >>> >>>> > > >>> >> >>> > > > >> - Ease of integration with third party
> auth
> >>> >>>> (i.e.
> >>> >>>> > LDAP,
> >>> >>>> > > >>> AD,
> >>> >>>> > > >>> >> >>> OAuth,
> >>> >>>> > > >>> >> >>> > > etc.)
> >>> >>>> > > >>> >> >>> > > > >> - Any other thoughts/concerns
> >>> >>>> > > >>> >> >>> > > > >>
> >>> >>>> > > >>> >> >>> > > > >> Thanks a lot!
> >>> >>>> > > >>> >> >>> > > > >>
> >>> >>>> > > >>> >> >>> > > > >> Cheers,
> >>> >>>> > > >>> >> >>> > > > >> Joy
> >>> >>>> > > >>> >> >>> > > > >>
> >>> >>>> > > >>> >> >>> > > >
> >>> >>>> > > >>> >> >>> > > >
> >>> >>>> > > >>> >> >>> > >
> >>> >>>> > > >>> >> >>> >
> >>> >>>> > > >>> >> >>>
> >>> >>>> > > >>> >> >>
> >>> >>>> > > >>> >> >>
> >>> >>>> > > >>> >> >
> >>> >>>> > > >>> >>
> >>> >>>> > > >>> >
> >>> >>>> > > >>> >
> >>> >>>> > > >>>
> >>> >>>> > > >>
> >>> >>>> > > >>
> >>> >>>> > > >
> >>> >>>> > >
> >>> >>>> >
> >>> >>>>
> >>> >>>
> >>> >>>
> >>> >>
> >>> >
> >>>
> >>
> >>
> >
>
Re: DAG Level permissions (was Re: RBAC Update)
Posted by Tao Feng <fe...@gmail.com>.
Hi,
Just want to give an update that Airflow DAG level access just checked in
today(https://github.com/apache/incubator-airflow/pull/3197). Thanks a lot
for Max and Joy's review which helps me improving the pr. I create the
following three tickets as a follow up:
https://issues.apache.org/jira/browse/AIRFLOW-2694 # Allow parsing access
control in DAG file
https://issues.apache.org/jira/browse/AIRFLOW-2693 # Implement sync_perm
model view endpoint
https://issues.apache.org/jira/browse/AIRFLOW-2695 # Support assign groups
of dag permission to a role
I will start working on them in Q3.
Thanks a lot,
-Tao
On Sun, Apr 8, 2018 at 11:26 AM, Tao Feng <fe...@gmail.com> wrote:
> Hi Max, Joy and Everyone,
>
> Based on the discussion, I put up a work in progress pr (
> https://github.com/apache/incubator-airflow/pull/3197/) with a doc(
> https://docs.google.com/document/d/1qs26lE9kAuCY0Qa0ga-8
> 0EQ7d7m4s-590lhjtMBjmxw/edit#) for DAG level access. I would like to get
> some early feedbacks and see if I miss anything or am in the wrong
> direction as it touches the core part.
>
> In the meantime, I will still continue improving the pr for couples of
> todos.
>
> Looking forward to your feedback.
>
> Thanks,
> -Tao
>
> On Mon, Apr 2, 2018 at 2:44 PM, Tao Feng <fe...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Thanks a lot for all the great discussions. To summarize in brief, here
>> are the few approaches we discussed so far:
>>
>> 1. One permission per DAG. The user has homogenous rights on the dag.
>> The concerns:
>>
>> - not flexible to certain use cases(e.g the user has view only access
>> on certain dags and write access on certain other dags );
>> - difficult to implement as it breaks the existing FAB security model.
>>
>> 2. Extend current model(ab_permission_view_role) with an additional
>> optional column (dag_id).
>> The concerns:
>>
>> - introduce a 3rd column to existing permission_view table.
>> - It requires us to handle db migration carefully from view only
>> access UI to dag level access UI.
>>
>> 3. Define a set of pre-defined dag-level permissions. Reuse the current
>> existing permission_view model and consider each individual dag as a new
>> view.
>>
>> It seems that the 3rd approach is a preferable approach which makes the
>> security model easy to extend without introducing additional complexity. If
>> no other concern, I will work with Max to create an initial proposal / PR
>> based on 3rd approach for the work(https://issues.apache.org
>> /jira/browse/AIRFLOW-2267).
>>
>> Thanks,
>> -Tao
>>
>> On Sat, Mar 31, 2018 at 12:09 AM, Joy Gao <jo...@wepay.com> wrote:
>>
>>> +1!
>>>
>>> I was originally tempted to re-use existing perms and views for dag-level
>>> access control since dag-level perm/view is a subset of view-level
>>> perm/view, but your proposal of defining new dag-level perms/views
>>> independent from view-level perms/views is interesting. This actually
>>> makes
>>> a lot of sense, since even in the existing models, views can also be menu
>>> options, so we are simply extending the concept of views to include dags
>>> as
>>> well.
>>>
>>> On Fri, Mar 30, 2018 at 5:24 PM Maxime Beauchemin <
>>> maximebeauchemin@gmail.com> wrote:
>>>
>>> > I'd suggest something else that avoids having to add a 3rd column. I
>>> think
>>> > we can fit our use case into the existing structure nicely.
>>> >
>>> > My idea is to mimic what FAB does with its own Models.
>>> >
>>> > When you create a Model and ModelView in FAB (say DagRun for example),
>>> it
>>> > creates a new view_menu (DagRun) and matches it with existing
>>> permission
>>> > (can_read, can_delete, can_edit, can_show, ...) to create a new set of
>>> > "permission_views" which are combinations of permission and view_menu
>>> as in
>>> > "DagRun - can_read", "DagRun - can_edit", ... It's not a cartesian
>>> product
>>> > of all perms and view_menus, it's a predetermined list of
>>> model-specific
>>> > perms that get combined with DagRun here.
>>> >
>>> > Similarly, when Airflow would detect a new DAG (example "my_dag"), it
>>> > would create a new view_menu `my_dag`, and match it with permissions
>>> that
>>> > are identified as "combinable" with DAG. To avoid potential conflicts
>>> I'd
>>> > create new permissions that are DAG-specific like "dag_clear",
>>> "dag_run",
>>> > "dag_view".
>>> >
>>> > I'm arguing about the how to use the FAB model here specifically. I
>>> think
>>> > this allows for all the flexibility we need without changing the
>>> model. I
>>> > care less about what exactly the atomicity of the per DAG perms should
>>> look
>>> > like. As far as I'm concerned, per-DAG read and write is probably
>>> granular
>>> > enough
>>> >
>>> > Also note that:
>>> > * we need an "_all_dags" logical DAG, meaning we'd have extra
>>> > permission_views "_all_dags - dag_view", "_all_dags - dag_run",
>>> "all_dags -
>>> > dag_clear"
>>> > * we probably want to derive FAB's SecurityManger and have an
>>> > AirflowSecurityManager that has an extra method "can_dag_action(user,
>>> > dag_id, action)". The SecurityManger class is neat because people can
>>> > provide their own if they want and override methods. That means that
>>> you
>>> > can defer any of the security-related checks to another system if you
>>> want
>>> > to. Many companies have some sort of company RBAC system and that can
>>> be
>>> > used to integrate with it.
>>> >
>>> > Max
>>> >
>>> >
>>> >
>>> > On Fri, Mar 30, 2018 at 4:54 PM, Joy Gao <jo...@wepay.com> wrote:
>>> >
>>> >> Hi all,
>>> >>
>>> >> I also agree that having view-only access to some dags while write
>>> access
>>> >> to other dags is useful, so I prefer option 2. Although option 2 is
>>> more
>>> >> difficult to manage, it is cleaner and more consistent with the
>>> current
>>> >> security model. (On the other hand, even though option 1 may be may be
>>> >> easier to manage, it might be trickier to implement: with one perm per
>>> >> dag, it breaks the existing FAB security model since the existing,
>>> more
>>> >> granular permissions will have to be grouped for each dag).
>>> >>
>>> >> What Brian suggested in the other thread makes sense to me...
>>> >>
>>> >> The current security model in FAB looks like the following:
>>> >>
>>> >> ---
>>> permissions
>>> >> user --- role --- permission_view ---|
>>> >> --- views
>>> >>
>>> >> FAB generates a few relationship tables to manage the mappings, one of
>>> >> them is *ab_permission_view_role*, which has all the
>>> >> role-to-permission_view mapping. We can add a dag_id column to this
>>> >> relationship that expresses dag-level granularity, so the security
>>> model
>>> >> becomes:
>>> >>
>>> >> ---
>>> >> permissions
>>> >> |
>>> >> user --- role --- dag_permission_view --- --- views
>>> >> |
>>> >> ---
>>> dags
>>> >>
>>> >> We can make the dag_id field an optional field in the relationship,
>>> and
>>> >> only lazily add new dag-level mappings for DAGs that specify 'access
>>> >> control'. This way we don't have to create new permissions or views.
>>> (We
>>> >> will still have to introduce new dag-level roles on top of the 5
>>> generic
>>> >> roles (public/viewer/user/op/admin)).
>>> >>
>>> >> (I think this is similar to what Arthur suggested earlier, but not
>>> sure
>>> >> if I interpreted correctly).
>>> >>
>>> >> Joy
>>> >>
>>> >> On Thu, Mar 29, 2018 at 10:01 AM, Arthur Wiedmer <
>>> >> arthur.wiedmer@gmail.com> wrote:
>>> >>
>>> >>> (Creating a new thread)
>>> >>>
>>> >>> Hi Max,
>>> >>>
>>> >>> I was just wondering about this. There are definite use cases for
>>> people
>>> >>> having only view access to some DAGs, mostly for monitoring. I want
>>> to know
>>> >>> what the upstream DAGs are doing, but maybe I don't need clear/run
>>> access.
>>> >>>
>>> >>> I feel like the granular operation permissions will start to become
>>> >>> unwieldy fast. It grows as (# users * # DAGs * # operations) but with
>>> >>> hopefully a small constant factor in front: most people should only
>>> have a
>>> >>> small number of DAGs they care about. The Ops team has a need to have
>>> >>> access to all on the other hand.
>>> >>>
>>> >>> I was wondering we could get by with something slightly more complex
>>> but
>>> >>> easier on the size of that permissions table :
>>> >>> 1) One toggle on the user level for broad access (ALL:ALL,
>>> >>> RUN/CLEAR:ALL, VIEW:ALL) default NULL
>>> >>> 2) More granular permissions at the DAG level.
>>> >>>
>>> >>> So in order, check the user's broad level permission first, then DAG
>>> >>> level. For large amounts of DAGs, this should help shave a little
>>> bit from
>>> >>> that table.
>>> >>>
>>> >>> Best,
>>> >>> Arthur
>>> >>>
>>> >>>
>>> >>> On Thu, Mar 29, 2018 at 8:27 AM, Maxime Beauchemin <
>>> >>> maximebeauchemin@gmail.com> wrote:
>>> >>>
>>> >>>> Hijacking the thread further here, any thoughts on how to breakdown
>>> per
>>> >>>> DAG
>>> >>>> access?
>>> >>>>
>>> >>>> Tao & I are talking about introducing per-DAG permissions and one
>>> big
>>> >>>> question is whether we'll need to support different operation-types
>>> at a
>>> >>>> per-DAG level, which changes the way we need to model the perms.
>>> >>>>
>>> >>>> First [simpler] option is to introduce one perm per DAG. If you have
>>> >>>> access
>>> >>>> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have
>>> >>>> homogenous
>>> >>>> rights on the DAGs you have access to.
>>> >>>>
>>> >>>> Second option is to have a breakdown per DAG. Meaning for each DAG
>>> we
>>> >>>> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify,
>>> ...). So
>>> >>>> one
>>> >>>> user could have modify on some DAGs, view on others, and other DAGs
>>> >>>> would
>>> >>>> be invisible. This could be broken down further ({dag_id}_can_clear,
>>> >>>> ...)
>>> >>>> but it gets hard to manage.
>>> >>>>
>>> >>>> Thoughts?
>>> >>>>
>>> >>>> Max
>>> >>>>
>>> >>>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fe...@gmail.com>
>>> wrote:
>>> >>>>
>>> >>>> > Great work Joy. This is awesome! I am interested in helping out
>>> the
>>> >>>> per dag
>>> >>>> > level access. Just created a ticket to check(AIRFLOW-2267). Let
>>> me
>>> >>>> know if
>>> >>>> > you have any suggestions. I will share my proposal once I am
>>> ready.
>>> >>>> >
>>> >>>> > On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
>>> >>>> >
>>> >>>> > > Hey guys!
>>> >>>> > >
>>> >>>> > > The RBAC UI <https://github.com/apache/inc
>>> ubator-airflow/pull/3015>
>>> >>>> has
>>> >>>> > > been merged to master. I'm looking forward to early adopters'
>>> >>>> feedback
>>> >>>> > and
>>> >>>> > > bug reports. I also hope to have more folks helping out with the
>>> >>>> RBAC UI,
>>> >>>> > > especially with introducing DAG-Level access control, which is a
>>> >>>> feature
>>> >>>> > > that a lot of people have been asking. If you are interested in
>>> >>>> helping
>>> >>>> > out
>>> >>>> > > with this effort, let's talk more!
>>> >>>> > >
>>> >>>> > > This commit will be in the 1.10.0 release, and we are going to
>>> >>>> maintain
>>> >>>> > > both UIs simultaneously for a short period of time. Once RBAC
>>> UI is
>>> >>>> > stable
>>> >>>> > > and battle-tested, we will deprecate the old UI and eventually
>>> >>>> remove it
>>> >>>> > > from the repo (around Airflow 2.0.0 or 2.1.0 release). This is
>>> to
>>> >>>> prevent
>>> >>>> > > two UIs from forking into separate paths, as that would become
>>> very
>>> >>>> > > difficult to maintain.
>>> >>>> > >
>>> >>>> > > Going forward while both UIs are up, if you are making a change
>>> to
>>> >>>> any
>>> >>>> > > files in airflow/www/ (old UI), where applicable, please also
>>> make
>>> >>>> the
>>> >>>> > > change to the airflow/www_rbac/ (new UI). If you rather not make
>>> >>>> changes
>>> >>>> > in
>>> >>>> > > both UIs, it is recommended that you only make the changes to
>>> the
>>> >>>> RBAC
>>> >>>> > UI,
>>> >>>> > > since that is the one we are maintaining in the long term.
>>> >>>> > >
>>> >>>> > > I'm excited that the RBAC UI will be able to bring additional
>>> >>>> security to
>>> >>>> > > Airflow, and with FAB framework in place we can look into
>>> >>>> leveraging it
>>> >>>> > for
>>> >>>> > > a unified set of APIs used by both UI and CLI.
>>> >>>> > >
>>> >>>> > > Joy
>>> >>>> > >
>>> >>>> > >
>>> >>>> > >
>>> >>>> > > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com>
>>> wrote:
>>> >>>> > >
>>> >>>> > > > Hi folks,
>>> >>>> > > >
>>> >>>> > > > I have a PR <https://github.com/apache/
>>> >>>> incubator-airflow/pull/3015>
>>> >>>> > out
>>> >>>> > > > for the new UI. I've included instructions on how to test it
>>> out
>>> >>>> in the
>>> >>>> > > PR
>>> >>>> > > > description. Looking forward to your feedbacks.
>>> >>>> > > >
>>> >>>> > > > Cheers,
>>> >>>> > > > Joy
>>> >>>> > > >
>>> >>>> > > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com>
>>> wrote:
>>> >>>> > > >
>>> >>>> > > >> Thanks for the background info. Would be really awesome for
>>> you
>>> >>>> to
>>> >>>> > have
>>> >>>> > > >> PyPi access :D I'll make the change to have Airflow
>>> Webserver's
>>> >>>> FAB
>>> >>>> > > >> dependency pointing to my fork for the mean time.
>>> >>>> > > >>
>>> >>>> > > >> For folks who are interested in RBAC, I will be giving a
>>> >>>> talk/demo at
>>> >>>> > > the Airflow
>>> >>>> > > >> Meet-Up
>>> >>>> > > >> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
>>> >>>> > > Meetup/events/244525050/>
>>> >>>> > > >> next Monday. Happy to chat afterwards about it as well :)
>>> >>>> > > >>
>>> >>>> > > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
>>> >>>> > > >> maximebeauchemin@gmail.com> wrote:
>>> >>>> > > >>
>>> >>>> > > >>> A bit of related history here:
>>> >>>> > > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
>>> >>>> > > >>>
>>> >>>> > > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
>>> >>>> > > >>> maximebeauchemin@gmail.com> wrote:
>>> >>>> > > >>>
>>> >>>> > > >>> > Given I have merge rights on FAB I could probably do
>>> another
>>> >>>> round
>>> >>>> > of
>>> >>>> > > >>> > review and get your PRs through. I would really like to
>>> get
>>> >>>> the
>>> >>>> > main
>>> >>>> > > >>> > maintainer's input on things that touch the core
>>> >>>> (composite-key
>>> >>>> > > >>> support) as
>>> >>>> > > >>> > he might have concerns/intuitions that we can't know
>>> about.
>>> >>>> > > >>> >
>>> >>>> > > >>> > I do not have Pypi access though so I cannot push new
>>> releases
>>> >>>> > out. I
>>> >>>> > > >>> > could ask for that.
>>> >>>> > > >>> >
>>> >>>> > > >>> > I've threatened to fork the project before, that's always
>>> an
>>> >>>> > option.
>>> >>>> > > >>> I've
>>> >>>> > > >>> > noticed his involvement is sporadic and comes in bursts.
>>> >>>> > > >>> >
>>> >>>> > > >>> > In the meantime, you can have the dependency in Airflow
>>> >>>> Webserver
>>> >>>> > > >>> pointing
>>> >>>> > > >>> > straight to your fork.
>>> >>>> > > >>> >
>>> >>>> > > >>> > Max
>>> >>>> > > >>> >
>>> >>>> > > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com>
>>> >>>> wrote:
>>> >>>> > > >>> >
>>> >>>> > > >>> >> I just created a new webserver instance if you haven't
>>> >>>> gotten a
>>> >>>> > > >>> chance to
>>> >>>> > > >>> >> fiddle around with the new web UI and the RBAC
>>> configurations
>>> >>>> > > (thanks
>>> >>>> > > >>> >> Maxime for getting started with this earlier!):
>>> >>>> > > >>> >>
>>> >>>> > > >>> >> http://104.209.38.171:8080/
>>> >>>> > > >>> >>
>>> >>>> > > >>> >> Admin Account
>>> >>>> > > >>> >> username: admin
>>> >>>> > > >>> >> password: admin
>>> >>>> > > >>> >>
>>> >>>> > > >>> >> Read-Only Account
>>> >>>> > > >>> >> username: viewer
>>> >>>> > > >>> >> password: password
>>> >>>> > > >>> >>
>>> >>>> > > >>> >>
>>> >>>> > > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <joyg@wepay.com
>>> >
>>> >>>> wrote:
>>> >>>> > > >>> >>
>>> >>>> > > >>> >> > Hi folks,
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > Thanks for all the feedback regarding to the new
>>> Airflow
>>> >>>> > Webserver
>>> >>>> > > >>> UI
>>> >>>> > > >>> >> > <https://github.com/wepay/airflow-webserver/>! I've
>>> been
>>> >>>> > actively
>>> >>>> > > >>> >> > addressing all the bugs that were raised on Github. So
>>> I
>>> >>>> want to
>>> >>>> > > >>> take
>>> >>>> > > >>> >> this
>>> >>>> > > >>> >> > opportunity to discuss two issues coming up:
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > The first issue is unaddressed PRs in FAB. If these PRs
>>> >>>> continue
>>> >>>> > > to
>>> >>>> > > >>> stay
>>> >>>> > > >>> >> > unaddressed, RBAC is blocked from making further
>>> progress.
>>> >>>> If
>>> >>>> > this
>>> >>>> > > >>> >> continue
>>> >>>> > > >>> >> > to be an issue, I'm inclined to fork FAB, even though
>>> it's
>>> >>>> not
>>> >>>> > > >>> >> idealistic.
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > - PR/631 <https://github.com/dpgaspar/F
>>> >>>> > > lask-AppBuilder/pull/631>
>>> >>>> > > >>> >> Binary
>>> >>>> > > >>> >> > column support (merged, unreleased)
>>> >>>> > > >>> >> > <https://github.com/dpgaspar/F
>>> lask-AppBuilder/pull/631>
>>> >>>> > > >>> >> > - PR/639 <https://github.com/dpgaspar/F
>>> >>>> > > lask-AppBuilder/pull/639>
>>> >>>> > > >>> >> Composite
>>> >>>> > > >>> >> > primary key support (unmerged)
>>> >>>> > > >>> >> > - PR/655 <https://github.com/dpgaspar/F
>>> >>>> > > lask-AppBuilder/pull/655>
>>> >>>> > > >>> >> Form
>>> >>>> > > >>> >> > prefill support (unmerged)
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > The second issue is an open question about the next
>>> step of
>>> >>>> > > Airflow
>>> >>>> > > >>> >> > Webserver itself. Here are the 3 potential directions
>>> we
>>> >>>> could
>>> >>>> > > >>> take, and
>>> >>>> > > >>> >> > I've added my thought on each.
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > 1. Permanently keep Airflow Webserver as a separated
>>> >>>> package
>>> >>>> > from
>>> >>>> > > >>> >> Airflow,
>>> >>>> > > >>> >> > and treat it as another UI option. Keep `www` in
>>> Airflow.
>>> >>>> Allow
>>> >>>> > > >>> >> development
>>> >>>> > > >>> >> > on both UIs.
>>> >>>> > > >>> >> > *I'm not a fan of this. When there is an existing UI in
>>> >>>> Airflow,
>>> >>>> > > >>> most
>>> >>>> > > >>> >> > contributors would prefer to maintain the official
>>> version
>>> >>>> that
>>> >>>> > is
>>> >>>> > > >>> >> > installed out-of-the-box. **Having a second UI outside
>>> of
>>> >>>> > Airflow
>>> >>>> > > >>> will
>>> >>>> > > >>> >> > make maintaining it very difficult, leading to an
>>> eventual
>>> >>>> death
>>> >>>> > > of
>>> >>>> > > >>> the
>>> >>>> > > >>> >> new
>>> >>>> > > >>> >> > UI :(*
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > 2. Permanently keep Airflow Webserver as a separated
>>> >>>> package
>>> >>>> > from
>>> >>>> > > >>> >> Airflow,
>>> >>>> > > >>> >> > but freeze all development on `www` and direct all
>>> future
>>> >>>> UI
>>> >>>> > > >>> >> development
>>> >>>> > > >>> >> > to Airflow Webserver, eventually removing `www`
>>> completely
>>> >>>> when
>>> >>>> > > >>> Airflow
>>> >>>> > > >>> >> > Webserver is stable.
>>> >>>> > > >>> >> > *I'm not a fan of this either. First of all, the views
>>> and
>>> >>>> > models
>>> >>>> > > >>> are
>>> >>>> > > >>> >> > tightly coupled in both old and new UI; until we have a
>>> >>>> > > full-fledged
>>> >>>> > > >>> >> REST
>>> >>>> > > >>> >> > API to build the UI (and cli) on top of it, separating
>>> >>>> them to a
>>> >>>> > > >>> >> separate
>>> >>>> > > >>> >> > package now will potentially cause dependency issues
>>> and
>>> >>>> add
>>> >>>> > > >>> >> complication
>>> >>>> > > >>> >> > to our release cycle. **Secondly, **majority of Airflow
>>> >>>> users
>>> >>>> > run
>>> >>>> > > >>> >> Airflow
>>> >>>> > > >>> >> > with the UI; it's one of Airflow's best features.
>>> >>>> Separating UI
>>> >>>> > > out
>>> >>>> > > >>> of
>>> >>>> > > >>> >> > Airflow core will complicate setup and configuration,
>>> while
>>> >>>> > making
>>> >>>> > > >>> >> Airflow
>>> >>>> > > >>> >> > core less complete.*
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`,
>>> >>>> freeze
>>> >>>> > all
>>> >>>> > > >>> >> > development on `www`, eventually removing `www`
>>> completely
>>> >>>> when
>>> >>>> > > >>> `www2`
>>> >>>> > > >>> >> is
>>> >>>> > > >>> >> > stable.
>>> >>>> > > >>> >> > *This makes the most sense to me. Airflow Webserver is
>>> >>>> developed
>>> >>>> > > >>> with
>>> >>>> > > >>> >> the
>>> >>>> > > >>> >> > goal of feature parity to the current UI, plus
>>> additional
>>> >>>> RBAC
>>> >>>> > > >>> >> capability,
>>> >>>> > > >>> >> > in hope to replace the old UI completely. Yes, this
>>> means
>>> >>>> there
>>> >>>> > > >>> will be
>>> >>>> > > >>> >> a
>>> >>>> > > >>> >> > short period of having to maintain two UIs, but once we
>>> >>>> freeze
>>> >>>> > > >>> >> development
>>> >>>> > > >>> >> > on www, it shouldn't be a concern for long.*
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > I'd love to hear everyone's thoughts on this! I'm
>>> excited
>>> >>>> about
>>> >>>> > > >>> bringing
>>> >>>> > > >>> >> > RBAC to airflow and I hope it's something others will
>>> find
>>> >>>> > useful
>>> >>>> > > as
>>> >>>> > > >>> >> well!
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > Cheers,
>>> >>>> > > >>> >> > Joy
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <
>>> joyg@wepay.com>
>>> >>>> > wrote:
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >> >> Thank you everyone for the active feedback so far, and
>>> >>>> thanks
>>> >>>> > for
>>> >>>> > > >>> >> setting
>>> >>>> > > >>> >> >> up the demo Maxime!
>>> >>>> > > >>> >> >>
>>> >>>> > > >>> >> >> Going to work on pruning through the issues in the
>>> >>>> upcoming
>>> >>>> > days.
>>> >>>> > > >>> >> >>
>>> >>>> > > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception
>>> >>>> message
>>> >>>> > so I
>>> >>>> > > >>> can
>>> >>>> > > >>> >> >> look into it? Otherwise I'll wait until it's down
>>> again =P
>>> >>>> > > >>> >> >>
>>> >>>> > > >>> >> >> Cheers,
>>> >>>> > > >>> >> >>
>>> >>>> > > >>> >> >> Joy
>>> >>>> > > >>> >> >>
>>> >>>> > > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
>>> >>>> > > >>> >> >> maximebeauchemin@gmail.com> wrote:
>>> >>>> > > >>> >> >>
>>> >>>> > > >>> >> >>> I just restarted it, not sure how long it will take
>>> to
>>> >>>> get in
>>> >>>> > a
>>> >>>> > > >>> bad
>>> >>>> > > >>> >> state
>>> >>>> > > >>> >> >>> again...
>>> >>>> > > >>> >> >>>
>>> >>>> > > >>> >> >>> Max
>>> >>>> > > >>> >> >>>
>>> >>>> > > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
>>> >>>> > > >>> >> <fokko@driesprong.frl
>>> >>>> > > >>> >> >>> >
>>> >>>> > > >>> >> >>> wrote:
>>> >>>> > > >>> >> >>>
>>> >>>> > > >>> >> >>> > Good morning,
>>> >>>> > > >>> >> >>> >
>>> >>>> > > >>> >> >>> > The demo provided by Max is down, it throws a
>>> >>>> > > >>> SQLAlchemyexception
>>> >>>> > > >>> >> :'(
>>> >>>> > > >>> >> >>> >
>>> >>>> > > >>> >> >>> > Cheers, Fokko
>>> >>>> > > >>> >> >>> >
>>> >>>> > > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
>>> >>>> > > >>> criccomini@apache.org>:
>>> >>>> > > >>> >> >>> >
>>> >>>> > > >>> >> >>> > > @bolke, open issues on the Github repo, please.
>>> >>>> > > >>> >> >>> > >
>>> >>>> > > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin
>>> <
>>> >>>> > > >>> >> bdbruin@gmail.com>
>>> >>>> > > >>> >> >>> > > wrote:
>>> >>>> > > >>> >> >>> > >
>>> >>>> > > >>> >> >>> > > > Chris,
>>> >>>> > > >>> >> >>> > > >
>>> >>>> > > >>> >> >>> > > > Do you want us to report bugs somewhere (I have
>>> >>>> > > encountered
>>> >>>> > > >>> a
>>> >>>> > > >>> >> >>> few)? Or
>>> >>>> > > >>> >> >>> > > > just generic user experiences posted here?
>>> >>>> > > >>> >> >>> > > >
>>> >>>> > > >>> >> >>> > > > Cheers
>>> >>>> > > >>> >> >>> > > > Bolke
>>> >>>> > > >>> >> >>> > > >
>>> >>>> > > >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
>>> >>>> > > >>> >> criccomini@apache.org
>>> >>>> > > >>> >> >>> >
>>> >>>> > > >>> >> >>> > > wrote:
>>> >>>> > > >>> >> >>> > > > >
>>> >>>> > > >>> >> >>> > > > > Hey all,
>>> >>>> > > >>> >> >>> > > > >
>>> >>>> > > >>> >> >>> > > > > I know the weekend is coming up, and for
>>> those
>>> >>>> of us
>>> >>>> > in
>>> >>>> > > >>> the
>>> >>>> > > >>> >> US,
>>> >>>> > > >>> >> >>> next
>>> >>>> > > >>> >> >>> > > week
>>> >>>> > > >>> >> >>> > > > > is a bit of a slow holiday week. Would love
>>> to
>>> >>>> get
>>> >>>> > some
>>> >>>> > > >>> >> feedback
>>> >>>> > > >>> >> >>> from
>>> >>>> > > >>> >> >>> > > > > everyone on this. The goal would ideally to
>>> be to
>>> >>>> > > >>> converge on
>>> >>>> > > >>> >> >>> this
>>> >>>> > > >>> >> >>> > and
>>> >>>> > > >>> >> >>> > > > > eventually replace the existing Airflow UI
>>> with
>>> >>>> this
>>> >>>> > > one.
>>> >>>> > > >>> >> >>> > > > >
>>> >>>> > > >>> >> >>> > > > > Cheers,
>>> >>>> > > >>> >> >>> > > > > Chris
>>> >>>> > > >>> >> >>> > > > >
>>> >>>> > > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
>>> >>>> > > joyg@wepay.com>
>>> >>>> > > >>> >> wrote:
>>> >>>> > > >>> >> >>> > > > >
>>> >>>> > > >>> >> >>> > > > >> Hi guys.
>>> >>>> > > >>> >> >>> > > > >>
>>> >>>> > > >>> >> >>> > > > >> I've been working on moving airflow from
>>> >>>> Flask-Admin
>>> >>>> > to
>>> >>>> > > >>> >> >>> > > Flask-AppBuilder
>>> >>>> > > >>> >> >>> > > > >> for RBAC
>>> >>>> > > >>> >> >>> > > > >> <https://cwiki.apache.org/
>>> >>>> > confluence/display/AIRFLOW/
>>> >>>> > > >>> >> >>> > > > Airflow+RBAC+proposal
>>> >>>> > > >>> >> >>> > > > >>> ,
>>> >>>> > > >>> >> >>> > > > >> check it out at
>>> https://github.com/wepay/airfl
>>> >>>> > > >>> ow-webserver.
>>> >>>> > > >>> >> >>> > > > >>
>>> >>>> > > >>> >> >>> > > > >> It's still a work-in-progress, but most
>>> >>>> features you
>>> >>>> > > see
>>> >>>> > > >>> in
>>> >>>> > > >>> >> the
>>> >>>> > > >>> >> >>> > > > webserver
>>> >>>> > > >>> >> >>> > > > >> UI today is available there. For those who
>>> are
>>> >>>> > > >>> interested in
>>> >>>> > > >>> >> >>> RBAC,
>>> >>>> > > >>> >> >>> > I'd
>>> >>>> > > >>> >> >>> > > > love
>>> >>>> > > >>> >> >>> > > > >> to get some early feedback in terms of the
>>> >>>> following:
>>> >>>> > > >>> >> >>> > > > >>
>>> >>>> > > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any
>>> bugs/regressions)
>>> >>>> > > >>> >> >>> > > > >> - Setup issues
>>> >>>> > > >>> >> >>> > > > >> - Ease of integration with third party auth
>>> >>>> (i.e.
>>> >>>> > LDAP,
>>> >>>> > > >>> AD,
>>> >>>> > > >>> >> >>> OAuth,
>>> >>>> > > >>> >> >>> > > etc.)
>>> >>>> > > >>> >> >>> > > > >> - Any other thoughts/concerns
>>> >>>> > > >>> >> >>> > > > >>
>>> >>>> > > >>> >> >>> > > > >> Thanks a lot!
>>> >>>> > > >>> >> >>> > > > >>
>>> >>>> > > >>> >> >>> > > > >> Cheers,
>>> >>>> > > >>> >> >>> > > > >> Joy
>>> >>>> > > >>> >> >>> > > > >>
>>> >>>> > > >>> >> >>> > > >
>>> >>>> > > >>> >> >>> > > >
>>> >>>> > > >>> >> >>> > >
>>> >>>> > > >>> >> >>> >
>>> >>>> > > >>> >> >>>
>>> >>>> > > >>> >> >>
>>> >>>> > > >>> >> >>
>>> >>>> > > >>> >> >
>>> >>>> > > >>> >>
>>> >>>> > > >>> >
>>> >>>> > > >>> >
>>> >>>> > > >>>
>>> >>>> > > >>
>>> >>>> > > >>
>>> >>>> > > >
>>> >>>> > >
>>> >>>> >
>>> >>>>
>>> >>>
>>> >>>
>>> >>
>>> >
>>>
>>
>>
>
Re: DAG Level permissions (was Re: RBAC Update)
Posted by Tao Feng <fe...@gmail.com>.
Hi Max, Joy and Everyone,
Based on the discussion, I put up a work in progress pr (
https://github.com/apache/incubator-airflow/pull/3197/) with a doc(
https://docs.google.com/document/d/1qs26lE9kAuCY0Qa0ga-
80EQ7d7m4s-590lhjtMBjmxw/edit#) for DAG level access. I would like to get
some early feedbacks and see if I miss anything or am in the wrong
direction as it touches the core part.
In the meantime, I will still continue improving the pr for couples of
todos.
Looking forward to your feedback.
Thanks,
-Tao
On Mon, Apr 2, 2018 at 2:44 PM, Tao Feng <fe...@gmail.com> wrote:
> Hi everyone,
>
> Thanks a lot for all the great discussions. To summarize in brief, here
> are the few approaches we discussed so far:
>
> 1. One permission per DAG. The user has homogenous rights on the dag.
> The concerns:
>
> - not flexible to certain use cases(e.g the user has view only access
> on certain dags and write access on certain other dags );
> - difficult to implement as it breaks the existing FAB security model.
>
> 2. Extend current model(ab_permission_view_role) with an additional
> optional column (dag_id).
> The concerns:
>
> - introduce a 3rd column to existing permission_view table.
> - It requires us to handle db migration carefully from view only
> access UI to dag level access UI.
>
> 3. Define a set of pre-defined dag-level permissions. Reuse the current
> existing permission_view model and consider each individual dag as a new
> view.
>
> It seems that the 3rd approach is a preferable approach which makes the
> security model easy to extend without introducing additional complexity. If
> no other concern, I will work with Max to create an initial proposal / PR
> based on 3rd approach for the work(https://issues.apache.org
> /jira/browse/AIRFLOW-2267).
>
> Thanks,
> -Tao
>
> On Sat, Mar 31, 2018 at 12:09 AM, Joy Gao <jo...@wepay.com> wrote:
>
>> +1!
>>
>> I was originally tempted to re-use existing perms and views for dag-level
>> access control since dag-level perm/view is a subset of view-level
>> perm/view, but your proposal of defining new dag-level perms/views
>> independent from view-level perms/views is interesting. This actually
>> makes
>> a lot of sense, since even in the existing models, views can also be menu
>> options, so we are simply extending the concept of views to include dags
>> as
>> well.
>>
>> On Fri, Mar 30, 2018 at 5:24 PM Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>> > I'd suggest something else that avoids having to add a 3rd column. I
>> think
>> > we can fit our use case into the existing structure nicely.
>> >
>> > My idea is to mimic what FAB does with its own Models.
>> >
>> > When you create a Model and ModelView in FAB (say DagRun for example),
>> it
>> > creates a new view_menu (DagRun) and matches it with existing permission
>> > (can_read, can_delete, can_edit, can_show, ...) to create a new set of
>> > "permission_views" which are combinations of permission and view_menu
>> as in
>> > "DagRun - can_read", "DagRun - can_edit", ... It's not a cartesian
>> product
>> > of all perms and view_menus, it's a predetermined list of model-specific
>> > perms that get combined with DagRun here.
>> >
>> > Similarly, when Airflow would detect a new DAG (example "my_dag"), it
>> > would create a new view_menu `my_dag`, and match it with permissions
>> that
>> > are identified as "combinable" with DAG. To avoid potential conflicts
>> I'd
>> > create new permissions that are DAG-specific like "dag_clear",
>> "dag_run",
>> > "dag_view".
>> >
>> > I'm arguing about the how to use the FAB model here specifically. I
>> think
>> > this allows for all the flexibility we need without changing the model.
>> I
>> > care less about what exactly the atomicity of the per DAG perms should
>> look
>> > like. As far as I'm concerned, per-DAG read and write is probably
>> granular
>> > enough
>> >
>> > Also note that:
>> > * we need an "_all_dags" logical DAG, meaning we'd have extra
>> > permission_views "_all_dags - dag_view", "_all_dags - dag_run",
>> "all_dags -
>> > dag_clear"
>> > * we probably want to derive FAB's SecurityManger and have an
>> > AirflowSecurityManager that has an extra method "can_dag_action(user,
>> > dag_id, action)". The SecurityManger class is neat because people can
>> > provide their own if they want and override methods. That means that you
>> > can defer any of the security-related checks to another system if you
>> want
>> > to. Many companies have some sort of company RBAC system and that can be
>> > used to integrate with it.
>> >
>> > Max
>> >
>> >
>> >
>> > On Fri, Mar 30, 2018 at 4:54 PM, Joy Gao <jo...@wepay.com> wrote:
>> >
>> >> Hi all,
>> >>
>> >> I also agree that having view-only access to some dags while write
>> access
>> >> to other dags is useful, so I prefer option 2. Although option 2 is
>> more
>> >> difficult to manage, it is cleaner and more consistent with the current
>> >> security model. (On the other hand, even though option 1 may be may be
>> >> easier to manage, it might be trickier to implement: with one perm per
>> >> dag, it breaks the existing FAB security model since the existing, more
>> >> granular permissions will have to be grouped for each dag).
>> >>
>> >> What Brian suggested in the other thread makes sense to me...
>> >>
>> >> The current security model in FAB looks like the following:
>> >>
>> >> --- permissions
>> >> user --- role --- permission_view ---|
>> >> --- views
>> >>
>> >> FAB generates a few relationship tables to manage the mappings, one of
>> >> them is *ab_permission_view_role*, which has all the
>> >> role-to-permission_view mapping. We can add a dag_id column to this
>> >> relationship that expresses dag-level granularity, so the security
>> model
>> >> becomes:
>> >>
>> >> ---
>> >> permissions
>> >> |
>> >> user --- role --- dag_permission_view --- --- views
>> >> |
>> >> ---
>> dags
>> >>
>> >> We can make the dag_id field an optional field in the relationship, and
>> >> only lazily add new dag-level mappings for DAGs that specify 'access
>> >> control'. This way we don't have to create new permissions or views.
>> (We
>> >> will still have to introduce new dag-level roles on top of the 5
>> generic
>> >> roles (public/viewer/user/op/admin)).
>> >>
>> >> (I think this is similar to what Arthur suggested earlier, but not sure
>> >> if I interpreted correctly).
>> >>
>> >> Joy
>> >>
>> >> On Thu, Mar 29, 2018 at 10:01 AM, Arthur Wiedmer <
>> >> arthur.wiedmer@gmail.com> wrote:
>> >>
>> >>> (Creating a new thread)
>> >>>
>> >>> Hi Max,
>> >>>
>> >>> I was just wondering about this. There are definite use cases for
>> people
>> >>> having only view access to some DAGs, mostly for monitoring. I want
>> to know
>> >>> what the upstream DAGs are doing, but maybe I don't need clear/run
>> access.
>> >>>
>> >>> I feel like the granular operation permissions will start to become
>> >>> unwieldy fast. It grows as (# users * # DAGs * # operations) but with
>> >>> hopefully a small constant factor in front: most people should only
>> have a
>> >>> small number of DAGs they care about. The Ops team has a need to have
>> >>> access to all on the other hand.
>> >>>
>> >>> I was wondering we could get by with something slightly more complex
>> but
>> >>> easier on the size of that permissions table :
>> >>> 1) One toggle on the user level for broad access (ALL:ALL,
>> >>> RUN/CLEAR:ALL, VIEW:ALL) default NULL
>> >>> 2) More granular permissions at the DAG level.
>> >>>
>> >>> So in order, check the user's broad level permission first, then DAG
>> >>> level. For large amounts of DAGs, this should help shave a little bit
>> from
>> >>> that table.
>> >>>
>> >>> Best,
>> >>> Arthur
>> >>>
>> >>>
>> >>> On Thu, Mar 29, 2018 at 8:27 AM, Maxime Beauchemin <
>> >>> maximebeauchemin@gmail.com> wrote:
>> >>>
>> >>>> Hijacking the thread further here, any thoughts on how to breakdown
>> per
>> >>>> DAG
>> >>>> access?
>> >>>>
>> >>>> Tao & I are talking about introducing per-DAG permissions and one big
>> >>>> question is whether we'll need to support different operation-types
>> at a
>> >>>> per-DAG level, which changes the way we need to model the perms.
>> >>>>
>> >>>> First [simpler] option is to introduce one perm per DAG. If you have
>> >>>> access
>> >>>> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have
>> >>>> homogenous
>> >>>> rights on the DAGs you have access to.
>> >>>>
>> >>>> Second option is to have a breakdown per DAG. Meaning for each DAG we
>> >>>> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...).
>> So
>> >>>> one
>> >>>> user could have modify on some DAGs, view on others, and other DAGs
>> >>>> would
>> >>>> be invisible. This could be broken down further ({dag_id}_can_clear,
>> >>>> ...)
>> >>>> but it gets hard to manage.
>> >>>>
>> >>>> Thoughts?
>> >>>>
>> >>>> Max
>> >>>>
>> >>>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fe...@gmail.com>
>> wrote:
>> >>>>
>> >>>> > Great work Joy. This is awesome! I am interested in helping out the
>> >>>> per dag
>> >>>> > level access. Just created a ticket to check(AIRFLOW-2267). Let me
>> >>>> know if
>> >>>> > you have any suggestions. I will share my proposal once I am ready.
>> >>>> >
>> >>>> > On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
>> >>>> >
>> >>>> > > Hey guys!
>> >>>> > >
>> >>>> > > The RBAC UI <https://github.com/apache/inc
>> ubator-airflow/pull/3015>
>> >>>> has
>> >>>> > > been merged to master. I'm looking forward to early adopters'
>> >>>> feedback
>> >>>> > and
>> >>>> > > bug reports. I also hope to have more folks helping out with the
>> >>>> RBAC UI,
>> >>>> > > especially with introducing DAG-Level access control, which is a
>> >>>> feature
>> >>>> > > that a lot of people have been asking. If you are interested in
>> >>>> helping
>> >>>> > out
>> >>>> > > with this effort, let's talk more!
>> >>>> > >
>> >>>> > > This commit will be in the 1.10.0 release, and we are going to
>> >>>> maintain
>> >>>> > > both UIs simultaneously for a short period of time. Once RBAC UI
>> is
>> >>>> > stable
>> >>>> > > and battle-tested, we will deprecate the old UI and eventually
>> >>>> remove it
>> >>>> > > from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to
>> >>>> prevent
>> >>>> > > two UIs from forking into separate paths, as that would become
>> very
>> >>>> > > difficult to maintain.
>> >>>> > >
>> >>>> > > Going forward while both UIs are up, if you are making a change
>> to
>> >>>> any
>> >>>> > > files in airflow/www/ (old UI), where applicable, please also
>> make
>> >>>> the
>> >>>> > > change to the airflow/www_rbac/ (new UI). If you rather not make
>> >>>> changes
>> >>>> > in
>> >>>> > > both UIs, it is recommended that you only make the changes to the
>> >>>> RBAC
>> >>>> > UI,
>> >>>> > > since that is the one we are maintaining in the long term.
>> >>>> > >
>> >>>> > > I'm excited that the RBAC UI will be able to bring additional
>> >>>> security to
>> >>>> > > Airflow, and with FAB framework in place we can look into
>> >>>> leveraging it
>> >>>> > for
>> >>>> > > a unified set of APIs used by both UI and CLI.
>> >>>> > >
>> >>>> > > Joy
>> >>>> > >
>> >>>> > >
>> >>>> > >
>> >>>> > > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
>> >>>> > >
>> >>>> > > > Hi folks,
>> >>>> > > >
>> >>>> > > > I have a PR <https://github.com/apache/
>> >>>> incubator-airflow/pull/3015>
>> >>>> > out
>> >>>> > > > for the new UI. I've included instructions on how to test it
>> out
>> >>>> in the
>> >>>> > > PR
>> >>>> > > > description. Looking forward to your feedbacks.
>> >>>> > > >
>> >>>> > > > Cheers,
>> >>>> > > > Joy
>> >>>> > > >
>> >>>> > > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com>
>> wrote:
>> >>>> > > >
>> >>>> > > >> Thanks for the background info. Would be really awesome for
>> you
>> >>>> to
>> >>>> > have
>> >>>> > > >> PyPi access :D I'll make the change to have Airflow
>> Webserver's
>> >>>> FAB
>> >>>> > > >> dependency pointing to my fork for the mean time.
>> >>>> > > >>
>> >>>> > > >> For folks who are interested in RBAC, I will be giving a
>> >>>> talk/demo at
>> >>>> > > the Airflow
>> >>>> > > >> Meet-Up
>> >>>> > > >> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
>> >>>> > > Meetup/events/244525050/>
>> >>>> > > >> next Monday. Happy to chat afterwards about it as well :)
>> >>>> > > >>
>> >>>> > > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
>> >>>> > > >> maximebeauchemin@gmail.com> wrote:
>> >>>> > > >>
>> >>>> > > >>> A bit of related history here:
>> >>>> > > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
>> >>>> > > >>>
>> >>>> > > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
>> >>>> > > >>> maximebeauchemin@gmail.com> wrote:
>> >>>> > > >>>
>> >>>> > > >>> > Given I have merge rights on FAB I could probably do
>> another
>> >>>> round
>> >>>> > of
>> >>>> > > >>> > review and get your PRs through. I would really like to get
>> >>>> the
>> >>>> > main
>> >>>> > > >>> > maintainer's input on things that touch the core
>> >>>> (composite-key
>> >>>> > > >>> support) as
>> >>>> > > >>> > he might have concerns/intuitions that we can't know about.
>> >>>> > > >>> >
>> >>>> > > >>> > I do not have Pypi access though so I cannot push new
>> releases
>> >>>> > out. I
>> >>>> > > >>> > could ask for that.
>> >>>> > > >>> >
>> >>>> > > >>> > I've threatened to fork the project before, that's always
>> an
>> >>>> > option.
>> >>>> > > >>> I've
>> >>>> > > >>> > noticed his involvement is sporadic and comes in bursts.
>> >>>> > > >>> >
>> >>>> > > >>> > In the meantime, you can have the dependency in Airflow
>> >>>> Webserver
>> >>>> > > >>> pointing
>> >>>> > > >>> > straight to your fork.
>> >>>> > > >>> >
>> >>>> > > >>> > Max
>> >>>> > > >>> >
>> >>>> > > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com>
>> >>>> wrote:
>> >>>> > > >>> >
>> >>>> > > >>> >> I just created a new webserver instance if you haven't
>> >>>> gotten a
>> >>>> > > >>> chance to
>> >>>> > > >>> >> fiddle around with the new web UI and the RBAC
>> configurations
>> >>>> > > (thanks
>> >>>> > > >>> >> Maxime for getting started with this earlier!):
>> >>>> > > >>> >>
>> >>>> > > >>> >> http://104.209.38.171:8080/
>> >>>> > > >>> >>
>> >>>> > > >>> >> Admin Account
>> >>>> > > >>> >> username: admin
>> >>>> > > >>> >> password: admin
>> >>>> > > >>> >>
>> >>>> > > >>> >> Read-Only Account
>> >>>> > > >>> >> username: viewer
>> >>>> > > >>> >> password: password
>> >>>> > > >>> >>
>> >>>> > > >>> >>
>> >>>> > > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com>
>> >>>> wrote:
>> >>>> > > >>> >>
>> >>>> > > >>> >> > Hi folks,
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > Thanks for all the feedback regarding to the new Airflow
>> >>>> > Webserver
>> >>>> > > >>> UI
>> >>>> > > >>> >> > <https://github.com/wepay/airflow-webserver/>! I've
>> been
>> >>>> > actively
>> >>>> > > >>> >> > addressing all the bugs that were raised on Github. So I
>> >>>> want to
>> >>>> > > >>> take
>> >>>> > > >>> >> this
>> >>>> > > >>> >> > opportunity to discuss two issues coming up:
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > The first issue is unaddressed PRs in FAB. If these PRs
>> >>>> continue
>> >>>> > > to
>> >>>> > > >>> stay
>> >>>> > > >>> >> > unaddressed, RBAC is blocked from making further
>> progress.
>> >>>> If
>> >>>> > this
>> >>>> > > >>> >> continue
>> >>>> > > >>> >> > to be an issue, I'm inclined to fork FAB, even though
>> it's
>> >>>> not
>> >>>> > > >>> >> idealistic.
>> >>>> > > >>> >> >
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > - PR/631 <https://github.com/dpgaspar/F
>> >>>> > > lask-AppBuilder/pull/631>
>> >>>> > > >>> >> Binary
>> >>>> > > >>> >> > column support (merged, unreleased)
>> >>>> > > >>> >> > <https://github.com/dpgaspar/F
>> lask-AppBuilder/pull/631>
>> >>>> > > >>> >> > - PR/639 <https://github.com/dpgaspar/F
>> >>>> > > lask-AppBuilder/pull/639>
>> >>>> > > >>> >> Composite
>> >>>> > > >>> >> > primary key support (unmerged)
>> >>>> > > >>> >> > - PR/655 <https://github.com/dpgaspar/F
>> >>>> > > lask-AppBuilder/pull/655>
>> >>>> > > >>> >> Form
>> >>>> > > >>> >> > prefill support (unmerged)
>> >>>> > > >>> >> >
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > The second issue is an open question about the next
>> step of
>> >>>> > > Airflow
>> >>>> > > >>> >> > Webserver itself. Here are the 3 potential directions we
>> >>>> could
>> >>>> > > >>> take, and
>> >>>> > > >>> >> > I've added my thought on each.
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > 1. Permanently keep Airflow Webserver as a separated
>> >>>> package
>> >>>> > from
>> >>>> > > >>> >> Airflow,
>> >>>> > > >>> >> > and treat it as another UI option. Keep `www` in
>> Airflow.
>> >>>> Allow
>> >>>> > > >>> >> development
>> >>>> > > >>> >> > on both UIs.
>> >>>> > > >>> >> > *I'm not a fan of this. When there is an existing UI in
>> >>>> Airflow,
>> >>>> > > >>> most
>> >>>> > > >>> >> > contributors would prefer to maintain the official
>> version
>> >>>> that
>> >>>> > is
>> >>>> > > >>> >> > installed out-of-the-box. **Having a second UI outside
>> of
>> >>>> > Airflow
>> >>>> > > >>> will
>> >>>> > > >>> >> > make maintaining it very difficult, leading to an
>> eventual
>> >>>> death
>> >>>> > > of
>> >>>> > > >>> the
>> >>>> > > >>> >> new
>> >>>> > > >>> >> > UI :(*
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > 2. Permanently keep Airflow Webserver as a separated
>> >>>> package
>> >>>> > from
>> >>>> > > >>> >> Airflow,
>> >>>> > > >>> >> > but freeze all development on `www` and direct all
>> future
>> >>>> UI
>> >>>> > > >>> >> development
>> >>>> > > >>> >> > to Airflow Webserver, eventually removing `www`
>> completely
>> >>>> when
>> >>>> > > >>> Airflow
>> >>>> > > >>> >> > Webserver is stable.
>> >>>> > > >>> >> > *I'm not a fan of this either. First of all, the views
>> and
>> >>>> > models
>> >>>> > > >>> are
>> >>>> > > >>> >> > tightly coupled in both old and new UI; until we have a
>> >>>> > > full-fledged
>> >>>> > > >>> >> REST
>> >>>> > > >>> >> > API to build the UI (and cli) on top of it, separating
>> >>>> them to a
>> >>>> > > >>> >> separate
>> >>>> > > >>> >> > package now will potentially cause dependency issues and
>> >>>> add
>> >>>> > > >>> >> complication
>> >>>> > > >>> >> > to our release cycle. **Secondly, **majority of Airflow
>> >>>> users
>> >>>> > run
>> >>>> > > >>> >> Airflow
>> >>>> > > >>> >> > with the UI; it's one of Airflow's best features.
>> >>>> Separating UI
>> >>>> > > out
>> >>>> > > >>> of
>> >>>> > > >>> >> > Airflow core will complicate setup and configuration,
>> while
>> >>>> > making
>> >>>> > > >>> >> Airflow
>> >>>> > > >>> >> > core less complete.*
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`,
>> >>>> freeze
>> >>>> > all
>> >>>> > > >>> >> > development on `www`, eventually removing `www`
>> completely
>> >>>> when
>> >>>> > > >>> `www2`
>> >>>> > > >>> >> is
>> >>>> > > >>> >> > stable.
>> >>>> > > >>> >> > *This makes the most sense to me. Airflow Webserver is
>> >>>> developed
>> >>>> > > >>> with
>> >>>> > > >>> >> the
>> >>>> > > >>> >> > goal of feature parity to the current UI, plus
>> additional
>> >>>> RBAC
>> >>>> > > >>> >> capability,
>> >>>> > > >>> >> > in hope to replace the old UI completely. Yes, this
>> means
>> >>>> there
>> >>>> > > >>> will be
>> >>>> > > >>> >> a
>> >>>> > > >>> >> > short period of having to maintain two UIs, but once we
>> >>>> freeze
>> >>>> > > >>> >> development
>> >>>> > > >>> >> > on www, it shouldn't be a concern for long.*
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > I'd love to hear everyone's thoughts on this! I'm
>> excited
>> >>>> about
>> >>>> > > >>> bringing
>> >>>> > > >>> >> > RBAC to airflow and I hope it's something others will
>> find
>> >>>> > useful
>> >>>> > > as
>> >>>> > > >>> >> well!
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > Cheers,
>> >>>> > > >>> >> > Joy
>> >>>> > > >>> >> >
>> >>>> > > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <
>> joyg@wepay.com>
>> >>>> > wrote:
>> >>>> > > >>> >> >
>> >>>> > > >>> >> >> Thank you everyone for the active feedback so far, and
>> >>>> thanks
>> >>>> > for
>> >>>> > > >>> >> setting
>> >>>> > > >>> >> >> up the demo Maxime!
>> >>>> > > >>> >> >>
>> >>>> > > >>> >> >> Going to work on pruning through the issues in the
>> >>>> upcoming
>> >>>> > days.
>> >>>> > > >>> >> >>
>> >>>> > > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception
>> >>>> message
>> >>>> > so I
>> >>>> > > >>> can
>> >>>> > > >>> >> >> look into it? Otherwise I'll wait until it's down
>> again =P
>> >>>> > > >>> >> >>
>> >>>> > > >>> >> >> Cheers,
>> >>>> > > >>> >> >>
>> >>>> > > >>> >> >> Joy
>> >>>> > > >>> >> >>
>> >>>> > > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
>> >>>> > > >>> >> >> maximebeauchemin@gmail.com> wrote:
>> >>>> > > >>> >> >>
>> >>>> > > >>> >> >>> I just restarted it, not sure how long it will take to
>> >>>> get in
>> >>>> > a
>> >>>> > > >>> bad
>> >>>> > > >>> >> state
>> >>>> > > >>> >> >>> again...
>> >>>> > > >>> >> >>>
>> >>>> > > >>> >> >>> Max
>> >>>> > > >>> >> >>>
>> >>>> > > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
>> >>>> > > >>> >> <fokko@driesprong.frl
>> >>>> > > >>> >> >>> >
>> >>>> > > >>> >> >>> wrote:
>> >>>> > > >>> >> >>>
>> >>>> > > >>> >> >>> > Good morning,
>> >>>> > > >>> >> >>> >
>> >>>> > > >>> >> >>> > The demo provided by Max is down, it throws a
>> >>>> > > >>> SQLAlchemyexception
>> >>>> > > >>> >> :'(
>> >>>> > > >>> >> >>> >
>> >>>> > > >>> >> >>> > Cheers, Fokko
>> >>>> > > >>> >> >>> >
>> >>>> > > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
>> >>>> > > >>> criccomini@apache.org>:
>> >>>> > > >>> >> >>> >
>> >>>> > > >>> >> >>> > > @bolke, open issues on the Github repo, please.
>> >>>> > > >>> >> >>> > >
>> >>>> > > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
>> >>>> > > >>> >> bdbruin@gmail.com>
>> >>>> > > >>> >> >>> > > wrote:
>> >>>> > > >>> >> >>> > >
>> >>>> > > >>> >> >>> > > > Chris,
>> >>>> > > >>> >> >>> > > >
>> >>>> > > >>> >> >>> > > > Do you want us to report bugs somewhere (I have
>> >>>> > > encountered
>> >>>> > > >>> a
>> >>>> > > >>> >> >>> few)? Or
>> >>>> > > >>> >> >>> > > > just generic user experiences posted here?
>> >>>> > > >>> >> >>> > > >
>> >>>> > > >>> >> >>> > > > Cheers
>> >>>> > > >>> >> >>> > > > Bolke
>> >>>> > > >>> >> >>> > > >
>> >>>> > > >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
>> >>>> > > >>> >> criccomini@apache.org
>> >>>> > > >>> >> >>> >
>> >>>> > > >>> >> >>> > > wrote:
>> >>>> > > >>> >> >>> > > > >
>> >>>> > > >>> >> >>> > > > > Hey all,
>> >>>> > > >>> >> >>> > > > >
>> >>>> > > >>> >> >>> > > > > I know the weekend is coming up, and for those
>> >>>> of us
>> >>>> > in
>> >>>> > > >>> the
>> >>>> > > >>> >> US,
>> >>>> > > >>> >> >>> next
>> >>>> > > >>> >> >>> > > week
>> >>>> > > >>> >> >>> > > > > is a bit of a slow holiday week. Would love to
>> >>>> get
>> >>>> > some
>> >>>> > > >>> >> feedback
>> >>>> > > >>> >> >>> from
>> >>>> > > >>> >> >>> > > > > everyone on this. The goal would ideally to
>> be to
>> >>>> > > >>> converge on
>> >>>> > > >>> >> >>> this
>> >>>> > > >>> >> >>> > and
>> >>>> > > >>> >> >>> > > > > eventually replace the existing Airflow UI
>> with
>> >>>> this
>> >>>> > > one.
>> >>>> > > >>> >> >>> > > > >
>> >>>> > > >>> >> >>> > > > > Cheers,
>> >>>> > > >>> >> >>> > > > > Chris
>> >>>> > > >>> >> >>> > > > >
>> >>>> > > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
>> >>>> > > joyg@wepay.com>
>> >>>> > > >>> >> wrote:
>> >>>> > > >>> >> >>> > > > >
>> >>>> > > >>> >> >>> > > > >> Hi guys.
>> >>>> > > >>> >> >>> > > > >>
>> >>>> > > >>> >> >>> > > > >> I've been working on moving airflow from
>> >>>> Flask-Admin
>> >>>> > to
>> >>>> > > >>> >> >>> > > Flask-AppBuilder
>> >>>> > > >>> >> >>> > > > >> for RBAC
>> >>>> > > >>> >> >>> > > > >> <https://cwiki.apache.org/
>> >>>> > confluence/display/AIRFLOW/
>> >>>> > > >>> >> >>> > > > Airflow+RBAC+proposal
>> >>>> > > >>> >> >>> > > > >>> ,
>> >>>> > > >>> >> >>> > > > >> check it out at
>> https://github.com/wepay/airfl
>> >>>> > > >>> ow-webserver.
>> >>>> > > >>> >> >>> > > > >>
>> >>>> > > >>> >> >>> > > > >> It's still a work-in-progress, but most
>> >>>> features you
>> >>>> > > see
>> >>>> > > >>> in
>> >>>> > > >>> >> the
>> >>>> > > >>> >> >>> > > > webserver
>> >>>> > > >>> >> >>> > > > >> UI today is available there. For those who
>> are
>> >>>> > > >>> interested in
>> >>>> > > >>> >> >>> RBAC,
>> >>>> > > >>> >> >>> > I'd
>> >>>> > > >>> >> >>> > > > love
>> >>>> > > >>> >> >>> > > > >> to get some early feedback in terms of the
>> >>>> following:
>> >>>> > > >>> >> >>> > > > >>
>> >>>> > > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any
>> bugs/regressions)
>> >>>> > > >>> >> >>> > > > >> - Setup issues
>> >>>> > > >>> >> >>> > > > >> - Ease of integration with third party auth
>> >>>> (i.e.
>> >>>> > LDAP,
>> >>>> > > >>> AD,
>> >>>> > > >>> >> >>> OAuth,
>> >>>> > > >>> >> >>> > > etc.)
>> >>>> > > >>> >> >>> > > > >> - Any other thoughts/concerns
>> >>>> > > >>> >> >>> > > > >>
>> >>>> > > >>> >> >>> > > > >> Thanks a lot!
>> >>>> > > >>> >> >>> > > > >>
>> >>>> > > >>> >> >>> > > > >> Cheers,
>> >>>> > > >>> >> >>> > > > >> Joy
>> >>>> > > >>> >> >>> > > > >>
>> >>>> > > >>> >> >>> > > >
>> >>>> > > >>> >> >>> > > >
>> >>>> > > >>> >> >>> > >
>> >>>> > > >>> >> >>> >
>> >>>> > > >>> >> >>>
>> >>>> > > >>> >> >>
>> >>>> > > >>> >> >>
>> >>>> > > >>> >> >
>> >>>> > > >>> >>
>> >>>> > > >>> >
>> >>>> > > >>> >
>> >>>> > > >>>
>> >>>> > > >>
>> >>>> > > >>
>> >>>> > > >
>> >>>> > >
>> >>>> >
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>>
>
>