You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Joy Gao <jo...@wepay.com> on 2017/11/17 21:44:22 UTC
RBAC Update
Hi guys.
I've been working on moving airflow from Flask-Admin to Flask-AppBuilder
for RBAC
<https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+RBAC+proposal>,
check it out at https://github.com/wepay/airflow-webserver.
It's still a work-in-progress, but most features you see in the webserver
UI today is available there. For those who are interested in RBAC, I'd love
to get some early feedback in terms of the following:
- New Flask-AppBuilder UI (any bugs/regressions)
- Setup issues
- Ease of integration with third party auth (i.e. LDAP, AD, OAuth, etc.)
- Any other thoughts/concerns
Thanks a lot!
Cheers,
Joy
Re: RBAC Update
Posted by Bolke de Bruin <bd...@gmail.com>.
1) Short answer: currently now. Long answer: I am working on a patch (albeit slowly) for FAB to do this. Unfortunately, getting into this level with FAB is a bit daunting and not as intuitive as I had hoped.
2) The url proved by Max is live so that is even better than screenshots.
> On 18 Nov 2017, at 06:58, Ruslan Dautkhanov <da...@gmail.com> wrote:
>
> That's awesome.
>
> 1. would it be possible to map an ldap group for example to view level
> access roles?
> 2. screenshots would be nice
>
> Thank you.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Fri, Nov 17, 2017 at 2:44 PM, Joy Gao <jo...@wepay.com> wrote:
>
>> Hi guys.
>>
>> I've been working on moving airflow from Flask-Admin to Flask-AppBuilder
>> for RBAC
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+RBAC+proposal
>>> ,
>> check it out at https://github.com/wepay/airflow-webserver.
>>
>> It's still a work-in-progress, but most features you see in the webserver
>> UI today is available there. For those who are interested in RBAC, I'd love
>> to get some early feedback in terms of the following:
>>
>> - New Flask-AppBuilder UI (any bugs/regressions)
>> - Setup issues
>> - Ease of integration with third party auth (i.e. LDAP, AD, OAuth, etc.)
>> - Any other thoughts/concerns
>>
>> Thanks a lot!
>>
>> Cheers,
>> Joy
>>
Re: RBAC Update
Posted by Maxime Beauchemin <ma...@gmail.com>.
http://airflow.westus.cloudapp.azure.com:8080/
login: admin
psw: admin
On Fri, Nov 17, 2017 at 9:58 PM, Ruslan Dautkhanov <da...@gmail.com>
wrote:
> That's awesome.
>
> 1. would it be possible to map an ldap group for example to view level
> access roles?
> 2. screenshots would be nice
>
> Thank you.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Fri, Nov 17, 2017 at 2:44 PM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hi guys.
> >
> > I've been working on moving airflow from Flask-Admin to Flask-AppBuilder
> > for RBAC
> > <https://cwiki.apache.org/confluence/display/AIRFLOW/
> Airflow+RBAC+proposal
> > >,
> > check it out at https://github.com/wepay/airflow-webserver.
> >
> > It's still a work-in-progress, but most features you see in the webserver
> > UI today is available there. For those who are interested in RBAC, I'd
> love
> > to get some early feedback in terms of the following:
> >
> > - New Flask-AppBuilder UI (any bugs/regressions)
> > - Setup issues
> > - Ease of integration with third party auth (i.e. LDAP, AD, OAuth, etc.)
> > - Any other thoughts/concerns
> >
> > Thanks a lot!
> >
> > Cheers,
> > Joy
> >
>
Re: RBAC Update
Posted by Ruslan Dautkhanov <da...@gmail.com>.
That's awesome.
1. would it be possible to map an ldap group for example to view level
access roles?
2. screenshots would be nice
Thank you.
--
Ruslan Dautkhanov
On Fri, Nov 17, 2017 at 2:44 PM, Joy Gao <jo...@wepay.com> wrote:
> Hi guys.
>
> I've been working on moving airflow from Flask-Admin to Flask-AppBuilder
> for RBAC
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+RBAC+proposal
> >,
> check it out at https://github.com/wepay/airflow-webserver.
>
> It's still a work-in-progress, but most features you see in the webserver
> UI today is available there. For those who are interested in RBAC, I'd love
> to get some early feedback in terms of the following:
>
> - New Flask-AppBuilder UI (any bugs/regressions)
> - Setup issues
> - Ease of integration with third party auth (i.e. LDAP, AD, OAuth, etc.)
> - Any other thoughts/concerns
>
> Thanks a lot!
>
> Cheers,
> Joy
>
Re: RBAC Update
Posted by Maxime Beauchemin <ma...@gmail.com>.
Good work! I'll try to take the time and setup a demo on Azure this weekend
(they offer free credit) so that people can play with it.
For folks new to Flask App Builder, it has a good permission / role model
allowing people to compose roles. It should be trivial for people to create
roles needed in their organizations. Roles can define which operations
(show, edit, delete) on which models (task instances, dag runs, variable,
...), which views (tree view, graph view, graphs, ...) and eventually DAGs
as well.
FAB also ships with a REST api for CRUD operations on all models, meaning
that the security model is applied to REST as well, so that if we re-write
the CLI on top of FAB's REST, `airflow list_dags` would only show the user
the DAGs they can see.
Max
On Fri, Nov 17, 2017 at 3:47 PM, Chris Riccomini <cr...@apache.org>
wrote:
> Hey all,
>
> I know the weekend is coming up, and for those of us in the US, next week
> is a bit of a slow holiday week. Would love to get some feedback from
> everyone on this. The goal would ideally to be to converge on this and
> eventually replace the existing Airflow UI with this one.
>
> Cheers,
> Chris
>
> On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hi guys.
> >
> > I've been working on moving airflow from Flask-Admin to Flask-AppBuilder
> > for RBAC
> > <https://cwiki.apache.org/confluence/display/AIRFLOW/
> Airflow+RBAC+proposal
> > >,
> > check it out at https://github.com/wepay/airflow-webserver.
> >
> > It's still a work-in-progress, but most features you see in the webserver
> > UI today is available there. For those who are interested in RBAC, I'd
> love
> > to get some early feedback in terms of the following:
> >
> > - New Flask-AppBuilder UI (any bugs/regressions)
> > - Setup issues
> > - Ease of integration with third party auth (i.e. LDAP, AD, OAuth, etc.)
> > - Any other thoughts/concerns
> >
> > Thanks a lot!
> >
> > Cheers,
> > Joy
> >
>
Re: RBAC Update
Posted by James Meickle <jm...@quantopian.com>.
To my mind, I would expect the MVP of per-DAG RBAC to include three
settings: viewing DAG state, executing or modifying DAGs, and viewing tasks
within the DAG (logs/code/details). For instance we would love to expose a
view of the production dataload state to our engineers, without exposing
production logs or allowing them to run/modify anything.
Creating these mappings would be somewhat painful to manage, but that seems
fine for an MVP. I would expect some future version to add grouping/tagging
to DAGs and to users, to say e.g. "users in group X can see DAGs in group Y
and modify DAGs in group Z."
On Fri, Mar 30, 2018 at 1:26 PM, Brian Greene <
brian@heisenbergwoodworking.com> wrote:
> I’d think we’d have privilege ‘can_view’ etc, and then a join table (priv)
> <-> (dagid) <-> (user/group). Then it’s a simple query to get the perms
> for a given dag (as you list In option 2 below).
>
> It also makes a “secure by default” easy - a lack of entries in that table
> for a dag can mean only “admin” access or some such.
>
> Then any dag can have any combo of permissions for any combo of users.
> Adding the groups option raises complexity around nesting, so maybe skip it
> for r1?
>
> My $.02
>
> Brian
>
> Sent from a device with less than stellar autocorrect
>
> > On Mar 29, 2018, at 10:27 AM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
> >
> > Hijacking the thread further here, any thoughts on how to breakdown per
> DAG
> > access?
> >
> > Tao & I are talking about introducing per-DAG permissions and one big
> > question is whether we'll need to support different operation-types at a
> > per-DAG level, which changes the way we need to model the perms.
> >
> > First [simpler] option is to introduce one perm per DAG. If you have
> access
> > to 5 DAGs, and you have `can_clear` and `can_run`, you'll have homogenous
> > rights on the DAGs you have access to.
> >
> > Second option is to have a breakdown per DAG. Meaning for each DAG we
> > create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...). So
> one
> > user could have modify on some DAGs, view on others, and other DAGs would
> > be invisible. This could be broken down further ({dag_id}_can_clear, ...)
> > but it gets hard to manage.
> >
> > Thoughts?
> >
> > Max
> >
> >> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fe...@gmail.com> wrote:
> >>
> >> Great work Joy. This is awesome! I am interested in helping out the per
> dag
> >> level access. Just created a ticket to check(AIRFLOW-2267). Let me
> know if
> >> you have any suggestions. I will share my proposal once I am ready.
> >>
> >>> On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
> >>>
> >>> Hey guys!
> >>>
> >>> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015>
> has
> >>> been merged to master. I'm looking forward to early adopters' feedback
> >> and
> >>> bug reports. I also hope to have more folks helping out with the RBAC
> UI,
> >>> especially with introducing DAG-Level access control, which is a
> feature
> >>> that a lot of people have been asking. If you are interested in helping
> >> out
> >>> with this effort, let's talk more!
> >>>
> >>> This commit will be in the 1.10.0 release, and we are going to maintain
> >>> both UIs simultaneously for a short period of time. Once RBAC UI is
> >> stable
> >>> and battle-tested, we will deprecate the old UI and eventually remove
> it
> >>> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to
> prevent
> >>> two UIs from forking into separate paths, as that would become very
> >>> difficult to maintain.
> >>>
> >>> Going forward while both UIs are up, if you are making a change to any
> >>> files in airflow/www/ (old UI), where applicable, please also make the
> >>> change to the airflow/www_rbac/ (new UI). If you rather not make
> changes
> >> in
> >>> both UIs, it is recommended that you only make the changes to the RBAC
> >> UI,
> >>> since that is the one we are maintaining in the long term.
> >>>
> >>> I'm excited that the RBAC UI will be able to bring additional security
> to
> >>> Airflow, and with FAB framework in place we can look into leveraging it
> >> for
> >>> a unified set of APIs used by both UI and CLI.
> >>>
> >>> Joy
> >>>
> >>>
> >>>
> >>>> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
> >>>>
> >>>> Hi folks,
> >>>>
> >>>> I have a PR <https://github.com/apache/incubator-airflow/pull/3015>
> >> out
> >>>> for the new UI. I've included instructions on how to test it out in
> the
> >>> PR
> >>>> description. Looking forward to your feedbacks.
> >>>>
> >>>> Cheers,
> >>>> Joy
> >>>>
> >>>>> On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
> >>>>>
> >>>>> Thanks for the background info. Would be really awesome for you to
> >> have
> >>>>> PyPi access :D I'll make the change to have Airflow Webserver's FAB
> >>>>> dependency pointing to my fork for the mean time.
> >>>>>
> >>>>> For folks who are interested in RBAC, I will be giving a talk/demo at
> >>> the Airflow
> >>>>> Meet-Up
> >>>>> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> >>> Meetup/events/244525050/>
> >>>>> next Monday. Happy to chat afterwards about it as well :)
> >>>>>
> >>>>> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> >>>>> maximebeauchemin@gmail.com> wrote:
> >>>>>
> >>>>>> A bit of related history here:
> >>>>>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> >>>>>>
> >>>>>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> >>>>>> maximebeauchemin@gmail.com> wrote:
> >>>>>>
> >>>>>>> Given I have merge rights on FAB I could probably do another round
> >> of
> >>>>>>> review and get your PRs through. I would really like to get the
> >> main
> >>>>>>> maintainer's input on things that touch the core (composite-key
> >>>>>> support) as
> >>>>>>> he might have concerns/intuitions that we can't know about.
> >>>>>>>
> >>>>>>> I do not have Pypi access though so I cannot push new releases
> >> out. I
> >>>>>>> could ask for that.
> >>>>>>>
> >>>>>>> I've threatened to fork the project before, that's always an
> >> option.
> >>>>>> I've
> >>>>>>> noticed his involvement is sporadic and comes in bursts.
> >>>>>>>
> >>>>>>> In the meantime, you can have the dependency in Airflow Webserver
> >>>>>> pointing
> >>>>>>> straight to your fork.
> >>>>>>>
> >>>>>>> Max
> >>>>>>>
> >>>>>>>> On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
> >>>>>>>>
> >>>>>>>> I just created a new webserver instance if you haven't gotten a
> >>>>>> chance to
> >>>>>>>> fiddle around with the new web UI and the RBAC configurations
> >>> (thanks
> >>>>>>>> Maxime for getting started with this earlier!):
> >>>>>>>>
> >>>>>>>> http://104.209.38.171:8080/
> >>>>>>>>
> >>>>>>>> Admin Account
> >>>>>>>> username: admin
> >>>>>>>> password: admin
> >>>>>>>>
> >>>>>>>> Read-Only Account
> >>>>>>>> username: viewer
> >>>>>>>> password: password
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi folks,
> >>>>>>>>>
> >>>>>>>>> Thanks for all the feedback regarding to the new Airflow
> >> Webserver
> >>>>>> UI
> >>>>>>>>> <https://github.com/wepay/airflow-webserver/>! I've been
> >> actively
> >>>>>>>>> addressing all the bugs that were raised on Github. So I want to
> >>>>>> take
> >>>>>>>> this
> >>>>>>>>> opportunity to discuss two issues coming up:
> >>>>>>>>>
> >>>>>>>>> The first issue is unaddressed PRs in FAB. If these PRs continue
> >>> to
> >>>>>> stay
> >>>>>>>>> unaddressed, RBAC is blocked from making further progress. If
> >> this
> >>>>>>>> continue
> >>>>>>>>> to be an issue, I'm inclined to fork FAB, even though it's not
> >>>>>>>> idealistic.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> - PR/631 <https://github.com/dpgaspar/F
> >>> lask-AppBuilder/pull/631>
> >>>>>>>> Binary
> >>>>>>>>> column support (merged, unreleased)
> >>>>>>>>> <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> >>>>>>>>> - PR/639 <https://github.com/dpgaspar/F
> >>> lask-AppBuilder/pull/639>
> >>>>>>>> Composite
> >>>>>>>>> primary key support (unmerged)
> >>>>>>>>> - PR/655 <https://github.com/dpgaspar/F
> >>> lask-AppBuilder/pull/655>
> >>>>>>>> Form
> >>>>>>>>> prefill support (unmerged)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> The second issue is an open question about the next step of
> >>> Airflow
> >>>>>>>>> Webserver itself. Here are the 3 potential directions we could
> >>>>>> take, and
> >>>>>>>>> I've added my thought on each.
> >>>>>>>>>
> >>>>>>>>> 1. Permanently keep Airflow Webserver as a separated package
> >> from
> >>>>>>>> Airflow,
> >>>>>>>>> and treat it as another UI option. Keep `www` in Airflow. Allow
> >>>>>>>> development
> >>>>>>>>> on both UIs.
> >>>>>>>>> *I'm not a fan of this. When there is an existing UI in Airflow,
> >>>>>> most
> >>>>>>>>> contributors would prefer to maintain the official version that
> >> is
> >>>>>>>>> installed out-of-the-box. **Having a second UI outside of
> >> Airflow
> >>>>>> will
> >>>>>>>>> make maintaining it very difficult, leading to an eventual death
> >>> of
> >>>>>> the
> >>>>>>>> new
> >>>>>>>>> UI :(*
> >>>>>>>>>
> >>>>>>>>> 2. Permanently keep Airflow Webserver as a separated package
> >> from
> >>>>>>>> Airflow,
> >>>>>>>>> but freeze all development on `www` and direct all future UI
> >>>>>>>> development
> >>>>>>>>> to Airflow Webserver, eventually removing `www` completely when
> >>>>>> Airflow
> >>>>>>>>> Webserver is stable.
> >>>>>>>>> *I'm not a fan of this either. First of all, the views and
> >> models
> >>>>>> are
> >>>>>>>>> tightly coupled in both old and new UI; until we have a
> >>> full-fledged
> >>>>>>>> REST
> >>>>>>>>> API to build the UI (and cli) on top of it, separating them to a
> >>>>>>>> separate
> >>>>>>>>> package now will potentially cause dependency issues and add
> >>>>>>>> complication
> >>>>>>>>> to our release cycle. **Secondly, **majority of Airflow users
> >> run
> >>>>>>>> Airflow
> >>>>>>>>> with the UI; it's one of Airflow's best features. Separating UI
> >>> out
> >>>>>> of
> >>>>>>>>> Airflow core will complicate setup and configuration, while
> >> making
> >>>>>>>> Airflow
> >>>>>>>>> core less complete.*
> >>>>>>>>>
> >>>>>>>>> 3. Merge Airflow Webserver back into Airflow as `www2`, freeze
> >> all
> >>>>>>>>> development on `www`, eventually removing `www` completely when
> >>>>>> `www2`
> >>>>>>>> is
> >>>>>>>>> stable.
> >>>>>>>>> *This makes the most sense to me. Airflow Webserver is developed
> >>>>>> with
> >>>>>>>> the
> >>>>>>>>> goal of feature parity to the current UI, plus additional RBAC
> >>>>>>>> capability,
> >>>>>>>>> in hope to replace the old UI completely. Yes, this means there
> >>>>>> will be
> >>>>>>>> a
> >>>>>>>>> short period of having to maintain two UIs, but once we freeze
> >>>>>>>> development
> >>>>>>>>> on www, it shouldn't be a concern for long.*
> >>>>>>>>>
> >>>>>>>>> I'd love to hear everyone's thoughts on this! I'm excited about
> >>>>>> bringing
> >>>>>>>>> RBAC to airflow and I hope it's something others will find
> >> useful
> >>> as
> >>>>>>>> well!
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>> Joy
> >>>>>>>>>
> >>>>>>>>> On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com>
> >> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thank you everyone for the active feedback so far, and thanks
> >> for
> >>>>>>>> setting
> >>>>>>>>>> up the demo Maxime!
> >>>>>>>>>>
> >>>>>>>>>> Going to work on pruning through the issues in the upcoming
> >> days.
> >>>>>>>>>>
> >>>>>>>>>> Fokko/Maxime, do you recall the SQLAlchemy Exception message
> >> so I
> >>>>>> can
> >>>>>>>>>> look into it? Otherwise I'll wait until it's down again =P
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>>
> >>>>>>>>>> Joy
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >>>>>>>>>> maximebeauchemin@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I just restarted it, not sure how long it will take to get in
> >> a
> >>>>>> bad
> >>>>>>>> state
> >>>>>>>>>>> again...
> >>>>>>>>>>>
> >>>>>>>>>>> Max
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> >>>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Good morning,
> >>>>>>>>>>>>
> >>>>>>>>>>>> The demo provided by Max is down, it throws a
> >>>>>> SQLAlchemyexception
> >>>>>>>> :'(
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> >>>>>> criccomini@apache.org>:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> @bolke, open issues on the Github repo, please.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> >>>>>>>> bdbruin@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Chris,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Do you want us to report bugs somewhere (I have
> >>> encountered
> >>>>>> a
> >>>>>>>>>>> few)? Or
> >>>>>>>>>>>>>> just generic user experiences posted here?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 18 Nov 2017, at 00:47, Chris Riccomini <
> >>>>>>>> criccomini@apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I know the weekend is coming up, and for those of us
> >> in
> >>>>>> the
> >>>>>>>> US,
> >>>>>>>>>>> next
> >>>>>>>>>>>>> week
> >>>>>>>>>>>>>>> is a bit of a slow holiday week. Would love to get
> >> some
> >>>>>>>> feedback
> >>>>>>>>>>> from
> >>>>>>>>>>>>>>> everyone on this. The goal would ideally to be to
> >>>>>> converge on
> >>>>>>>>>>> this
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> eventually replace the existing Airflow UI with this
> >>> one.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> >>> joyg@wepay.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi guys.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I've been working on moving airflow from Flask-Admin
> >> to
> >>>>>>>>>>>>> Flask-AppBuilder
> >>>>>>>>>>>>>>>> for RBAC
> >>>>>>>>>>>>>>>> <https://cwiki.apache.org/
> >> confluence/display/AIRFLOW/
> >>>>>>>>>>>>>> Airflow+RBAC+proposal
> >>>>>>>>>>>>>>>>> ,
> >>>>>>>>>>>>>>>> check it out at https://github.com/wepay/airfl
> >>>>>> ow-webserver.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> It's still a work-in-progress, but most features you
> >>> see
> >>>>>> in
> >>>>>>>> the
> >>>>>>>>>>>>>> webserver
> >>>>>>>>>>>>>>>> UI today is available there. For those who are
> >>>>>> interested in
> >>>>>>>>>>> RBAC,
> >>>>>>>>>>>> I'd
> >>>>>>>>>>>>>> love
> >>>>>>>>>>>>>>>> to get some early feedback in terms of the following:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - New Flask-AppBuilder UI (any bugs/regressions)
> >>>>>>>>>>>>>>>> - Setup issues
> >>>>>>>>>>>>>>>> - Ease of integration with third party auth (i.e.
> >> LDAP,
> >>>>>> AD,
> >>>>>>>>>>> OAuth,
> >>>>>>>>>>>>> etc.)
> >>>>>>>>>>>>>>>> - Any other thoughts/concerns
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks a lot!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>> Joy
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
Re: RBAC Update
Posted by Brian Greene <br...@heisenbergwoodworking.com>.
I’d think we’d have privilege ‘can_view’ etc, and then a join table (priv) <-> (dagid) <-> (user/group). Then it’s a simple query to get the perms for a given dag (as you list In option 2 below).
It also makes a “secure by default” easy - a lack of entries in that table for a dag can mean only “admin” access or some such.
Then any dag can have any combo of permissions for any combo of users. Adding the groups option raises complexity around nesting, so maybe skip it for r1?
My $.02
Brian
Sent from a device with less than stellar autocorrect
> On Mar 29, 2018, at 10:27 AM, Maxime Beauchemin <ma...@gmail.com> wrote:
>
> Hijacking the thread further here, any thoughts on how to breakdown per DAG
> access?
>
> Tao & I are talking about introducing per-DAG permissions and one big
> question is whether we'll need to support different operation-types at a
> per-DAG level, which changes the way we need to model the perms.
>
> First [simpler] option is to introduce one perm per DAG. If you have access
> to 5 DAGs, and you have `can_clear` and `can_run`, you'll have homogenous
> rights on the DAGs you have access to.
>
> Second option is to have a breakdown per DAG. Meaning for each DAG we
> create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...). So one
> user could have modify on some DAGs, view on others, and other DAGs would
> be invisible. This could be broken down further ({dag_id}_can_clear, ...)
> but it gets hard to manage.
>
> Thoughts?
>
> Max
>
>> On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fe...@gmail.com> wrote:
>>
>> Great work Joy. This is awesome! I am interested in helping out the per dag
>> level access. Just created a ticket to check(AIRFLOW-2267). Let me know if
>> you have any suggestions. I will share my proposal once I am ready.
>>
>>> On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
>>>
>>> Hey guys!
>>>
>>> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
>>> been merged to master. I'm looking forward to early adopters' feedback
>> and
>>> bug reports. I also hope to have more folks helping out with the RBAC UI,
>>> especially with introducing DAG-Level access control, which is a feature
>>> that a lot of people have been asking. If you are interested in helping
>> out
>>> with this effort, let's talk more!
>>>
>>> This commit will be in the 1.10.0 release, and we are going to maintain
>>> both UIs simultaneously for a short period of time. Once RBAC UI is
>> stable
>>> and battle-tested, we will deprecate the old UI and eventually remove it
>>> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
>>> two UIs from forking into separate paths, as that would become very
>>> difficult to maintain.
>>>
>>> Going forward while both UIs are up, if you are making a change to any
>>> files in airflow/www/ (old UI), where applicable, please also make the
>>> change to the airflow/www_rbac/ (new UI). If you rather not make changes
>> in
>>> both UIs, it is recommended that you only make the changes to the RBAC
>> UI,
>>> since that is the one we are maintaining in the long term.
>>>
>>> I'm excited that the RBAC UI will be able to bring additional security to
>>> Airflow, and with FAB framework in place we can look into leveraging it
>> for
>>> a unified set of APIs used by both UI and CLI.
>>>
>>> Joy
>>>
>>>
>>>
>>>> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
>>>>
>>>> Hi folks,
>>>>
>>>> I have a PR <https://github.com/apache/incubator-airflow/pull/3015>
>> out
>>>> for the new UI. I've included instructions on how to test it out in the
>>> PR
>>>> description. Looking forward to your feedbacks.
>>>>
>>>> Cheers,
>>>> Joy
>>>>
>>>>> On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
>>>>>
>>>>> Thanks for the background info. Would be really awesome for you to
>> have
>>>>> PyPi access :D I'll make the change to have Airflow Webserver's FAB
>>>>> dependency pointing to my fork for the mean time.
>>>>>
>>>>> For folks who are interested in RBAC, I will be giving a talk/demo at
>>> the Airflow
>>>>> Meet-Up
>>>>> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
>>> Meetup/events/244525050/>
>>>>> next Monday. Happy to chat afterwards about it as well :)
>>>>>
>>>>> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>
>>>>>> A bit of related history here:
>>>>>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
>>>>>>
>>>>>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>
>>>>>>> Given I have merge rights on FAB I could probably do another round
>> of
>>>>>>> review and get your PRs through. I would really like to get the
>> main
>>>>>>> maintainer's input on things that touch the core (composite-key
>>>>>> support) as
>>>>>>> he might have concerns/intuitions that we can't know about.
>>>>>>>
>>>>>>> I do not have Pypi access though so I cannot push new releases
>> out. I
>>>>>>> could ask for that.
>>>>>>>
>>>>>>> I've threatened to fork the project before, that's always an
>> option.
>>>>>> I've
>>>>>>> noticed his involvement is sporadic and comes in bursts.
>>>>>>>
>>>>>>> In the meantime, you can have the dependency in Airflow Webserver
>>>>>> pointing
>>>>>>> straight to your fork.
>>>>>>>
>>>>>>> Max
>>>>>>>
>>>>>>>> On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
>>>>>>>>
>>>>>>>> I just created a new webserver instance if you haven't gotten a
>>>>>> chance to
>>>>>>>> fiddle around with the new web UI and the RBAC configurations
>>> (thanks
>>>>>>>> Maxime for getting started with this earlier!):
>>>>>>>>
>>>>>>>> http://104.209.38.171:8080/
>>>>>>>>
>>>>>>>> Admin Account
>>>>>>>> username: admin
>>>>>>>> password: admin
>>>>>>>>
>>>>>>>> Read-Only Account
>>>>>>>> username: viewer
>>>>>>>> password: password
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> Thanks for all the feedback regarding to the new Airflow
>> Webserver
>>>>>> UI
>>>>>>>>> <https://github.com/wepay/airflow-webserver/>! I've been
>> actively
>>>>>>>>> addressing all the bugs that were raised on Github. So I want to
>>>>>> take
>>>>>>>> this
>>>>>>>>> opportunity to discuss two issues coming up:
>>>>>>>>>
>>>>>>>>> The first issue is unaddressed PRs in FAB. If these PRs continue
>>> to
>>>>>> stay
>>>>>>>>> unaddressed, RBAC is blocked from making further progress. If
>> this
>>>>>>>> continue
>>>>>>>>> to be an issue, I'm inclined to fork FAB, even though it's not
>>>>>>>> idealistic.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - PR/631 <https://github.com/dpgaspar/F
>>> lask-AppBuilder/pull/631>
>>>>>>>> Binary
>>>>>>>>> column support (merged, unreleased)
>>>>>>>>> <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>>>>>>>>> - PR/639 <https://github.com/dpgaspar/F
>>> lask-AppBuilder/pull/639>
>>>>>>>> Composite
>>>>>>>>> primary key support (unmerged)
>>>>>>>>> - PR/655 <https://github.com/dpgaspar/F
>>> lask-AppBuilder/pull/655>
>>>>>>>> Form
>>>>>>>>> prefill support (unmerged)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The second issue is an open question about the next step of
>>> Airflow
>>>>>>>>> Webserver itself. Here are the 3 potential directions we could
>>>>>> take, and
>>>>>>>>> I've added my thought on each.
>>>>>>>>>
>>>>>>>>> 1. Permanently keep Airflow Webserver as a separated package
>> from
>>>>>>>> Airflow,
>>>>>>>>> and treat it as another UI option. Keep `www` in Airflow. Allow
>>>>>>>> development
>>>>>>>>> on both UIs.
>>>>>>>>> *I'm not a fan of this. When there is an existing UI in Airflow,
>>>>>> most
>>>>>>>>> contributors would prefer to maintain the official version that
>> is
>>>>>>>>> installed out-of-the-box. **Having a second UI outside of
>> Airflow
>>>>>> will
>>>>>>>>> make maintaining it very difficult, leading to an eventual death
>>> of
>>>>>> the
>>>>>>>> new
>>>>>>>>> UI :(*
>>>>>>>>>
>>>>>>>>> 2. Permanently keep Airflow Webserver as a separated package
>> from
>>>>>>>> Airflow,
>>>>>>>>> but freeze all development on `www` and direct all future UI
>>>>>>>> development
>>>>>>>>> to Airflow Webserver, eventually removing `www` completely when
>>>>>> Airflow
>>>>>>>>> Webserver is stable.
>>>>>>>>> *I'm not a fan of this either. First of all, the views and
>> models
>>>>>> are
>>>>>>>>> tightly coupled in both old and new UI; until we have a
>>> full-fledged
>>>>>>>> REST
>>>>>>>>> API to build the UI (and cli) on top of it, separating them to a
>>>>>>>> separate
>>>>>>>>> package now will potentially cause dependency issues and add
>>>>>>>> complication
>>>>>>>>> to our release cycle. **Secondly, **majority of Airflow users
>> run
>>>>>>>> Airflow
>>>>>>>>> with the UI; it's one of Airflow's best features. Separating UI
>>> out
>>>>>> of
>>>>>>>>> Airflow core will complicate setup and configuration, while
>> making
>>>>>>>> Airflow
>>>>>>>>> core less complete.*
>>>>>>>>>
>>>>>>>>> 3. Merge Airflow Webserver back into Airflow as `www2`, freeze
>> all
>>>>>>>>> development on `www`, eventually removing `www` completely when
>>>>>> `www2`
>>>>>>>> is
>>>>>>>>> stable.
>>>>>>>>> *This makes the most sense to me. Airflow Webserver is developed
>>>>>> with
>>>>>>>> the
>>>>>>>>> goal of feature parity to the current UI, plus additional RBAC
>>>>>>>> capability,
>>>>>>>>> in hope to replace the old UI completely. Yes, this means there
>>>>>> will be
>>>>>>>> a
>>>>>>>>> short period of having to maintain two UIs, but once we freeze
>>>>>>>> development
>>>>>>>>> on www, it shouldn't be a concern for long.*
>>>>>>>>>
>>>>>>>>> I'd love to hear everyone's thoughts on this! I'm excited about
>>>>>> bringing
>>>>>>>>> RBAC to airflow and I hope it's something others will find
>> useful
>>> as
>>>>>>>> well!
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Joy
>>>>>>>>>
>>>>>>>>> On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com>
>> wrote:
>>>>>>>>>
>>>>>>>>>> Thank you everyone for the active feedback so far, and thanks
>> for
>>>>>>>> setting
>>>>>>>>>> up the demo Maxime!
>>>>>>>>>>
>>>>>>>>>> Going to work on pruning through the issues in the upcoming
>> days.
>>>>>>>>>>
>>>>>>>>>> Fokko/Maxime, do you recall the SQLAlchemy Exception message
>> so I
>>>>>> can
>>>>>>>>>> look into it? Otherwise I'll wait until it's down again =P
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Joy
>>>>>>>>>>
>>>>>>>>>> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
>>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I just restarted it, not sure how long it will take to get in
>> a
>>>>>> bad
>>>>>>>> state
>>>>>>>>>>> again...
>>>>>>>>>>>
>>>>>>>>>>> Max
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
>>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Good morning,
>>>>>>>>>>>>
>>>>>>>>>>>> The demo provided by Max is down, it throws a
>>>>>> SQLAlchemyexception
>>>>>>>> :'(
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>>
>>>>>>>>>>>> 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
>>>>>> criccomini@apache.org>:
>>>>>>>>>>>>
>>>>>>>>>>>>> @bolke, open issues on the Github repo, please.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
>>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Chris,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you want us to report bugs somewhere (I have
>>> encountered
>>>>>> a
>>>>>>>>>>> few)? Or
>>>>>>>>>>>>>> just generic user experiences posted here?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 18 Nov 2017, at 00:47, Chris Riccomini <
>>>>>>>> criccomini@apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I know the weekend is coming up, and for those of us
>> in
>>>>>> the
>>>>>>>> US,
>>>>>>>>>>> next
>>>>>>>>>>>>> week
>>>>>>>>>>>>>>> is a bit of a slow holiday week. Would love to get
>> some
>>>>>>>> feedback
>>>>>>>>>>> from
>>>>>>>>>>>>>>> everyone on this. The goal would ideally to be to
>>>>>> converge on
>>>>>>>>>>> this
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> eventually replace the existing Airflow UI with this
>>> one.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
>>> joyg@wepay.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi guys.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've been working on moving airflow from Flask-Admin
>> to
>>>>>>>>>>>>> Flask-AppBuilder
>>>>>>>>>>>>>>>> for RBAC
>>>>>>>>>>>>>>>> <https://cwiki.apache.org/
>> confluence/display/AIRFLOW/
>>>>>>>>>>>>>> Airflow+RBAC+proposal
>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>> check it out at https://github.com/wepay/airfl
>>>>>> ow-webserver.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's still a work-in-progress, but most features you
>>> see
>>>>>> in
>>>>>>>> the
>>>>>>>>>>>>>> webserver
>>>>>>>>>>>>>>>> UI today is available there. For those who are
>>>>>> interested in
>>>>>>>>>>> RBAC,
>>>>>>>>>>>> I'd
>>>>>>>>>>>>>> love
>>>>>>>>>>>>>>>> to get some early feedback in terms of the following:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - New Flask-AppBuilder UI (any bugs/regressions)
>>>>>>>>>>>>>>>> - Setup issues
>>>>>>>>>>>>>>>> - Ease of integration with third party auth (i.e.
>> LDAP,
>>>>>> AD,
>>>>>>>>>>> OAuth,
>>>>>>>>>>>>> etc.)
>>>>>>>>>>>>>>>> - Any other thoughts/concerns
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks a lot!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Joy
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: RBAC Update
Posted by Maxime Beauchemin <ma...@gmail.com>.
Hijacking the thread further here, any thoughts on how to breakdown per DAG
access?
Tao & I are talking about introducing per-DAG permissions and one big
question is whether we'll need to support different operation-types at a
per-DAG level, which changes the way we need to model the perms.
First [simpler] option is to introduce one perm per DAG. If you have access
to 5 DAGs, and you have `can_clear` and `can_run`, you'll have homogenous
rights on the DAGs you have access to.
Second option is to have a breakdown per DAG. Meaning for each DAG we
create a set of perms ({dag_id}_can_view, {dag_id}_can_modify, ...). So one
user could have modify on some DAGs, view on others, and other DAGs would
be invisible. This could be broken down further ({dag_id}_can_clear, ...)
but it gets hard to manage.
Thoughts?
Max
On Wed, Mar 28, 2018 at 10:02 PM, Tao Feng <fe...@gmail.com> wrote:
> Great work Joy. This is awesome! I am interested in helping out the per dag
> level access. Just created a ticket to check(AIRFLOW-2267). Let me know if
> you have any suggestions. I will share my proposal once I am ready.
>
> On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hey guys!
> >
> > The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
> > been merged to master. I'm looking forward to early adopters' feedback
> and
> > bug reports. I also hope to have more folks helping out with the RBAC UI,
> > especially with introducing DAG-Level access control, which is a feature
> > that a lot of people have been asking. If you are interested in helping
> out
> > with this effort, let's talk more!
> >
> > This commit will be in the 1.10.0 release, and we are going to maintain
> > both UIs simultaneously for a short period of time. Once RBAC UI is
> stable
> > and battle-tested, we will deprecate the old UI and eventually remove it
> > from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
> > two UIs from forking into separate paths, as that would become very
> > difficult to maintain.
> >
> > Going forward while both UIs are up, if you are making a change to any
> > files in airflow/www/ (old UI), where applicable, please also make the
> > change to the airflow/www_rbac/ (new UI). If you rather not make changes
> in
> > both UIs, it is recommended that you only make the changes to the RBAC
> UI,
> > since that is the one we are maintaining in the long term.
> >
> > I'm excited that the RBAC UI will be able to bring additional security to
> > Airflow, and with FAB framework in place we can look into leveraging it
> for
> > a unified set of APIs used by both UI and CLI.
> >
> > Joy
> >
> >
> >
> > On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
> >
> > > Hi folks,
> > >
> > > I have a PR <https://github.com/apache/incubator-airflow/pull/3015>
> out
> > > for the new UI. I've included instructions on how to test it out in the
> > PR
> > > description. Looking forward to your feedbacks.
> > >
> > > Cheers,
> > > Joy
> > >
> > > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
> > >
> > >> Thanks for the background info. Would be really awesome for you to
> have
> > >> PyPi access :D I'll make the change to have Airflow Webserver's FAB
> > >> dependency pointing to my fork for the mean time.
> > >>
> > >> For folks who are interested in RBAC, I will be giving a talk/demo at
> > the Airflow
> > >> Meet-Up
> > >> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> > Meetup/events/244525050/>
> > >> next Monday. Happy to chat afterwards about it as well :)
> > >>
> > >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> > >> maximebeauchemin@gmail.com> wrote:
> > >>
> > >>> A bit of related history here:
> > >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> > >>>
> > >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> > >>> maximebeauchemin@gmail.com> wrote:
> > >>>
> > >>> > Given I have merge rights on FAB I could probably do another round
> of
> > >>> > review and get your PRs through. I would really like to get the
> main
> > >>> > maintainer's input on things that touch the core (composite-key
> > >>> support) as
> > >>> > he might have concerns/intuitions that we can't know about.
> > >>> >
> > >>> > I do not have Pypi access though so I cannot push new releases
> out. I
> > >>> > could ask for that.
> > >>> >
> > >>> > I've threatened to fork the project before, that's always an
> option.
> > >>> I've
> > >>> > noticed his involvement is sporadic and comes in bursts.
> > >>> >
> > >>> > In the meantime, you can have the dependency in Airflow Webserver
> > >>> pointing
> > >>> > straight to your fork.
> > >>> >
> > >>> > Max
> > >>> >
> > >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
> > >>> >
> > >>> >> I just created a new webserver instance if you haven't gotten a
> > >>> chance to
> > >>> >> fiddle around with the new web UI and the RBAC configurations
> > (thanks
> > >>> >> Maxime for getting started with this earlier!):
> > >>> >>
> > >>> >> http://104.209.38.171:8080/
> > >>> >>
> > >>> >> Admin Account
> > >>> >> username: admin
> > >>> >> password: admin
> > >>> >>
> > >>> >> Read-Only Account
> > >>> >> username: viewer
> > >>> >> password: password
> > >>> >>
> > >>> >>
> > >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
> > >>> >>
> > >>> >> > Hi folks,
> > >>> >> >
> > >>> >> > Thanks for all the feedback regarding to the new Airflow
> Webserver
> > >>> UI
> > >>> >> > <https://github.com/wepay/airflow-webserver/>! I've been
> actively
> > >>> >> > addressing all the bugs that were raised on Github. So I want to
> > >>> take
> > >>> >> this
> > >>> >> > opportunity to discuss two issues coming up:
> > >>> >> >
> > >>> >> > The first issue is unaddressed PRs in FAB. If these PRs continue
> > to
> > >>> stay
> > >>> >> > unaddressed, RBAC is blocked from making further progress. If
> this
> > >>> >> continue
> > >>> >> > to be an issue, I'm inclined to fork FAB, even though it's not
> > >>> >> idealistic.
> > >>> >> >
> > >>> >> >
> > >>> >> > - PR/631 <https://github.com/dpgaspar/F
> > lask-AppBuilder/pull/631>
> > >>> >> Binary
> > >>> >> > column support (merged, unreleased)
> > >>> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> > >>> >> > - PR/639 <https://github.com/dpgaspar/F
> > lask-AppBuilder/pull/639>
> > >>> >> Composite
> > >>> >> > primary key support (unmerged)
> > >>> >> > - PR/655 <https://github.com/dpgaspar/F
> > lask-AppBuilder/pull/655>
> > >>> >> Form
> > >>> >> > prefill support (unmerged)
> > >>> >> >
> > >>> >> >
> > >>> >> > The second issue is an open question about the next step of
> > Airflow
> > >>> >> > Webserver itself. Here are the 3 potential directions we could
> > >>> take, and
> > >>> >> > I've added my thought on each.
> > >>> >> >
> > >>> >> > 1. Permanently keep Airflow Webserver as a separated package
> from
> > >>> >> Airflow,
> > >>> >> > and treat it as another UI option. Keep `www` in Airflow. Allow
> > >>> >> development
> > >>> >> > on both UIs.
> > >>> >> > *I'm not a fan of this. When there is an existing UI in Airflow,
> > >>> most
> > >>> >> > contributors would prefer to maintain the official version that
> is
> > >>> >> > installed out-of-the-box. **Having a second UI outside of
> Airflow
> > >>> will
> > >>> >> > make maintaining it very difficult, leading to an eventual death
> > of
> > >>> the
> > >>> >> new
> > >>> >> > UI :(*
> > >>> >> >
> > >>> >> > 2. Permanently keep Airflow Webserver as a separated package
> from
> > >>> >> Airflow,
> > >>> >> > but freeze all development on `www` and direct all future UI
> > >>> >> development
> > >>> >> > to Airflow Webserver, eventually removing `www` completely when
> > >>> Airflow
> > >>> >> > Webserver is stable.
> > >>> >> > *I'm not a fan of this either. First of all, the views and
> models
> > >>> are
> > >>> >> > tightly coupled in both old and new UI; until we have a
> > full-fledged
> > >>> >> REST
> > >>> >> > API to build the UI (and cli) on top of it, separating them to a
> > >>> >> separate
> > >>> >> > package now will potentially cause dependency issues and add
> > >>> >> complication
> > >>> >> > to our release cycle. **Secondly, **majority of Airflow users
> run
> > >>> >> Airflow
> > >>> >> > with the UI; it's one of Airflow's best features. Separating UI
> > out
> > >>> of
> > >>> >> > Airflow core will complicate setup and configuration, while
> making
> > >>> >> Airflow
> > >>> >> > core less complete.*
> > >>> >> >
> > >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze
> all
> > >>> >> > development on `www`, eventually removing `www` completely when
> > >>> `www2`
> > >>> >> is
> > >>> >> > stable.
> > >>> >> > *This makes the most sense to me. Airflow Webserver is developed
> > >>> with
> > >>> >> the
> > >>> >> > goal of feature parity to the current UI, plus additional RBAC
> > >>> >> capability,
> > >>> >> > in hope to replace the old UI completely. Yes, this means there
> > >>> will be
> > >>> >> a
> > >>> >> > short period of having to maintain two UIs, but once we freeze
> > >>> >> development
> > >>> >> > on www, it shouldn't be a concern for long.*
> > >>> >> >
> > >>> >> > I'd love to hear everyone's thoughts on this! I'm excited about
> > >>> bringing
> > >>> >> > RBAC to airflow and I hope it's something others will find
> useful
> > as
> > >>> >> well!
> > >>> >> >
> > >>> >> > Cheers,
> > >>> >> > Joy
> > >>> >> >
> > >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com>
> wrote:
> > >>> >> >
> > >>> >> >> Thank you everyone for the active feedback so far, and thanks
> for
> > >>> >> setting
> > >>> >> >> up the demo Maxime!
> > >>> >> >>
> > >>> >> >> Going to work on pruning through the issues in the upcoming
> days.
> > >>> >> >>
> > >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message
> so I
> > >>> can
> > >>> >> >> look into it? Otherwise I'll wait until it's down again =P
> > >>> >> >>
> > >>> >> >> Cheers,
> > >>> >> >>
> > >>> >> >> Joy
> > >>> >> >>
> > >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> > >>> >> >> maximebeauchemin@gmail.com> wrote:
> > >>> >> >>
> > >>> >> >>> I just restarted it, not sure how long it will take to get in
> a
> > >>> bad
> > >>> >> state
> > >>> >> >>> again...
> > >>> >> >>>
> > >>> >> >>> Max
> > >>> >> >>>
> > >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> > >>> >> <fokko@driesprong.frl
> > >>> >> >>> >
> > >>> >> >>> wrote:
> > >>> >> >>>
> > >>> >> >>> > Good morning,
> > >>> >> >>> >
> > >>> >> >>> > The demo provided by Max is down, it throws a
> > >>> SQLAlchemyexception
> > >>> >> :'(
> > >>> >> >>> >
> > >>> >> >>> > Cheers, Fokko
> > >>> >> >>> >
> > >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> > >>> criccomini@apache.org>:
> > >>> >> >>> >
> > >>> >> >>> > > @bolke, open issues on the Github repo, please.
> > >>> >> >>> > >
> > >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> > >>> >> bdbruin@gmail.com>
> > >>> >> >>> > > wrote:
> > >>> >> >>> > >
> > >>> >> >>> > > > Chris,
> > >>> >> >>> > > >
> > >>> >> >>> > > > Do you want us to report bugs somewhere (I have
> > encountered
> > >>> a
> > >>> >> >>> few)? Or
> > >>> >> >>> > > > just generic user experiences posted here?
> > >>> >> >>> > > >
> > >>> >> >>> > > > Cheers
> > >>> >> >>> > > > Bolke
> > >>> >> >>> > > >
> > >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> > >>> >> criccomini@apache.org
> > >>> >> >>> >
> > >>> >> >>> > > wrote:
> > >>> >> >>> > > > >
> > >>> >> >>> > > > > Hey all,
> > >>> >> >>> > > > >
> > >>> >> >>> > > > > I know the weekend is coming up, and for those of us
> in
> > >>> the
> > >>> >> US,
> > >>> >> >>> next
> > >>> >> >>> > > week
> > >>> >> >>> > > > > is a bit of a slow holiday week. Would love to get
> some
> > >>> >> feedback
> > >>> >> >>> from
> > >>> >> >>> > > > > everyone on this. The goal would ideally to be to
> > >>> converge on
> > >>> >> >>> this
> > >>> >> >>> > and
> > >>> >> >>> > > > > eventually replace the existing Airflow UI with this
> > one.
> > >>> >> >>> > > > >
> > >>> >> >>> > > > > Cheers,
> > >>> >> >>> > > > > Chris
> > >>> >> >>> > > > >
> > >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> > joyg@wepay.com>
> > >>> >> wrote:
> > >>> >> >>> > > > >
> > >>> >> >>> > > > >> Hi guys.
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> I've been working on moving airflow from Flask-Admin
> to
> > >>> >> >>> > > Flask-AppBuilder
> > >>> >> >>> > > > >> for RBAC
> > >>> >> >>> > > > >> <https://cwiki.apache.org/
> confluence/display/AIRFLOW/
> > >>> >> >>> > > > Airflow+RBAC+proposal
> > >>> >> >>> > > > >>> ,
> > >>> >> >>> > > > >> check it out at https://github.com/wepay/airfl
> > >>> ow-webserver.
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> It's still a work-in-progress, but most features you
> > see
> > >>> in
> > >>> >> the
> > >>> >> >>> > > > webserver
> > >>> >> >>> > > > >> UI today is available there. For those who are
> > >>> interested in
> > >>> >> >>> RBAC,
> > >>> >> >>> > I'd
> > >>> >> >>> > > > love
> > >>> >> >>> > > > >> to get some early feedback in terms of the following:
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> > >>> >> >>> > > > >> - Setup issues
> > >>> >> >>> > > > >> - Ease of integration with third party auth (i.e.
> LDAP,
> > >>> AD,
> > >>> >> >>> OAuth,
> > >>> >> >>> > > etc.)
> > >>> >> >>> > > > >> - Any other thoughts/concerns
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> Thanks a lot!
> > >>> >> >>> > > > >>
> > >>> >> >>> > > > >> Cheers,
> > >>> >> >>> > > > >> Joy
> > >>> >> >>> > > > >>
> > >>> >> >>> > > >
> > >>> >> >>> > > >
> > >>> >> >>> > >
> > >>> >> >>> >
> > >>> >> >>>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >
> > >>> >>
> > >>> >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: RBAC Update
Posted by Tao Feng <fe...@gmail.com>.
Great work Joy. This is awesome! I am interested in helping out the per dag
level access. Just created a ticket to check(AIRFLOW-2267). Let me know if
you have any suggestions. I will share my proposal once I am ready.
On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
> Hey guys!
>
> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
> been merged to master. I'm looking forward to early adopters' feedback and
> bug reports. I also hope to have more folks helping out with the RBAC UI,
> especially with introducing DAG-Level access control, which is a feature
> that a lot of people have been asking. If you are interested in helping out
> with this effort, let's talk more!
>
> This commit will be in the 1.10.0 release, and we are going to maintain
> both UIs simultaneously for a short period of time. Once RBAC UI is stable
> and battle-tested, we will deprecate the old UI and eventually remove it
> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
> two UIs from forking into separate paths, as that would become very
> difficult to maintain.
>
> Going forward while both UIs are up, if you are making a change to any
> files in airflow/www/ (old UI), where applicable, please also make the
> change to the airflow/www_rbac/ (new UI). If you rather not make changes in
> both UIs, it is recommended that you only make the changes to the RBAC UI,
> since that is the one we are maintaining in the long term.
>
> I'm excited that the RBAC UI will be able to bring additional security to
> Airflow, and with FAB framework in place we can look into leveraging it for
> a unified set of APIs used by both UI and CLI.
>
> Joy
>
>
>
> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hi folks,
> >
> > I have a PR <https://github.com/apache/incubator-airflow/pull/3015> out
> > for the new UI. I've included instructions on how to test it out in the
> PR
> > description. Looking forward to your feedbacks.
> >
> > Cheers,
> > Joy
> >
> > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> >> Thanks for the background info. Would be really awesome for you to have
> >> PyPi access :D I'll make the change to have Airflow Webserver's FAB
> >> dependency pointing to my fork for the mean time.
> >>
> >> For folks who are interested in RBAC, I will be giving a talk/demo at
> the Airflow
> >> Meet-Up
> >> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-
> Meetup/events/244525050/>
> >> next Monday. Happy to chat afterwards about it as well :)
> >>
> >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> >> maximebeauchemin@gmail.com> wrote:
> >>
> >>> A bit of related history here:
> >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> >>>
> >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> >>> maximebeauchemin@gmail.com> wrote:
> >>>
> >>> > Given I have merge rights on FAB I could probably do another round of
> >>> > review and get your PRs through. I would really like to get the main
> >>> > maintainer's input on things that touch the core (composite-key
> >>> support) as
> >>> > he might have concerns/intuitions that we can't know about.
> >>> >
> >>> > I do not have Pypi access though so I cannot push new releases out. I
> >>> > could ask for that.
> >>> >
> >>> > I've threatened to fork the project before, that's always an option.
> >>> I've
> >>> > noticed his involvement is sporadic and comes in bursts.
> >>> >
> >>> > In the meantime, you can have the dependency in Airflow Webserver
> >>> pointing
> >>> > straight to your fork.
> >>> >
> >>> > Max
> >>> >
> >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
> >>> >
> >>> >> I just created a new webserver instance if you haven't gotten a
> >>> chance to
> >>> >> fiddle around with the new web UI and the RBAC configurations
> (thanks
> >>> >> Maxime for getting started with this earlier!):
> >>> >>
> >>> >> http://104.209.38.171:8080/
> >>> >>
> >>> >> Admin Account
> >>> >> username: admin
> >>> >> password: admin
> >>> >>
> >>> >> Read-Only Account
> >>> >> username: viewer
> >>> >> password: password
> >>> >>
> >>> >>
> >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
> >>> >>
> >>> >> > Hi folks,
> >>> >> >
> >>> >> > Thanks for all the feedback regarding to the new Airflow Webserver
> >>> UI
> >>> >> > <https://github.com/wepay/airflow-webserver/>! I've been actively
> >>> >> > addressing all the bugs that were raised on Github. So I want to
> >>> take
> >>> >> this
> >>> >> > opportunity to discuss two issues coming up:
> >>> >> >
> >>> >> > The first issue is unaddressed PRs in FAB. If these PRs continue
> to
> >>> stay
> >>> >> > unaddressed, RBAC is blocked from making further progress. If this
> >>> >> continue
> >>> >> > to be an issue, I'm inclined to fork FAB, even though it's not
> >>> >> idealistic.
> >>> >> >
> >>> >> >
> >>> >> > - PR/631 <https://github.com/dpgaspar/F
> lask-AppBuilder/pull/631>
> >>> >> Binary
> >>> >> > column support (merged, unreleased)
> >>> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> >>> >> > - PR/639 <https://github.com/dpgaspar/F
> lask-AppBuilder/pull/639>
> >>> >> Composite
> >>> >> > primary key support (unmerged)
> >>> >> > - PR/655 <https://github.com/dpgaspar/F
> lask-AppBuilder/pull/655>
> >>> >> Form
> >>> >> > prefill support (unmerged)
> >>> >> >
> >>> >> >
> >>> >> > The second issue is an open question about the next step of
> Airflow
> >>> >> > Webserver itself. Here are the 3 potential directions we could
> >>> take, and
> >>> >> > I've added my thought on each.
> >>> >> >
> >>> >> > 1. Permanently keep Airflow Webserver as a separated package from
> >>> >> Airflow,
> >>> >> > and treat it as another UI option. Keep `www` in Airflow. Allow
> >>> >> development
> >>> >> > on both UIs.
> >>> >> > *I'm not a fan of this. When there is an existing UI in Airflow,
> >>> most
> >>> >> > contributors would prefer to maintain the official version that is
> >>> >> > installed out-of-the-box. **Having a second UI outside of Airflow
> >>> will
> >>> >> > make maintaining it very difficult, leading to an eventual death
> of
> >>> the
> >>> >> new
> >>> >> > UI :(*
> >>> >> >
> >>> >> > 2. Permanently keep Airflow Webserver as a separated package from
> >>> >> Airflow,
> >>> >> > but freeze all development on `www` and direct all future UI
> >>> >> development
> >>> >> > to Airflow Webserver, eventually removing `www` completely when
> >>> Airflow
> >>> >> > Webserver is stable.
> >>> >> > *I'm not a fan of this either. First of all, the views and models
> >>> are
> >>> >> > tightly coupled in both old and new UI; until we have a
> full-fledged
> >>> >> REST
> >>> >> > API to build the UI (and cli) on top of it, separating them to a
> >>> >> separate
> >>> >> > package now will potentially cause dependency issues and add
> >>> >> complication
> >>> >> > to our release cycle. **Secondly, **majority of Airflow users run
> >>> >> Airflow
> >>> >> > with the UI; it's one of Airflow's best features. Separating UI
> out
> >>> of
> >>> >> > Airflow core will complicate setup and configuration, while making
> >>> >> Airflow
> >>> >> > core less complete.*
> >>> >> >
> >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
> >>> >> > development on `www`, eventually removing `www` completely when
> >>> `www2`
> >>> >> is
> >>> >> > stable.
> >>> >> > *This makes the most sense to me. Airflow Webserver is developed
> >>> with
> >>> >> the
> >>> >> > goal of feature parity to the current UI, plus additional RBAC
> >>> >> capability,
> >>> >> > in hope to replace the old UI completely. Yes, this means there
> >>> will be
> >>> >> a
> >>> >> > short period of having to maintain two UIs, but once we freeze
> >>> >> development
> >>> >> > on www, it shouldn't be a concern for long.*
> >>> >> >
> >>> >> > I'd love to hear everyone's thoughts on this! I'm excited about
> >>> bringing
> >>> >> > RBAC to airflow and I hope it's something others will find useful
> as
> >>> >> well!
> >>> >> >
> >>> >> > Cheers,
> >>> >> > Joy
> >>> >> >
> >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
> >>> >> >
> >>> >> >> Thank you everyone for the active feedback so far, and thanks for
> >>> >> setting
> >>> >> >> up the demo Maxime!
> >>> >> >>
> >>> >> >> Going to work on pruning through the issues in the upcoming days.
> >>> >> >>
> >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I
> >>> can
> >>> >> >> look into it? Otherwise I'll wait until it's down again =P
> >>> >> >>
> >>> >> >> Cheers,
> >>> >> >>
> >>> >> >> Joy
> >>> >> >>
> >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >>> >> >> maximebeauchemin@gmail.com> wrote:
> >>> >> >>
> >>> >> >>> I just restarted it, not sure how long it will take to get in a
> >>> bad
> >>> >> state
> >>> >> >>> again...
> >>> >> >>>
> >>> >> >>> Max
> >>> >> >>>
> >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> >>> >> <fokko@driesprong.frl
> >>> >> >>> >
> >>> >> >>> wrote:
> >>> >> >>>
> >>> >> >>> > Good morning,
> >>> >> >>> >
> >>> >> >>> > The demo provided by Max is down, it throws a
> >>> SQLAlchemyexception
> >>> >> :'(
> >>> >> >>> >
> >>> >> >>> > Cheers, Fokko
> >>> >> >>> >
> >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> >>> criccomini@apache.org>:
> >>> >> >>> >
> >>> >> >>> > > @bolke, open issues on the Github repo, please.
> >>> >> >>> > >
> >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> >>> >> bdbruin@gmail.com>
> >>> >> >>> > > wrote:
> >>> >> >>> > >
> >>> >> >>> > > > Chris,
> >>> >> >>> > > >
> >>> >> >>> > > > Do you want us to report bugs somewhere (I have
> encountered
> >>> a
> >>> >> >>> few)? Or
> >>> >> >>> > > > just generic user experiences posted here?
> >>> >> >>> > > >
> >>> >> >>> > > > Cheers
> >>> >> >>> > > > Bolke
> >>> >> >>> > > >
> >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> >>> >> criccomini@apache.org
> >>> >> >>> >
> >>> >> >>> > > wrote:
> >>> >> >>> > > > >
> >>> >> >>> > > > > Hey all,
> >>> >> >>> > > > >
> >>> >> >>> > > > > I know the weekend is coming up, and for those of us in
> >>> the
> >>> >> US,
> >>> >> >>> next
> >>> >> >>> > > week
> >>> >> >>> > > > > is a bit of a slow holiday week. Would love to get some
> >>> >> feedback
> >>> >> >>> from
> >>> >> >>> > > > > everyone on this. The goal would ideally to be to
> >>> converge on
> >>> >> >>> this
> >>> >> >>> > and
> >>> >> >>> > > > > eventually replace the existing Airflow UI with this
> one.
> >>> >> >>> > > > >
> >>> >> >>> > > > > Cheers,
> >>> >> >>> > > > > Chris
> >>> >> >>> > > > >
> >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> joyg@wepay.com>
> >>> >> wrote:
> >>> >> >>> > > > >
> >>> >> >>> > > > >> Hi guys.
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> I've been working on moving airflow from Flask-Admin to
> >>> >> >>> > > Flask-AppBuilder
> >>> >> >>> > > > >> for RBAC
> >>> >> >>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> >>> >> >>> > > > Airflow+RBAC+proposal
> >>> >> >>> > > > >>> ,
> >>> >> >>> > > > >> check it out at https://github.com/wepay/airfl
> >>> ow-webserver.
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> It's still a work-in-progress, but most features you
> see
> >>> in
> >>> >> the
> >>> >> >>> > > > webserver
> >>> >> >>> > > > >> UI today is available there. For those who are
> >>> interested in
> >>> >> >>> RBAC,
> >>> >> >>> > I'd
> >>> >> >>> > > > love
> >>> >> >>> > > > >> to get some early feedback in terms of the following:
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> >>> >> >>> > > > >> - Setup issues
> >>> >> >>> > > > >> - Ease of integration with third party auth (i.e. LDAP,
> >>> AD,
> >>> >> >>> OAuth,
> >>> >> >>> > > etc.)
> >>> >> >>> > > > >> - Any other thoughts/concerns
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> Thanks a lot!
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> Cheers,
> >>> >> >>> > > > >> Joy
> >>> >> >>> > > > >>
> >>> >> >>> > > >
> >>> >> >>> > > >
> >>> >> >>> > >
> >>> >> >>> >
> >>> >> >>>
> >>> >> >>
> >>> >> >>
> >>> >> >
> >>> >>
> >>> >
> >>> >
> >>>
> >>
> >>
> >
>
Re: RBAC Update
Posted by Ace Haidrey <ac...@gmail.com>.
Amazing work! Super excited about this. I know you've spent a lot of time and energy on it :)
> On Mar 23, 2018, at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
>
> Hey guys!
>
> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
> been merged to master. I'm looking forward to early adopters' feedback and
> bug reports. I also hope to have more folks helping out with the RBAC UI,
> especially with introducing DAG-Level access control, which is a feature
> that a lot of people have been asking. If you are interested in helping out
> with this effort, let's talk more!
>
> This commit will be in the 1.10.0 release, and we are going to maintain
> both UIs simultaneously for a short period of time. Once RBAC UI is stable
> and battle-tested, we will deprecate the old UI and eventually remove it
> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
> two UIs from forking into separate paths, as that would become very
> difficult to maintain.
>
> Going forward while both UIs are up, if you are making a change to any
> files in airflow/www/ (old UI), where applicable, please also make the
> change to the airflow/www_rbac/ (new UI). If you rather not make changes in
> both UIs, it is recommended that you only make the changes to the RBAC UI,
> since that is the one we are maintaining in the long term.
>
> I'm excited that the RBAC UI will be able to bring additional security to
> Airflow, and with FAB framework in place we can look into leveraging it for
> a unified set of APIs used by both UI and CLI.
>
> Joy
>
>
>
> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
>
>> Hi folks,
>>
>> I have a PR <https://github.com/apache/incubator-airflow/pull/3015> out
>> for the new UI. I've included instructions on how to test it out in the PR
>> description. Looking forward to your feedbacks.
>>
>> Cheers,
>> Joy
>>
>> On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
>>
>>> Thanks for the background info. Would be really awesome for you to have
>>> PyPi access :D I'll make the change to have Airflow Webserver's FAB
>>> dependency pointing to my fork for the mean time.
>>>
>>> For folks who are interested in RBAC, I will be giving a talk/demo at the Airflow
>>> Meet-Up
>>> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/>
>>> next Monday. Happy to chat afterwards about it as well :)
>>>
>>> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
>>> maximebeauchemin@gmail.com> wrote:
>>>
>>>> A bit of related history here:
>>>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
>>>>
>>>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
>>>> maximebeauchemin@gmail.com> wrote:
>>>>
>>>>> Given I have merge rights on FAB I could probably do another round of
>>>>> review and get your PRs through. I would really like to get the main
>>>>> maintainer's input on things that touch the core (composite-key
>>>> support) as
>>>>> he might have concerns/intuitions that we can't know about.
>>>>>
>>>>> I do not have Pypi access though so I cannot push new releases out. I
>>>>> could ask for that.
>>>>>
>>>>> I've threatened to fork the project before, that's always an option.
>>>> I've
>>>>> noticed his involvement is sporadic and comes in bursts.
>>>>>
>>>>> In the meantime, you can have the dependency in Airflow Webserver
>>>> pointing
>>>>> straight to your fork.
>>>>>
>>>>> Max
>>>>>
>>>>> On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
>>>>>
>>>>>> I just created a new webserver instance if you haven't gotten a
>>>> chance to
>>>>>> fiddle around with the new web UI and the RBAC configurations (thanks
>>>>>> Maxime for getting started with this earlier!):
>>>>>>
>>>>>> http://104.209.38.171:8080/
>>>>>>
>>>>>> Admin Account
>>>>>> username: admin
>>>>>> password: admin
>>>>>>
>>>>>> Read-Only Account
>>>>>> username: viewer
>>>>>> password: password
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> Thanks for all the feedback regarding to the new Airflow Webserver
>>>> UI
>>>>>>> <https://github.com/wepay/airflow-webserver/>! I've been actively
>>>>>>> addressing all the bugs that were raised on Github. So I want to
>>>> take
>>>>>> this
>>>>>>> opportunity to discuss two issues coming up:
>>>>>>>
>>>>>>> The first issue is unaddressed PRs in FAB. If these PRs continue to
>>>> stay
>>>>>>> unaddressed, RBAC is blocked from making further progress. If this
>>>>>> continue
>>>>>>> to be an issue, I'm inclined to fork FAB, even though it's not
>>>>>> idealistic.
>>>>>>>
>>>>>>>
>>>>>>> - PR/631 <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>>>>>> Binary
>>>>>>> column support (merged, unreleased)
>>>>>>> <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>>>>>>> - PR/639 <https://github.com/dpgaspar/Flask-AppBuilder/pull/639>
>>>>>> Composite
>>>>>>> primary key support (unmerged)
>>>>>>> - PR/655 <https://github.com/dpgaspar/Flask-AppBuilder/pull/655>
>>>>>> Form
>>>>>>> prefill support (unmerged)
>>>>>>>
>>>>>>>
>>>>>>> The second issue is an open question about the next step of Airflow
>>>>>>> Webserver itself. Here are the 3 potential directions we could
>>>> take, and
>>>>>>> I've added my thought on each.
>>>>>>>
>>>>>>> 1. Permanently keep Airflow Webserver as a separated package from
>>>>>> Airflow,
>>>>>>> and treat it as another UI option. Keep `www` in Airflow. Allow
>>>>>> development
>>>>>>> on both UIs.
>>>>>>> *I'm not a fan of this. When there is an existing UI in Airflow,
>>>> most
>>>>>>> contributors would prefer to maintain the official version that is
>>>>>>> installed out-of-the-box. **Having a second UI outside of Airflow
>>>> will
>>>>>>> make maintaining it very difficult, leading to an eventual death of
>>>> the
>>>>>> new
>>>>>>> UI :(*
>>>>>>>
>>>>>>> 2. Permanently keep Airflow Webserver as a separated package from
>>>>>> Airflow,
>>>>>>> but freeze all development on `www` and direct all future UI
>>>>>> development
>>>>>>> to Airflow Webserver, eventually removing `www` completely when
>>>> Airflow
>>>>>>> Webserver is stable.
>>>>>>> *I'm not a fan of this either. First of all, the views and models
>>>> are
>>>>>>> tightly coupled in both old and new UI; until we have a full-fledged
>>>>>> REST
>>>>>>> API to build the UI (and cli) on top of it, separating them to a
>>>>>> separate
>>>>>>> package now will potentially cause dependency issues and add
>>>>>> complication
>>>>>>> to our release cycle. **Secondly, **majority of Airflow users run
>>>>>> Airflow
>>>>>>> with the UI; it's one of Airflow's best features. Separating UI out
>>>> of
>>>>>>> Airflow core will complicate setup and configuration, while making
>>>>>> Airflow
>>>>>>> core less complete.*
>>>>>>>
>>>>>>> 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
>>>>>>> development on `www`, eventually removing `www` completely when
>>>> `www2`
>>>>>> is
>>>>>>> stable.
>>>>>>> *This makes the most sense to me. Airflow Webserver is developed
>>>> with
>>>>>> the
>>>>>>> goal of feature parity to the current UI, plus additional RBAC
>>>>>> capability,
>>>>>>> in hope to replace the old UI completely. Yes, this means there
>>>> will be
>>>>>> a
>>>>>>> short period of having to maintain two UIs, but once we freeze
>>>>>> development
>>>>>>> on www, it shouldn't be a concern for long.*
>>>>>>>
>>>>>>> I'd love to hear everyone's thoughts on this! I'm excited about
>>>> bringing
>>>>>>> RBAC to airflow and I hope it's something others will find useful as
>>>>>> well!
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Joy
>>>>>>>
>>>>>>> On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
>>>>>>>
>>>>>>>> Thank you everyone for the active feedback so far, and thanks for
>>>>>> setting
>>>>>>>> up the demo Maxime!
>>>>>>>>
>>>>>>>> Going to work on pruning through the issues in the upcoming days.
>>>>>>>>
>>>>>>>> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I
>>>> can
>>>>>>>> look into it? Otherwise I'll wait until it's down again =P
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Joy
>>>>>>>>
>>>>>>>> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I just restarted it, not sure how long it will take to get in a
>>>> bad
>>>>>> state
>>>>>>>>> again...
>>>>>>>>>
>>>>>>>>> Max
>>>>>>>>>
>>>>>>>>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
>>>>>> <fokko@driesprong.frl
>>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Good morning,
>>>>>>>>>>
>>>>>>>>>> The demo provided by Max is down, it throws a
>>>> SQLAlchemyexception
>>>>>> :'(
>>>>>>>>>>
>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>
>>>>>>>>>> 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
>>>> criccomini@apache.org>:
>>>>>>>>>>
>>>>>>>>>>> @bolke, open issues on the Github repo, please.
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
>>>>>> bdbruin@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Chris,
>>>>>>>>>>>>
>>>>>>>>>>>> Do you want us to report bugs somewhere (I have encountered
>>>> a
>>>>>>>>> few)? Or
>>>>>>>>>>>> just generic user experiences posted here?
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Bolke
>>>>>>>>>>>>
>>>>>>>>>>>>> On 18 Nov 2017, at 00:47, Chris Riccomini <
>>>>>> criccomini@apache.org
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I know the weekend is coming up, and for those of us in
>>>> the
>>>>>> US,
>>>>>>>>> next
>>>>>>>>>>> week
>>>>>>>>>>>>> is a bit of a slow holiday week. Would love to get some
>>>>>> feedback
>>>>>>>>> from
>>>>>>>>>>>>> everyone on this. The goal would ideally to be to
>>>> converge on
>>>>>>>>> this
>>>>>>>>>> and
>>>>>>>>>>>>> eventually replace the existing Airflow UI with this one.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com>
>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi guys.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've been working on moving airflow from Flask-Admin to
>>>>>>>>>>> Flask-AppBuilder
>>>>>>>>>>>>>> for RBAC
>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/
>>>>>>>>>>>> Airflow+RBAC+proposal
>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>> check it out at https://github.com/wepay/airfl
>>>> ow-webserver.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's still a work-in-progress, but most features you see
>>>> in
>>>>>> the
>>>>>>>>>>>> webserver
>>>>>>>>>>>>>> UI today is available there. For those who are
>>>> interested in
>>>>>>>>> RBAC,
>>>>>>>>>> I'd
>>>>>>>>>>>> love
>>>>>>>>>>>>>> to get some early feedback in terms of the following:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - New Flask-AppBuilder UI (any bugs/regressions)
>>>>>>>>>>>>>> - Setup issues
>>>>>>>>>>>>>> - Ease of integration with third party auth (i.e. LDAP,
>>>> AD,
>>>>>>>>> OAuth,
>>>>>>>>>>> etc.)
>>>>>>>>>>>>>> - Any other thoughts/concerns
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks a lot!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Joy
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
Re: RBAC Update
Posted by Maxime Beauchemin <ma...@gmail.com>.
Awesome, great work Joy!
Max
On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
> Hey guys!
>
> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
> been merged to master. I'm looking forward to early adopters' feedback and
> bug reports. I also hope to have more folks helping out with the RBAC UI,
> especially with introducing DAG-Level access control, which is a feature
> that a lot of people have been asking. If you are interested in helping out
> with this effort, let's talk more!
>
> This commit will be in the 1.10.0 release, and we are going to maintain
> both UIs simultaneously for a short period of time. Once RBAC UI is stable
> and battle-tested, we will deprecate the old UI and eventually remove it
> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
> two UIs from forking into separate paths, as that would become very
> difficult to maintain.
>
> Going forward while both UIs are up, if you are making a change to any
> files in airflow/www/ (old UI), where applicable, please also make the
> change to the airflow/www_rbac/ (new UI). If you rather not make changes in
> both UIs, it is recommended that you only make the changes to the RBAC UI,
> since that is the one we are maintaining in the long term.
>
> I'm excited that the RBAC UI will be able to bring additional security to
> Airflow, and with FAB framework in place we can look into leveraging it for
> a unified set of APIs used by both UI and CLI.
>
> Joy
>
>
>
> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hi folks,
> >
> > I have a PR <https://github.com/apache/incubator-airflow/pull/3015> out
> > for the new UI. I've included instructions on how to test it out in the
> PR
> > description. Looking forward to your feedbacks.
> >
> > Cheers,
> > Joy
> >
> > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> >> Thanks for the background info. Would be really awesome for you to have
> >> PyPi access :D I'll make the change to have Airflow Webserver's FAB
> >> dependency pointing to my fork for the mean time.
> >>
> >> For folks who are interested in RBAC, I will be giving a talk/demo at
> the Airflow
> >> Meet-Up
> >> <https://www.meetup.com/Bay-Area-Apache-Airflow-
> Incubating-Meetup/events/244525050/>
> >> next Monday. Happy to chat afterwards about it as well :)
> >>
> >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> >> maximebeauchemin@gmail.com> wrote:
> >>
> >>> A bit of related history here:
> >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> >>>
> >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> >>> maximebeauchemin@gmail.com> wrote:
> >>>
> >>> > Given I have merge rights on FAB I could probably do another round of
> >>> > review and get your PRs through. I would really like to get the main
> >>> > maintainer's input on things that touch the core (composite-key
> >>> support) as
> >>> > he might have concerns/intuitions that we can't know about.
> >>> >
> >>> > I do not have Pypi access though so I cannot push new releases out. I
> >>> > could ask for that.
> >>> >
> >>> > I've threatened to fork the project before, that's always an option.
> >>> I've
> >>> > noticed his involvement is sporadic and comes in bursts.
> >>> >
> >>> > In the meantime, you can have the dependency in Airflow Webserver
> >>> pointing
> >>> > straight to your fork.
> >>> >
> >>> > Max
> >>> >
> >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
> >>> >
> >>> >> I just created a new webserver instance if you haven't gotten a
> >>> chance to
> >>> >> fiddle around with the new web UI and the RBAC configurations
> (thanks
> >>> >> Maxime for getting started with this earlier!):
> >>> >>
> >>> >> http://104.209.38.171:8080/
> >>> >>
> >>> >> Admin Account
> >>> >> username: admin
> >>> >> password: admin
> >>> >>
> >>> >> Read-Only Account
> >>> >> username: viewer
> >>> >> password: password
> >>> >>
> >>> >>
> >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
> >>> >>
> >>> >> > Hi folks,
> >>> >> >
> >>> >> > Thanks for all the feedback regarding to the new Airflow Webserver
> >>> UI
> >>> >> > <https://github.com/wepay/airflow-webserver/>! I've been actively
> >>> >> > addressing all the bugs that were raised on Github. So I want to
> >>> take
> >>> >> this
> >>> >> > opportunity to discuss two issues coming up:
> >>> >> >
> >>> >> > The first issue is unaddressed PRs in FAB. If these PRs continue
> to
> >>> stay
> >>> >> > unaddressed, RBAC is blocked from making further progress. If this
> >>> >> continue
> >>> >> > to be an issue, I'm inclined to fork FAB, even though it's not
> >>> >> idealistic.
> >>> >> >
> >>> >> >
> >>> >> > - PR/631 <https://github.com/dpgaspar/
> Flask-AppBuilder/pull/631>
> >>> >> Binary
> >>> >> > column support (merged, unreleased)
> >>> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> >>> >> > - PR/639 <https://github.com/dpgaspar/
> Flask-AppBuilder/pull/639>
> >>> >> Composite
> >>> >> > primary key support (unmerged)
> >>> >> > - PR/655 <https://github.com/dpgaspar/
> Flask-AppBuilder/pull/655>
> >>> >> Form
> >>> >> > prefill support (unmerged)
> >>> >> >
> >>> >> >
> >>> >> > The second issue is an open question about the next step of
> Airflow
> >>> >> > Webserver itself. Here are the 3 potential directions we could
> >>> take, and
> >>> >> > I've added my thought on each.
> >>> >> >
> >>> >> > 1. Permanently keep Airflow Webserver as a separated package from
> >>> >> Airflow,
> >>> >> > and treat it as another UI option. Keep `www` in Airflow. Allow
> >>> >> development
> >>> >> > on both UIs.
> >>> >> > *I'm not a fan of this. When there is an existing UI in Airflow,
> >>> most
> >>> >> > contributors would prefer to maintain the official version that is
> >>> >> > installed out-of-the-box. **Having a second UI outside of Airflow
> >>> will
> >>> >> > make maintaining it very difficult, leading to an eventual death
> of
> >>> the
> >>> >> new
> >>> >> > UI :(*
> >>> >> >
> >>> >> > 2. Permanently keep Airflow Webserver as a separated package from
> >>> >> Airflow,
> >>> >> > but freeze all development on `www` and direct all future UI
> >>> >> development
> >>> >> > to Airflow Webserver, eventually removing `www` completely when
> >>> Airflow
> >>> >> > Webserver is stable.
> >>> >> > *I'm not a fan of this either. First of all, the views and models
> >>> are
> >>> >> > tightly coupled in both old and new UI; until we have a
> full-fledged
> >>> >> REST
> >>> >> > API to build the UI (and cli) on top of it, separating them to a
> >>> >> separate
> >>> >> > package now will potentially cause dependency issues and add
> >>> >> complication
> >>> >> > to our release cycle. **Secondly, **majority of Airflow users run
> >>> >> Airflow
> >>> >> > with the UI; it's one of Airflow's best features. Separating UI
> out
> >>> of
> >>> >> > Airflow core will complicate setup and configuration, while making
> >>> >> Airflow
> >>> >> > core less complete.*
> >>> >> >
> >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
> >>> >> > development on `www`, eventually removing `www` completely when
> >>> `www2`
> >>> >> is
> >>> >> > stable.
> >>> >> > *This makes the most sense to me. Airflow Webserver is developed
> >>> with
> >>> >> the
> >>> >> > goal of feature parity to the current UI, plus additional RBAC
> >>> >> capability,
> >>> >> > in hope to replace the old UI completely. Yes, this means there
> >>> will be
> >>> >> a
> >>> >> > short period of having to maintain two UIs, but once we freeze
> >>> >> development
> >>> >> > on www, it shouldn't be a concern for long.*
> >>> >> >
> >>> >> > I'd love to hear everyone's thoughts on this! I'm excited about
> >>> bringing
> >>> >> > RBAC to airflow and I hope it's something others will find useful
> as
> >>> >> well!
> >>> >> >
> >>> >> > Cheers,
> >>> >> > Joy
> >>> >> >
> >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
> >>> >> >
> >>> >> >> Thank you everyone for the active feedback so far, and thanks for
> >>> >> setting
> >>> >> >> up the demo Maxime!
> >>> >> >>
> >>> >> >> Going to work on pruning through the issues in the upcoming days.
> >>> >> >>
> >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I
> >>> can
> >>> >> >> look into it? Otherwise I'll wait until it's down again =P
> >>> >> >>
> >>> >> >> Cheers,
> >>> >> >>
> >>> >> >> Joy
> >>> >> >>
> >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >>> >> >> maximebeauchemin@gmail.com> wrote:
> >>> >> >>
> >>> >> >>> I just restarted it, not sure how long it will take to get in a
> >>> bad
> >>> >> state
> >>> >> >>> again...
> >>> >> >>>
> >>> >> >>> Max
> >>> >> >>>
> >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> >>> >> <fokko@driesprong.frl
> >>> >> >>> >
> >>> >> >>> wrote:
> >>> >> >>>
> >>> >> >>> > Good morning,
> >>> >> >>> >
> >>> >> >>> > The demo provided by Max is down, it throws a
> >>> SQLAlchemyexception
> >>> >> :'(
> >>> >> >>> >
> >>> >> >>> > Cheers, Fokko
> >>> >> >>> >
> >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> >>> criccomini@apache.org>:
> >>> >> >>> >
> >>> >> >>> > > @bolke, open issues on the Github repo, please.
> >>> >> >>> > >
> >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> >>> >> bdbruin@gmail.com>
> >>> >> >>> > > wrote:
> >>> >> >>> > >
> >>> >> >>> > > > Chris,
> >>> >> >>> > > >
> >>> >> >>> > > > Do you want us to report bugs somewhere (I have
> encountered
> >>> a
> >>> >> >>> few)? Or
> >>> >> >>> > > > just generic user experiences posted here?
> >>> >> >>> > > >
> >>> >> >>> > > > Cheers
> >>> >> >>> > > > Bolke
> >>> >> >>> > > >
> >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> >>> >> criccomini@apache.org
> >>> >> >>> >
> >>> >> >>> > > wrote:
> >>> >> >>> > > > >
> >>> >> >>> > > > > Hey all,
> >>> >> >>> > > > >
> >>> >> >>> > > > > I know the weekend is coming up, and for those of us in
> >>> the
> >>> >> US,
> >>> >> >>> next
> >>> >> >>> > > week
> >>> >> >>> > > > > is a bit of a slow holiday week. Would love to get some
> >>> >> feedback
> >>> >> >>> from
> >>> >> >>> > > > > everyone on this. The goal would ideally to be to
> >>> converge on
> >>> >> >>> this
> >>> >> >>> > and
> >>> >> >>> > > > > eventually replace the existing Airflow UI with this
> one.
> >>> >> >>> > > > >
> >>> >> >>> > > > > Cheers,
> >>> >> >>> > > > > Chris
> >>> >> >>> > > > >
> >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> joyg@wepay.com>
> >>> >> wrote:
> >>> >> >>> > > > >
> >>> >> >>> > > > >> Hi guys.
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> I've been working on moving airflow from Flask-Admin to
> >>> >> >>> > > Flask-AppBuilder
> >>> >> >>> > > > >> for RBAC
> >>> >> >>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> >>> >> >>> > > > Airflow+RBAC+proposal
> >>> >> >>> > > > >>> ,
> >>> >> >>> > > > >> check it out at https://github.com/wepay/airfl
> >>> ow-webserver.
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> It's still a work-in-progress, but most features you
> see
> >>> in
> >>> >> the
> >>> >> >>> > > > webserver
> >>> >> >>> > > > >> UI today is available there. For those who are
> >>> interested in
> >>> >> >>> RBAC,
> >>> >> >>> > I'd
> >>> >> >>> > > > love
> >>> >> >>> > > > >> to get some early feedback in terms of the following:
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> >>> >> >>> > > > >> - Setup issues
> >>> >> >>> > > > >> - Ease of integration with third party auth (i.e. LDAP,
> >>> AD,
> >>> >> >>> OAuth,
> >>> >> >>> > > etc.)
> >>> >> >>> > > > >> - Any other thoughts/concerns
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> Thanks a lot!
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> Cheers,
> >>> >> >>> > > > >> Joy
> >>> >> >>> > > > >>
> >>> >> >>> > > >
> >>> >> >>> > > >
> >>> >> >>> > >
> >>> >> >>> >
> >>> >> >>>
> >>> >> >>
> >>> >> >>
> >>> >> >
> >>> >>
> >>> >
> >>> >
> >>>
> >>
> >>
> >
>
Re: RBAC Update
Posted by James Meickle <jm...@quantopian.com>.
This is super exciting for us, as we want one of our non-technical teams to
be able to re-run failed DAGs. Will be giving this a try soon as I'm back
from SREcon!
On Fri, Mar 23, 2018 at 9:45 PM, Joy Gao <jo...@wepay.com> wrote:
> Hey guys!
>
> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
> been merged to master. I'm looking forward to early adopters' feedback and
> bug reports. I also hope to have more folks helping out with the RBAC UI,
> especially with introducing DAG-Level access control, which is a feature
> that a lot of people have been asking. If you are interested in helping out
> with this effort, let's talk more!
>
> This commit will be in the 1.10.0 release, and we are going to maintain
> both UIs simultaneously for a short period of time. Once RBAC UI is stable
> and battle-tested, we will deprecate the old UI and eventually remove it
> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
> two UIs from forking into separate paths, as that would become very
> difficult to maintain.
>
> Going forward while both UIs are up, if you are making a change to any
> files in airflow/www/ (old UI), where applicable, please also make the
> change to the airflow/www_rbac/ (new UI). If you rather not make changes in
> both UIs, it is recommended that you only make the changes to the RBAC UI,
> since that is the one we are maintaining in the long term.
>
> I'm excited that the RBAC UI will be able to bring additional security to
> Airflow, and with FAB framework in place we can look into leveraging it for
> a unified set of APIs used by both UI and CLI.
>
> Joy
>
>
>
> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hi folks,
> >
> > I have a PR <https://github.com/apache/incubator-airflow/pull/3015> out
> > for the new UI. I've included instructions on how to test it out in the
> PR
> > description. Looking forward to your feedbacks.
> >
> > Cheers,
> > Joy
> >
> > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> >> Thanks for the background info. Would be really awesome for you to have
> >> PyPi access :D I'll make the change to have Airflow Webserver's FAB
> >> dependency pointing to my fork for the mean time.
> >>
> >> For folks who are interested in RBAC, I will be giving a talk/demo at
> the Airflow
> >> Meet-Up
> >> <https://www.meetup.com/Bay-Area-Apache-Airflow-
> Incubating-Meetup/events/244525050/>
> >> next Monday. Happy to chat afterwards about it as well :)
> >>
> >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> >> maximebeauchemin@gmail.com> wrote:
> >>
> >>> A bit of related history here:
> >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> >>>
> >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> >>> maximebeauchemin@gmail.com> wrote:
> >>>
> >>> > Given I have merge rights on FAB I could probably do another round of
> >>> > review and get your PRs through. I would really like to get the main
> >>> > maintainer's input on things that touch the core (composite-key
> >>> support) as
> >>> > he might have concerns/intuitions that we can't know about.
> >>> >
> >>> > I do not have Pypi access though so I cannot push new releases out. I
> >>> > could ask for that.
> >>> >
> >>> > I've threatened to fork the project before, that's always an option.
> >>> I've
> >>> > noticed his involvement is sporadic and comes in bursts.
> >>> >
> >>> > In the meantime, you can have the dependency in Airflow Webserver
> >>> pointing
> >>> > straight to your fork.
> >>> >
> >>> > Max
> >>> >
> >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
> >>> >
> >>> >> I just created a new webserver instance if you haven't gotten a
> >>> chance to
> >>> >> fiddle around with the new web UI and the RBAC configurations
> (thanks
> >>> >> Maxime for getting started with this earlier!):
> >>> >>
> >>> >> http://104.209.38.171:8080/
> >>> >>
> >>> >> Admin Account
> >>> >> username: admin
> >>> >> password: admin
> >>> >>
> >>> >> Read-Only Account
> >>> >> username: viewer
> >>> >> password: password
> >>> >>
> >>> >>
> >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
> >>> >>
> >>> >> > Hi folks,
> >>> >> >
> >>> >> > Thanks for all the feedback regarding to the new Airflow Webserver
> >>> UI
> >>> >> > <https://github.com/wepay/airflow-webserver/>! I've been actively
> >>> >> > addressing all the bugs that were raised on Github. So I want to
> >>> take
> >>> >> this
> >>> >> > opportunity to discuss two issues coming up:
> >>> >> >
> >>> >> > The first issue is unaddressed PRs in FAB. If these PRs continue
> to
> >>> stay
> >>> >> > unaddressed, RBAC is blocked from making further progress. If this
> >>> >> continue
> >>> >> > to be an issue, I'm inclined to fork FAB, even though it's not
> >>> >> idealistic.
> >>> >> >
> >>> >> >
> >>> >> > - PR/631 <https://github.com/dpgaspar/
> Flask-AppBuilder/pull/631>
> >>> >> Binary
> >>> >> > column support (merged, unreleased)
> >>> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> >>> >> > - PR/639 <https://github.com/dpgaspar/
> Flask-AppBuilder/pull/639>
> >>> >> Composite
> >>> >> > primary key support (unmerged)
> >>> >> > - PR/655 <https://github.com/dpgaspar/
> Flask-AppBuilder/pull/655>
> >>> >> Form
> >>> >> > prefill support (unmerged)
> >>> >> >
> >>> >> >
> >>> >> > The second issue is an open question about the next step of
> Airflow
> >>> >> > Webserver itself. Here are the 3 potential directions we could
> >>> take, and
> >>> >> > I've added my thought on each.
> >>> >> >
> >>> >> > 1. Permanently keep Airflow Webserver as a separated package from
> >>> >> Airflow,
> >>> >> > and treat it as another UI option. Keep `www` in Airflow. Allow
> >>> >> development
> >>> >> > on both UIs.
> >>> >> > *I'm not a fan of this. When there is an existing UI in Airflow,
> >>> most
> >>> >> > contributors would prefer to maintain the official version that is
> >>> >> > installed out-of-the-box. **Having a second UI outside of Airflow
> >>> will
> >>> >> > make maintaining it very difficult, leading to an eventual death
> of
> >>> the
> >>> >> new
> >>> >> > UI :(*
> >>> >> >
> >>> >> > 2. Permanently keep Airflow Webserver as a separated package from
> >>> >> Airflow,
> >>> >> > but freeze all development on `www` and direct all future UI
> >>> >> development
> >>> >> > to Airflow Webserver, eventually removing `www` completely when
> >>> Airflow
> >>> >> > Webserver is stable.
> >>> >> > *I'm not a fan of this either. First of all, the views and models
> >>> are
> >>> >> > tightly coupled in both old and new UI; until we have a
> full-fledged
> >>> >> REST
> >>> >> > API to build the UI (and cli) on top of it, separating them to a
> >>> >> separate
> >>> >> > package now will potentially cause dependency issues and add
> >>> >> complication
> >>> >> > to our release cycle. **Secondly, **majority of Airflow users run
> >>> >> Airflow
> >>> >> > with the UI; it's one of Airflow's best features. Separating UI
> out
> >>> of
> >>> >> > Airflow core will complicate setup and configuration, while making
> >>> >> Airflow
> >>> >> > core less complete.*
> >>> >> >
> >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
> >>> >> > development on `www`, eventually removing `www` completely when
> >>> `www2`
> >>> >> is
> >>> >> > stable.
> >>> >> > *This makes the most sense to me. Airflow Webserver is developed
> >>> with
> >>> >> the
> >>> >> > goal of feature parity to the current UI, plus additional RBAC
> >>> >> capability,
> >>> >> > in hope to replace the old UI completely. Yes, this means there
> >>> will be
> >>> >> a
> >>> >> > short period of having to maintain two UIs, but once we freeze
> >>> >> development
> >>> >> > on www, it shouldn't be a concern for long.*
> >>> >> >
> >>> >> > I'd love to hear everyone's thoughts on this! I'm excited about
> >>> bringing
> >>> >> > RBAC to airflow and I hope it's something others will find useful
> as
> >>> >> well!
> >>> >> >
> >>> >> > Cheers,
> >>> >> > Joy
> >>> >> >
> >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
> >>> >> >
> >>> >> >> Thank you everyone for the active feedback so far, and thanks for
> >>> >> setting
> >>> >> >> up the demo Maxime!
> >>> >> >>
> >>> >> >> Going to work on pruning through the issues in the upcoming days.
> >>> >> >>
> >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I
> >>> can
> >>> >> >> look into it? Otherwise I'll wait until it's down again =P
> >>> >> >>
> >>> >> >> Cheers,
> >>> >> >>
> >>> >> >> Joy
> >>> >> >>
> >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >>> >> >> maximebeauchemin@gmail.com> wrote:
> >>> >> >>
> >>> >> >>> I just restarted it, not sure how long it will take to get in a
> >>> bad
> >>> >> state
> >>> >> >>> again...
> >>> >> >>>
> >>> >> >>> Max
> >>> >> >>>
> >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> >>> >> <fokko@driesprong.frl
> >>> >> >>> >
> >>> >> >>> wrote:
> >>> >> >>>
> >>> >> >>> > Good morning,
> >>> >> >>> >
> >>> >> >>> > The demo provided by Max is down, it throws a
> >>> SQLAlchemyexception
> >>> >> :'(
> >>> >> >>> >
> >>> >> >>> > Cheers, Fokko
> >>> >> >>> >
> >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> >>> criccomini@apache.org>:
> >>> >> >>> >
> >>> >> >>> > > @bolke, open issues on the Github repo, please.
> >>> >> >>> > >
> >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> >>> >> bdbruin@gmail.com>
> >>> >> >>> > > wrote:
> >>> >> >>> > >
> >>> >> >>> > > > Chris,
> >>> >> >>> > > >
> >>> >> >>> > > > Do you want us to report bugs somewhere (I have
> encountered
> >>> a
> >>> >> >>> few)? Or
> >>> >> >>> > > > just generic user experiences posted here?
> >>> >> >>> > > >
> >>> >> >>> > > > Cheers
> >>> >> >>> > > > Bolke
> >>> >> >>> > > >
> >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> >>> >> criccomini@apache.org
> >>> >> >>> >
> >>> >> >>> > > wrote:
> >>> >> >>> > > > >
> >>> >> >>> > > > > Hey all,
> >>> >> >>> > > > >
> >>> >> >>> > > > > I know the weekend is coming up, and for those of us in
> >>> the
> >>> >> US,
> >>> >> >>> next
> >>> >> >>> > > week
> >>> >> >>> > > > > is a bit of a slow holiday week. Would love to get some
> >>> >> feedback
> >>> >> >>> from
> >>> >> >>> > > > > everyone on this. The goal would ideally to be to
> >>> converge on
> >>> >> >>> this
> >>> >> >>> > and
> >>> >> >>> > > > > eventually replace the existing Airflow UI with this
> one.
> >>> >> >>> > > > >
> >>> >> >>> > > > > Cheers,
> >>> >> >>> > > > > Chris
> >>> >> >>> > > > >
> >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> joyg@wepay.com>
> >>> >> wrote:
> >>> >> >>> > > > >
> >>> >> >>> > > > >> Hi guys.
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> I've been working on moving airflow from Flask-Admin to
> >>> >> >>> > > Flask-AppBuilder
> >>> >> >>> > > > >> for RBAC
> >>> >> >>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> >>> >> >>> > > > Airflow+RBAC+proposal
> >>> >> >>> > > > >>> ,
> >>> >> >>> > > > >> check it out at https://github.com/wepay/airfl
> >>> ow-webserver.
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> It's still a work-in-progress, but most features you
> see
> >>> in
> >>> >> the
> >>> >> >>> > > > webserver
> >>> >> >>> > > > >> UI today is available there. For those who are
> >>> interested in
> >>> >> >>> RBAC,
> >>> >> >>> > I'd
> >>> >> >>> > > > love
> >>> >> >>> > > > >> to get some early feedback in terms of the following:
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> >>> >> >>> > > > >> - Setup issues
> >>> >> >>> > > > >> - Ease of integration with third party auth (i.e. LDAP,
> >>> AD,
> >>> >> >>> OAuth,
> >>> >> >>> > > etc.)
> >>> >> >>> > > > >> - Any other thoughts/concerns
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> Thanks a lot!
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> Cheers,
> >>> >> >>> > > > >> Joy
> >>> >> >>> > > > >>
> >>> >> >>> > > >
> >>> >> >>> > > >
> >>> >> >>> > >
> >>> >> >>> >
> >>> >> >>>
> >>> >> >>
> >>> >> >>
> >>> >> >
> >>> >>
> >>> >
> >>> >
> >>>
> >>
> >>
> >
>
Re: RBAC Update
Posted by Sumit Maheshwari <su...@gmail.com>.
Awesome.. Great work indeed..
On Sat, Mar 24, 2018, 13:43 Bolke de Bruin <bd...@gmail.com> wrote:
> Hear hear!
>
> Bolke.
>
> > On 24 Mar 2018, at 06:01, Arthur Wiedmer <ar...@gmail.com>
> wrote:
> >
> > Thanks a lot for all the effort Joy!
> >
> > Great job!
> >
> > Best,
> > Arthur
> >
> > On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> >> Hey guys!
> >>
> >> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
> >> been merged to master. I'm looking forward to early adopters' feedback
> and
> >> bug reports. I also hope to have more folks helping out with the RBAC
> UI,
> >> especially with introducing DAG-Level access control, which is a feature
> >> that a lot of people have been asking. If you are interested in helping
> out
> >> with this effort, let's talk more!
> >>
> >> This commit will be in the 1.10.0 release, and we are going to maintain
> >> both UIs simultaneously for a short period of time. Once RBAC UI is
> stable
> >> and battle-tested, we will deprecate the old UI and eventually remove it
> >> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to
> prevent
> >> two UIs from forking into separate paths, as that would become very
> >> difficult to maintain.
> >>
> >> Going forward while both UIs are up, if you are making a change to any
> >> files in airflow/www/ (old UI), where applicable, please also make the
> >> change to the airflow/www_rbac/ (new UI). If you rather not make
> changes in
> >> both UIs, it is recommended that you only make the changes to the RBAC
> UI,
> >> since that is the one we are maintaining in the long term.
> >>
> >> I'm excited that the RBAC UI will be able to bring additional security
> to
> >> Airflow, and with FAB framework in place we can look into leveraging it
> for
> >> a unified set of APIs used by both UI and CLI.
> >>
> >> Joy
> >>
> >>
> >>
> >> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> I have a PR <https://github.com/apache/incubator-airflow/pull/3015>
> out
> >>> for the new UI. I've included instructions on how to test it out in the
> >> PR
> >>> description. Looking forward to your feedbacks.
> >>>
> >>> Cheers,
> >>> Joy
> >>>
> >>> On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
> >>>
> >>>> Thanks for the background info. Would be really awesome for you to
> have
> >>>> PyPi access :D I'll make the change to have Airflow Webserver's FAB
> >>>> dependency pointing to my fork for the mean time.
> >>>>
> >>>> For folks who are interested in RBAC, I will be giving a talk/demo at
> >> the Airflow
> >>>> Meet-Up
> >>>> <https://www.meetup.com/Bay-Area-Apache-Airflow-
> >> Incubating-Meetup/events/244525050/>
> >>>> next Monday. Happy to chat afterwards about it as well :)
> >>>>
> >>>> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> >>>> maximebeauchemin@gmail.com> wrote:
> >>>>
> >>>>> A bit of related history here:
> >>>>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> >>>>>
> >>>>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> >>>>> maximebeauchemin@gmail.com> wrote:
> >>>>>
> >>>>>> Given I have merge rights on FAB I could probably do another round
> of
> >>>>>> review and get your PRs through. I would really like to get the main
> >>>>>> maintainer's input on things that touch the core (composite-key
> >>>>> support) as
> >>>>>> he might have concerns/intuitions that we can't know about.
> >>>>>>
> >>>>>> I do not have Pypi access though so I cannot push new releases out.
> I
> >>>>>> could ask for that.
> >>>>>>
> >>>>>> I've threatened to fork the project before, that's always an option.
> >>>>> I've
> >>>>>> noticed his involvement is sporadic and comes in bursts.
> >>>>>>
> >>>>>> In the meantime, you can have the dependency in Airflow Webserver
> >>>>> pointing
> >>>>>> straight to your fork.
> >>>>>>
> >>>>>> Max
> >>>>>>
> >>>>>> On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
> >>>>>>
> >>>>>>> I just created a new webserver instance if you haven't gotten a
> >>>>> chance to
> >>>>>>> fiddle around with the new web UI and the RBAC configurations
> >> (thanks
> >>>>>>> Maxime for getting started with this earlier!):
> >>>>>>>
> >>>>>>> http://104.209.38.171:8080/
> >>>>>>>
> >>>>>>> Admin Account
> >>>>>>> username: admin
> >>>>>>> password: admin
> >>>>>>>
> >>>>>>> Read-Only Account
> >>>>>>> username: viewer
> >>>>>>> password: password
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
> >>>>>>>
> >>>>>>>> Hi folks,
> >>>>>>>>
> >>>>>>>> Thanks for all the feedback regarding to the new Airflow Webserver
> >>>>> UI
> >>>>>>>> <https://github.com/wepay/airflow-webserver/>! I've been actively
> >>>>>>>> addressing all the bugs that were raised on Github. So I want to
> >>>>> take
> >>>>>>> this
> >>>>>>>> opportunity to discuss two issues coming up:
> >>>>>>>>
> >>>>>>>> The first issue is unaddressed PRs in FAB. If these PRs continue
> >> to
> >>>>> stay
> >>>>>>>> unaddressed, RBAC is blocked from making further progress. If this
> >>>>>>> continue
> >>>>>>>> to be an issue, I'm inclined to fork FAB, even though it's not
> >>>>>>> idealistic.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> - PR/631 <https://github.com/dpgaspar/
> >> Flask-AppBuilder/pull/631>
> >>>>>>> Binary
> >>>>>>>> column support (merged, unreleased)
> >>>>>>>> <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> >>>>>>>> - PR/639 <https://github.com/dpgaspar/
> >> Flask-AppBuilder/pull/639>
> >>>>>>> Composite
> >>>>>>>> primary key support (unmerged)
> >>>>>>>> - PR/655 <https://github.com/dpgaspar/
> >> Flask-AppBuilder/pull/655>
> >>>>>>> Form
> >>>>>>>> prefill support (unmerged)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> The second issue is an open question about the next step of
> >> Airflow
> >>>>>>>> Webserver itself. Here are the 3 potential directions we could
> >>>>> take, and
> >>>>>>>> I've added my thought on each.
> >>>>>>>>
> >>>>>>>> 1. Permanently keep Airflow Webserver as a separated package from
> >>>>>>> Airflow,
> >>>>>>>> and treat it as another UI option. Keep `www` in Airflow. Allow
> >>>>>>> development
> >>>>>>>> on both UIs.
> >>>>>>>> *I'm not a fan of this. When there is an existing UI in Airflow,
> >>>>> most
> >>>>>>>> contributors would prefer to maintain the official version that is
> >>>>>>>> installed out-of-the-box. **Having a second UI outside of Airflow
> >>>>> will
> >>>>>>>> make maintaining it very difficult, leading to an eventual death
> >> of
> >>>>> the
> >>>>>>> new
> >>>>>>>> UI :(*
> >>>>>>>>
> >>>>>>>> 2. Permanently keep Airflow Webserver as a separated package from
> >>>>>>> Airflow,
> >>>>>>>> but freeze all development on `www` and direct all future UI
> >>>>>>> development
> >>>>>>>> to Airflow Webserver, eventually removing `www` completely when
> >>>>> Airflow
> >>>>>>>> Webserver is stable.
> >>>>>>>> *I'm not a fan of this either. First of all, the views and models
> >>>>> are
> >>>>>>>> tightly coupled in both old and new UI; until we have a
> >> full-fledged
> >>>>>>> REST
> >>>>>>>> API to build the UI (and cli) on top of it, separating them to a
> >>>>>>> separate
> >>>>>>>> package now will potentially cause dependency issues and add
> >>>>>>> complication
> >>>>>>>> to our release cycle. **Secondly, **majority of Airflow users run
> >>>>>>> Airflow
> >>>>>>>> with the UI; it's one of Airflow's best features. Separating UI
> >> out
> >>>>> of
> >>>>>>>> Airflow core will complicate setup and configuration, while making
> >>>>>>> Airflow
> >>>>>>>> core less complete.*
> >>>>>>>>
> >>>>>>>> 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
> >>>>>>>> development on `www`, eventually removing `www` completely when
> >>>>> `www2`
> >>>>>>> is
> >>>>>>>> stable.
> >>>>>>>> *This makes the most sense to me. Airflow Webserver is developed
> >>>>> with
> >>>>>>> the
> >>>>>>>> goal of feature parity to the current UI, plus additional RBAC
> >>>>>>> capability,
> >>>>>>>> in hope to replace the old UI completely. Yes, this means there
> >>>>> will be
> >>>>>>> a
> >>>>>>>> short period of having to maintain two UIs, but once we freeze
> >>>>>>> development
> >>>>>>>> on www, it shouldn't be a concern for long.*
> >>>>>>>>
> >>>>>>>> I'd love to hear everyone's thoughts on this! I'm excited about
> >>>>> bringing
> >>>>>>>> RBAC to airflow and I hope it's something others will find useful
> >> as
> >>>>>>> well!
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Joy
> >>>>>>>>
> >>>>>>>> On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
> >>>>>>>>
> >>>>>>>>> Thank you everyone for the active feedback so far, and thanks for
> >>>>>>> setting
> >>>>>>>>> up the demo Maxime!
> >>>>>>>>>
> >>>>>>>>> Going to work on pruning through the issues in the upcoming days.
> >>>>>>>>>
> >>>>>>>>> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I
> >>>>> can
> >>>>>>>>> look into it? Otherwise I'll wait until it's down again =P
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>>
> >>>>>>>>> Joy
> >>>>>>>>>
> >>>>>>>>> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >>>>>>>>> maximebeauchemin@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> I just restarted it, not sure how long it will take to get in a
> >>>>> bad
> >>>>>>> state
> >>>>>>>>>> again...
> >>>>>>>>>>
> >>>>>>>>>> Max
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> >>>>>>> <fokko@driesprong.frl
> >>>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Good morning,
> >>>>>>>>>>>
> >>>>>>>>>>> The demo provided by Max is down, it throws a
> >>>>> SQLAlchemyexception
> >>>>>>> :'(
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers, Fokko
> >>>>>>>>>>>
> >>>>>>>>>>> 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> >>>>> criccomini@apache.org>:
> >>>>>>>>>>>
> >>>>>>>>>>>> @bolke, open issues on the Github repo, please.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> >>>>>>> bdbruin@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Chris,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Do you want us to report bugs somewhere (I have
> >> encountered
> >>>>> a
> >>>>>>>>>> few)? Or
> >>>>>>>>>>>>> just generic user experiences posted here?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>> Bolke
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 18 Nov 2017, at 00:47, Chris Riccomini <
> >>>>>>> criccomini@apache.org
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hey all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I know the weekend is coming up, and for those of us in
> >>>>> the
> >>>>>>> US,
> >>>>>>>>>> next
> >>>>>>>>>>>> week
> >>>>>>>>>>>>>> is a bit of a slow holiday week. Would love to get some
> >>>>>>> feedback
> >>>>>>>>>> from
> >>>>>>>>>>>>>> everyone on this. The goal would ideally to be to
> >>>>> converge on
> >>>>>>>>>> this
> >>>>>>>>>>> and
> >>>>>>>>>>>>>> eventually replace the existing Airflow UI with this
> >> one.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>> Chris
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> >> joyg@wepay.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi guys.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I've been working on moving airflow from Flask-Admin to
> >>>>>>>>>>>> Flask-AppBuilder
> >>>>>>>>>>>>>>> for RBAC
> >>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> >>>>>>>>>>>>> Airflow+RBAC+proposal
> >>>>>>>>>>>>>>>> ,
> >>>>>>>>>>>>>>> check it out at https://github.com/wepay/airfl
> >>>>> ow-webserver.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It's still a work-in-progress, but most features you
> >> see
> >>>>> in
> >>>>>>> the
> >>>>>>>>>>>>> webserver
> >>>>>>>>>>>>>>> UI today is available there. For those who are
> >>>>> interested in
> >>>>>>>>>> RBAC,
> >>>>>>>>>>> I'd
> >>>>>>>>>>>>> love
> >>>>>>>>>>>>>>> to get some early feedback in terms of the following:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - New Flask-AppBuilder UI (any bugs/regressions)
> >>>>>>>>>>>>>>> - Setup issues
> >>>>>>>>>>>>>>> - Ease of integration with third party auth (i.e. LDAP,
> >>>>> AD,
> >>>>>>>>>> OAuth,
> >>>>>>>>>>>> etc.)
> >>>>>>>>>>>>>>> - Any other thoughts/concerns
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks a lot!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> Joy
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>
Re: RBAC Update
Posted by Bolke de Bruin <bd...@gmail.com>.
Hear hear!
Bolke.
> On 24 Mar 2018, at 06:01, Arthur Wiedmer <ar...@gmail.com> wrote:
>
> Thanks a lot for all the effort Joy!
>
> Great job!
>
> Best,
> Arthur
>
> On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
>
>> Hey guys!
>>
>> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
>> been merged to master. I'm looking forward to early adopters' feedback and
>> bug reports. I also hope to have more folks helping out with the RBAC UI,
>> especially with introducing DAG-Level access control, which is a feature
>> that a lot of people have been asking. If you are interested in helping out
>> with this effort, let's talk more!
>>
>> This commit will be in the 1.10.0 release, and we are going to maintain
>> both UIs simultaneously for a short period of time. Once RBAC UI is stable
>> and battle-tested, we will deprecate the old UI and eventually remove it
>> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
>> two UIs from forking into separate paths, as that would become very
>> difficult to maintain.
>>
>> Going forward while both UIs are up, if you are making a change to any
>> files in airflow/www/ (old UI), where applicable, please also make the
>> change to the airflow/www_rbac/ (new UI). If you rather not make changes in
>> both UIs, it is recommended that you only make the changes to the RBAC UI,
>> since that is the one we are maintaining in the long term.
>>
>> I'm excited that the RBAC UI will be able to bring additional security to
>> Airflow, and with FAB framework in place we can look into leveraging it for
>> a unified set of APIs used by both UI and CLI.
>>
>> Joy
>>
>>
>>
>> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
>>
>>> Hi folks,
>>>
>>> I have a PR <https://github.com/apache/incubator-airflow/pull/3015> out
>>> for the new UI. I've included instructions on how to test it out in the
>> PR
>>> description. Looking forward to your feedbacks.
>>>
>>> Cheers,
>>> Joy
>>>
>>> On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
>>>
>>>> Thanks for the background info. Would be really awesome for you to have
>>>> PyPi access :D I'll make the change to have Airflow Webserver's FAB
>>>> dependency pointing to my fork for the mean time.
>>>>
>>>> For folks who are interested in RBAC, I will be giving a talk/demo at
>> the Airflow
>>>> Meet-Up
>>>> <https://www.meetup.com/Bay-Area-Apache-Airflow-
>> Incubating-Meetup/events/244525050/>
>>>> next Monday. Happy to chat afterwards about it as well :)
>>>>
>>>> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
>>>> maximebeauchemin@gmail.com> wrote:
>>>>
>>>>> A bit of related history here:
>>>>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
>>>>>
>>>>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>
>>>>>> Given I have merge rights on FAB I could probably do another round of
>>>>>> review and get your PRs through. I would really like to get the main
>>>>>> maintainer's input on things that touch the core (composite-key
>>>>> support) as
>>>>>> he might have concerns/intuitions that we can't know about.
>>>>>>
>>>>>> I do not have Pypi access though so I cannot push new releases out. I
>>>>>> could ask for that.
>>>>>>
>>>>>> I've threatened to fork the project before, that's always an option.
>>>>> I've
>>>>>> noticed his involvement is sporadic and comes in bursts.
>>>>>>
>>>>>> In the meantime, you can have the dependency in Airflow Webserver
>>>>> pointing
>>>>>> straight to your fork.
>>>>>>
>>>>>> Max
>>>>>>
>>>>>> On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
>>>>>>
>>>>>>> I just created a new webserver instance if you haven't gotten a
>>>>> chance to
>>>>>>> fiddle around with the new web UI and the RBAC configurations
>> (thanks
>>>>>>> Maxime for getting started with this earlier!):
>>>>>>>
>>>>>>> http://104.209.38.171:8080/
>>>>>>>
>>>>>>> Admin Account
>>>>>>> username: admin
>>>>>>> password: admin
>>>>>>>
>>>>>>> Read-Only Account
>>>>>>> username: viewer
>>>>>>> password: password
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> Thanks for all the feedback regarding to the new Airflow Webserver
>>>>> UI
>>>>>>>> <https://github.com/wepay/airflow-webserver/>! I've been actively
>>>>>>>> addressing all the bugs that were raised on Github. So I want to
>>>>> take
>>>>>>> this
>>>>>>>> opportunity to discuss two issues coming up:
>>>>>>>>
>>>>>>>> The first issue is unaddressed PRs in FAB. If these PRs continue
>> to
>>>>> stay
>>>>>>>> unaddressed, RBAC is blocked from making further progress. If this
>>>>>>> continue
>>>>>>>> to be an issue, I'm inclined to fork FAB, even though it's not
>>>>>>> idealistic.
>>>>>>>>
>>>>>>>>
>>>>>>>> - PR/631 <https://github.com/dpgaspar/
>> Flask-AppBuilder/pull/631>
>>>>>>> Binary
>>>>>>>> column support (merged, unreleased)
>>>>>>>> <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>>>>>>>> - PR/639 <https://github.com/dpgaspar/
>> Flask-AppBuilder/pull/639>
>>>>>>> Composite
>>>>>>>> primary key support (unmerged)
>>>>>>>> - PR/655 <https://github.com/dpgaspar/
>> Flask-AppBuilder/pull/655>
>>>>>>> Form
>>>>>>>> prefill support (unmerged)
>>>>>>>>
>>>>>>>>
>>>>>>>> The second issue is an open question about the next step of
>> Airflow
>>>>>>>> Webserver itself. Here are the 3 potential directions we could
>>>>> take, and
>>>>>>>> I've added my thought on each.
>>>>>>>>
>>>>>>>> 1. Permanently keep Airflow Webserver as a separated package from
>>>>>>> Airflow,
>>>>>>>> and treat it as another UI option. Keep `www` in Airflow. Allow
>>>>>>> development
>>>>>>>> on both UIs.
>>>>>>>> *I'm not a fan of this. When there is an existing UI in Airflow,
>>>>> most
>>>>>>>> contributors would prefer to maintain the official version that is
>>>>>>>> installed out-of-the-box. **Having a second UI outside of Airflow
>>>>> will
>>>>>>>> make maintaining it very difficult, leading to an eventual death
>> of
>>>>> the
>>>>>>> new
>>>>>>>> UI :(*
>>>>>>>>
>>>>>>>> 2. Permanently keep Airflow Webserver as a separated package from
>>>>>>> Airflow,
>>>>>>>> but freeze all development on `www` and direct all future UI
>>>>>>> development
>>>>>>>> to Airflow Webserver, eventually removing `www` completely when
>>>>> Airflow
>>>>>>>> Webserver is stable.
>>>>>>>> *I'm not a fan of this either. First of all, the views and models
>>>>> are
>>>>>>>> tightly coupled in both old and new UI; until we have a
>> full-fledged
>>>>>>> REST
>>>>>>>> API to build the UI (and cli) on top of it, separating them to a
>>>>>>> separate
>>>>>>>> package now will potentially cause dependency issues and add
>>>>>>> complication
>>>>>>>> to our release cycle. **Secondly, **majority of Airflow users run
>>>>>>> Airflow
>>>>>>>> with the UI; it's one of Airflow's best features. Separating UI
>> out
>>>>> of
>>>>>>>> Airflow core will complicate setup and configuration, while making
>>>>>>> Airflow
>>>>>>>> core less complete.*
>>>>>>>>
>>>>>>>> 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
>>>>>>>> development on `www`, eventually removing `www` completely when
>>>>> `www2`
>>>>>>> is
>>>>>>>> stable.
>>>>>>>> *This makes the most sense to me. Airflow Webserver is developed
>>>>> with
>>>>>>> the
>>>>>>>> goal of feature parity to the current UI, plus additional RBAC
>>>>>>> capability,
>>>>>>>> in hope to replace the old UI completely. Yes, this means there
>>>>> will be
>>>>>>> a
>>>>>>>> short period of having to maintain two UIs, but once we freeze
>>>>>>> development
>>>>>>>> on www, it shouldn't be a concern for long.*
>>>>>>>>
>>>>>>>> I'd love to hear everyone's thoughts on this! I'm excited about
>>>>> bringing
>>>>>>>> RBAC to airflow and I hope it's something others will find useful
>> as
>>>>>>> well!
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Joy
>>>>>>>>
>>>>>>>> On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
>>>>>>>>
>>>>>>>>> Thank you everyone for the active feedback so far, and thanks for
>>>>>>> setting
>>>>>>>>> up the demo Maxime!
>>>>>>>>>
>>>>>>>>> Going to work on pruning through the issues in the upcoming days.
>>>>>>>>>
>>>>>>>>> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I
>>>>> can
>>>>>>>>> look into it? Otherwise I'll wait until it's down again =P
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Joy
>>>>>>>>>
>>>>>>>>> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
>>>>>>>>> maximebeauchemin@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I just restarted it, not sure how long it will take to get in a
>>>>> bad
>>>>>>> state
>>>>>>>>>> again...
>>>>>>>>>>
>>>>>>>>>> Max
>>>>>>>>>>
>>>>>>>>>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
>>>>>>> <fokko@driesprong.frl
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Good morning,
>>>>>>>>>>>
>>>>>>>>>>> The demo provided by Max is down, it throws a
>>>>> SQLAlchemyexception
>>>>>>> :'(
>>>>>>>>>>>
>>>>>>>>>>> Cheers, Fokko
>>>>>>>>>>>
>>>>>>>>>>> 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
>>>>> criccomini@apache.org>:
>>>>>>>>>>>
>>>>>>>>>>>> @bolke, open issues on the Github repo, please.
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
>>>>>>> bdbruin@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Chris,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do you want us to report bugs somewhere (I have
>> encountered
>>>>> a
>>>>>>>>>> few)? Or
>>>>>>>>>>>>> just generic user experiences posted here?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Bolke
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 18 Nov 2017, at 00:47, Chris Riccomini <
>>>>>>> criccomini@apache.org
>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I know the weekend is coming up, and for those of us in
>>>>> the
>>>>>>> US,
>>>>>>>>>> next
>>>>>>>>>>>> week
>>>>>>>>>>>>>> is a bit of a slow holiday week. Would love to get some
>>>>>>> feedback
>>>>>>>>>> from
>>>>>>>>>>>>>> everyone on this. The goal would ideally to be to
>>>>> converge on
>>>>>>>>>> this
>>>>>>>>>>> and
>>>>>>>>>>>>>> eventually replace the existing Airflow UI with this
>> one.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
>> joyg@wepay.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi guys.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've been working on moving airflow from Flask-Admin to
>>>>>>>>>>>> Flask-AppBuilder
>>>>>>>>>>>>>>> for RBAC
>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/
>>>>>>>>>>>>> Airflow+RBAC+proposal
>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>> check it out at https://github.com/wepay/airfl
>>>>> ow-webserver.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's still a work-in-progress, but most features you
>> see
>>>>> in
>>>>>>> the
>>>>>>>>>>>>> webserver
>>>>>>>>>>>>>>> UI today is available there. For those who are
>>>>> interested in
>>>>>>>>>> RBAC,
>>>>>>>>>>> I'd
>>>>>>>>>>>>> love
>>>>>>>>>>>>>>> to get some early feedback in terms of the following:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - New Flask-AppBuilder UI (any bugs/regressions)
>>>>>>>>>>>>>>> - Setup issues
>>>>>>>>>>>>>>> - Ease of integration with third party auth (i.e. LDAP,
>>>>> AD,
>>>>>>>>>> OAuth,
>>>>>>>>>>>> etc.)
>>>>>>>>>>>>>>> - Any other thoughts/concerns
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks a lot!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Joy
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
Re: RBAC Update
Posted by Arthur Wiedmer <ar...@gmail.com>.
Thanks a lot for all the effort Joy!
Great job!
Best,
Arthur
On Fri, Mar 23, 2018 at 6:45 PM, Joy Gao <jo...@wepay.com> wrote:
> Hey guys!
>
> The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
> been merged to master. I'm looking forward to early adopters' feedback and
> bug reports. I also hope to have more folks helping out with the RBAC UI,
> especially with introducing DAG-Level access control, which is a feature
> that a lot of people have been asking. If you are interested in helping out
> with this effort, let's talk more!
>
> This commit will be in the 1.10.0 release, and we are going to maintain
> both UIs simultaneously for a short period of time. Once RBAC UI is stable
> and battle-tested, we will deprecate the old UI and eventually remove it
> from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
> two UIs from forking into separate paths, as that would become very
> difficult to maintain.
>
> Going forward while both UIs are up, if you are making a change to any
> files in airflow/www/ (old UI), where applicable, please also make the
> change to the airflow/www_rbac/ (new UI). If you rather not make changes in
> both UIs, it is recommended that you only make the changes to the RBAC UI,
> since that is the one we are maintaining in the long term.
>
> I'm excited that the RBAC UI will be able to bring additional security to
> Airflow, and with FAB framework in place we can look into leveraging it for
> a unified set of APIs used by both UI and CLI.
>
> Joy
>
>
>
> On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hi folks,
> >
> > I have a PR <https://github.com/apache/incubator-airflow/pull/3015> out
> > for the new UI. I've included instructions on how to test it out in the
> PR
> > description. Looking forward to your feedbacks.
> >
> > Cheers,
> > Joy
> >
> > On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> >> Thanks for the background info. Would be really awesome for you to have
> >> PyPi access :D I'll make the change to have Airflow Webserver's FAB
> >> dependency pointing to my fork for the mean time.
> >>
> >> For folks who are interested in RBAC, I will be giving a talk/demo at
> the Airflow
> >> Meet-Up
> >> <https://www.meetup.com/Bay-Area-Apache-Airflow-
> Incubating-Meetup/events/244525050/>
> >> next Monday. Happy to chat afterwards about it as well :)
> >>
> >> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> >> maximebeauchemin@gmail.com> wrote:
> >>
> >>> A bit of related history here:
> >>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
> >>>
> >>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> >>> maximebeauchemin@gmail.com> wrote:
> >>>
> >>> > Given I have merge rights on FAB I could probably do another round of
> >>> > review and get your PRs through. I would really like to get the main
> >>> > maintainer's input on things that touch the core (composite-key
> >>> support) as
> >>> > he might have concerns/intuitions that we can't know about.
> >>> >
> >>> > I do not have Pypi access though so I cannot push new releases out. I
> >>> > could ask for that.
> >>> >
> >>> > I've threatened to fork the project before, that's always an option.
> >>> I've
> >>> > noticed his involvement is sporadic and comes in bursts.
> >>> >
> >>> > In the meantime, you can have the dependency in Airflow Webserver
> >>> pointing
> >>> > straight to your fork.
> >>> >
> >>> > Max
> >>> >
> >>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
> >>> >
> >>> >> I just created a new webserver instance if you haven't gotten a
> >>> chance to
> >>> >> fiddle around with the new web UI and the RBAC configurations
> (thanks
> >>> >> Maxime for getting started with this earlier!):
> >>> >>
> >>> >> http://104.209.38.171:8080/
> >>> >>
> >>> >> Admin Account
> >>> >> username: admin
> >>> >> password: admin
> >>> >>
> >>> >> Read-Only Account
> >>> >> username: viewer
> >>> >> password: password
> >>> >>
> >>> >>
> >>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
> >>> >>
> >>> >> > Hi folks,
> >>> >> >
> >>> >> > Thanks for all the feedback regarding to the new Airflow Webserver
> >>> UI
> >>> >> > <https://github.com/wepay/airflow-webserver/>! I've been actively
> >>> >> > addressing all the bugs that were raised on Github. So I want to
> >>> take
> >>> >> this
> >>> >> > opportunity to discuss two issues coming up:
> >>> >> >
> >>> >> > The first issue is unaddressed PRs in FAB. If these PRs continue
> to
> >>> stay
> >>> >> > unaddressed, RBAC is blocked from making further progress. If this
> >>> >> continue
> >>> >> > to be an issue, I'm inclined to fork FAB, even though it's not
> >>> >> idealistic.
> >>> >> >
> >>> >> >
> >>> >> > - PR/631 <https://github.com/dpgaspar/
> Flask-AppBuilder/pull/631>
> >>> >> Binary
> >>> >> > column support (merged, unreleased)
> >>> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> >>> >> > - PR/639 <https://github.com/dpgaspar/
> Flask-AppBuilder/pull/639>
> >>> >> Composite
> >>> >> > primary key support (unmerged)
> >>> >> > - PR/655 <https://github.com/dpgaspar/
> Flask-AppBuilder/pull/655>
> >>> >> Form
> >>> >> > prefill support (unmerged)
> >>> >> >
> >>> >> >
> >>> >> > The second issue is an open question about the next step of
> Airflow
> >>> >> > Webserver itself. Here are the 3 potential directions we could
> >>> take, and
> >>> >> > I've added my thought on each.
> >>> >> >
> >>> >> > 1. Permanently keep Airflow Webserver as a separated package from
> >>> >> Airflow,
> >>> >> > and treat it as another UI option. Keep `www` in Airflow. Allow
> >>> >> development
> >>> >> > on both UIs.
> >>> >> > *I'm not a fan of this. When there is an existing UI in Airflow,
> >>> most
> >>> >> > contributors would prefer to maintain the official version that is
> >>> >> > installed out-of-the-box. **Having a second UI outside of Airflow
> >>> will
> >>> >> > make maintaining it very difficult, leading to an eventual death
> of
> >>> the
> >>> >> new
> >>> >> > UI :(*
> >>> >> >
> >>> >> > 2. Permanently keep Airflow Webserver as a separated package from
> >>> >> Airflow,
> >>> >> > but freeze all development on `www` and direct all future UI
> >>> >> development
> >>> >> > to Airflow Webserver, eventually removing `www` completely when
> >>> Airflow
> >>> >> > Webserver is stable.
> >>> >> > *I'm not a fan of this either. First of all, the views and models
> >>> are
> >>> >> > tightly coupled in both old and new UI; until we have a
> full-fledged
> >>> >> REST
> >>> >> > API to build the UI (and cli) on top of it, separating them to a
> >>> >> separate
> >>> >> > package now will potentially cause dependency issues and add
> >>> >> complication
> >>> >> > to our release cycle. **Secondly, **majority of Airflow users run
> >>> >> Airflow
> >>> >> > with the UI; it's one of Airflow's best features. Separating UI
> out
> >>> of
> >>> >> > Airflow core will complicate setup and configuration, while making
> >>> >> Airflow
> >>> >> > core less complete.*
> >>> >> >
> >>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
> >>> >> > development on `www`, eventually removing `www` completely when
> >>> `www2`
> >>> >> is
> >>> >> > stable.
> >>> >> > *This makes the most sense to me. Airflow Webserver is developed
> >>> with
> >>> >> the
> >>> >> > goal of feature parity to the current UI, plus additional RBAC
> >>> >> capability,
> >>> >> > in hope to replace the old UI completely. Yes, this means there
> >>> will be
> >>> >> a
> >>> >> > short period of having to maintain two UIs, but once we freeze
> >>> >> development
> >>> >> > on www, it shouldn't be a concern for long.*
> >>> >> >
> >>> >> > I'd love to hear everyone's thoughts on this! I'm excited about
> >>> bringing
> >>> >> > RBAC to airflow and I hope it's something others will find useful
> as
> >>> >> well!
> >>> >> >
> >>> >> > Cheers,
> >>> >> > Joy
> >>> >> >
> >>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
> >>> >> >
> >>> >> >> Thank you everyone for the active feedback so far, and thanks for
> >>> >> setting
> >>> >> >> up the demo Maxime!
> >>> >> >>
> >>> >> >> Going to work on pruning through the issues in the upcoming days.
> >>> >> >>
> >>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I
> >>> can
> >>> >> >> look into it? Otherwise I'll wait until it's down again =P
> >>> >> >>
> >>> >> >> Cheers,
> >>> >> >>
> >>> >> >> Joy
> >>> >> >>
> >>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >>> >> >> maximebeauchemin@gmail.com> wrote:
> >>> >> >>
> >>> >> >>> I just restarted it, not sure how long it will take to get in a
> >>> bad
> >>> >> state
> >>> >> >>> again...
> >>> >> >>>
> >>> >> >>> Max
> >>> >> >>>
> >>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> >>> >> <fokko@driesprong.frl
> >>> >> >>> >
> >>> >> >>> wrote:
> >>> >> >>>
> >>> >> >>> > Good morning,
> >>> >> >>> >
> >>> >> >>> > The demo provided by Max is down, it throws a
> >>> SQLAlchemyexception
> >>> >> :'(
> >>> >> >>> >
> >>> >> >>> > Cheers, Fokko
> >>> >> >>> >
> >>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
> >>> criccomini@apache.org>:
> >>> >> >>> >
> >>> >> >>> > > @bolke, open issues on the Github repo, please.
> >>> >> >>> > >
> >>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> >>> >> bdbruin@gmail.com>
> >>> >> >>> > > wrote:
> >>> >> >>> > >
> >>> >> >>> > > > Chris,
> >>> >> >>> > > >
> >>> >> >>> > > > Do you want us to report bugs somewhere (I have
> encountered
> >>> a
> >>> >> >>> few)? Or
> >>> >> >>> > > > just generic user experiences posted here?
> >>> >> >>> > > >
> >>> >> >>> > > > Cheers
> >>> >> >>> > > > Bolke
> >>> >> >>> > > >
> >>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> >>> >> criccomini@apache.org
> >>> >> >>> >
> >>> >> >>> > > wrote:
> >>> >> >>> > > > >
> >>> >> >>> > > > > Hey all,
> >>> >> >>> > > > >
> >>> >> >>> > > > > I know the weekend is coming up, and for those of us in
> >>> the
> >>> >> US,
> >>> >> >>> next
> >>> >> >>> > > week
> >>> >> >>> > > > > is a bit of a slow holiday week. Would love to get some
> >>> >> feedback
> >>> >> >>> from
> >>> >> >>> > > > > everyone on this. The goal would ideally to be to
> >>> converge on
> >>> >> >>> this
> >>> >> >>> > and
> >>> >> >>> > > > > eventually replace the existing Airflow UI with this
> one.
> >>> >> >>> > > > >
> >>> >> >>> > > > > Cheers,
> >>> >> >>> > > > > Chris
> >>> >> >>> > > > >
> >>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <
> joyg@wepay.com>
> >>> >> wrote:
> >>> >> >>> > > > >
> >>> >> >>> > > > >> Hi guys.
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> I've been working on moving airflow from Flask-Admin to
> >>> >> >>> > > Flask-AppBuilder
> >>> >> >>> > > > >> for RBAC
> >>> >> >>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> >>> >> >>> > > > Airflow+RBAC+proposal
> >>> >> >>> > > > >>> ,
> >>> >> >>> > > > >> check it out at https://github.com/wepay/airfl
> >>> ow-webserver.
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> It's still a work-in-progress, but most features you
> see
> >>> in
> >>> >> the
> >>> >> >>> > > > webserver
> >>> >> >>> > > > >> UI today is available there. For those who are
> >>> interested in
> >>> >> >>> RBAC,
> >>> >> >>> > I'd
> >>> >> >>> > > > love
> >>> >> >>> > > > >> to get some early feedback in terms of the following:
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> >>> >> >>> > > > >> - Setup issues
> >>> >> >>> > > > >> - Ease of integration with third party auth (i.e. LDAP,
> >>> AD,
> >>> >> >>> OAuth,
> >>> >> >>> > > etc.)
> >>> >> >>> > > > >> - Any other thoughts/concerns
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> Thanks a lot!
> >>> >> >>> > > > >>
> >>> >> >>> > > > >> Cheers,
> >>> >> >>> > > > >> Joy
> >>> >> >>> > > > >>
> >>> >> >>> > > >
> >>> >> >>> > > >
> >>> >> >>> > >
> >>> >> >>> >
> >>> >> >>>
> >>> >> >>
> >>> >> >>
> >>> >> >
> >>> >>
> >>> >
> >>> >
> >>>
> >>
> >>
> >
>
Re: RBAC Update
Posted by Joy Gao <jo...@wepay.com>.
Hey guys!
The RBAC UI <https://github.com/apache/incubator-airflow/pull/3015> has
been merged to master. I'm looking forward to early adopters' feedback and
bug reports. I also hope to have more folks helping out with the RBAC UI,
especially with introducing DAG-Level access control, which is a feature
that a lot of people have been asking. If you are interested in helping out
with this effort, let's talk more!
This commit will be in the 1.10.0 release, and we are going to maintain
both UIs simultaneously for a short period of time. Once RBAC UI is stable
and battle-tested, we will deprecate the old UI and eventually remove it
from the repo (around Airflow 2.0.0 or 2.1.0 release). This is to prevent
two UIs from forking into separate paths, as that would become very
difficult to maintain.
Going forward while both UIs are up, if you are making a change to any
files in airflow/www/ (old UI), where applicable, please also make the
change to the airflow/www_rbac/ (new UI). If you rather not make changes in
both UIs, it is recommended that you only make the changes to the RBAC UI,
since that is the one we are maintaining in the long term.
I'm excited that the RBAC UI will be able to bring additional security to
Airflow, and with FAB framework in place we can look into leveraging it for
a unified set of APIs used by both UI and CLI.
Joy
On Thu, Feb 8, 2018 at 11:31 AM, Joy Gao <jo...@wepay.com> wrote:
> Hi folks,
>
> I have a PR <https://github.com/apache/incubator-airflow/pull/3015> out
> for the new UI. I've included instructions on how to test it out in the PR
> description. Looking forward to your feedbacks.
>
> Cheers,
> Joy
>
> On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
>
>> Thanks for the background info. Would be really awesome for you to have
>> PyPi access :D I'll make the change to have Airflow Webserver's FAB
>> dependency pointing to my fork for the mean time.
>>
>> For folks who are interested in RBAC, I will be giving a talk/demo at the Airflow
>> Meet-Up
>> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/>
>> next Monday. Happy to chat afterwards about it as well :)
>>
>> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>>> A bit of related history here:
>>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
>>>
>>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
>>> maximebeauchemin@gmail.com> wrote:
>>>
>>> > Given I have merge rights on FAB I could probably do another round of
>>> > review and get your PRs through. I would really like to get the main
>>> > maintainer's input on things that touch the core (composite-key
>>> support) as
>>> > he might have concerns/intuitions that we can't know about.
>>> >
>>> > I do not have Pypi access though so I cannot push new releases out. I
>>> > could ask for that.
>>> >
>>> > I've threatened to fork the project before, that's always an option.
>>> I've
>>> > noticed his involvement is sporadic and comes in bursts.
>>> >
>>> > In the meantime, you can have the dependency in Airflow Webserver
>>> pointing
>>> > straight to your fork.
>>> >
>>> > Max
>>> >
>>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
>>> >
>>> >> I just created a new webserver instance if you haven't gotten a
>>> chance to
>>> >> fiddle around with the new web UI and the RBAC configurations (thanks
>>> >> Maxime for getting started with this earlier!):
>>> >>
>>> >> http://104.209.38.171:8080/
>>> >>
>>> >> Admin Account
>>> >> username: admin
>>> >> password: admin
>>> >>
>>> >> Read-Only Account
>>> >> username: viewer
>>> >> password: password
>>> >>
>>> >>
>>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
>>> >>
>>> >> > Hi folks,
>>> >> >
>>> >> > Thanks for all the feedback regarding to the new Airflow Webserver
>>> UI
>>> >> > <https://github.com/wepay/airflow-webserver/>! I've been actively
>>> >> > addressing all the bugs that were raised on Github. So I want to
>>> take
>>> >> this
>>> >> > opportunity to discuss two issues coming up:
>>> >> >
>>> >> > The first issue is unaddressed PRs in FAB. If these PRs continue to
>>> stay
>>> >> > unaddressed, RBAC is blocked from making further progress. If this
>>> >> continue
>>> >> > to be an issue, I'm inclined to fork FAB, even though it's not
>>> >> idealistic.
>>> >> >
>>> >> >
>>> >> > - PR/631 <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>>> >> Binary
>>> >> > column support (merged, unreleased)
>>> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>>> >> > - PR/639 <https://github.com/dpgaspar/Flask-AppBuilder/pull/639>
>>> >> Composite
>>> >> > primary key support (unmerged)
>>> >> > - PR/655 <https://github.com/dpgaspar/Flask-AppBuilder/pull/655>
>>> >> Form
>>> >> > prefill support (unmerged)
>>> >> >
>>> >> >
>>> >> > The second issue is an open question about the next step of Airflow
>>> >> > Webserver itself. Here are the 3 potential directions we could
>>> take, and
>>> >> > I've added my thought on each.
>>> >> >
>>> >> > 1. Permanently keep Airflow Webserver as a separated package from
>>> >> Airflow,
>>> >> > and treat it as another UI option. Keep `www` in Airflow. Allow
>>> >> development
>>> >> > on both UIs.
>>> >> > *I'm not a fan of this. When there is an existing UI in Airflow,
>>> most
>>> >> > contributors would prefer to maintain the official version that is
>>> >> > installed out-of-the-box. **Having a second UI outside of Airflow
>>> will
>>> >> > make maintaining it very difficult, leading to an eventual death of
>>> the
>>> >> new
>>> >> > UI :(*
>>> >> >
>>> >> > 2. Permanently keep Airflow Webserver as a separated package from
>>> >> Airflow,
>>> >> > but freeze all development on `www` and direct all future UI
>>> >> development
>>> >> > to Airflow Webserver, eventually removing `www` completely when
>>> Airflow
>>> >> > Webserver is stable.
>>> >> > *I'm not a fan of this either. First of all, the views and models
>>> are
>>> >> > tightly coupled in both old and new UI; until we have a full-fledged
>>> >> REST
>>> >> > API to build the UI (and cli) on top of it, separating them to a
>>> >> separate
>>> >> > package now will potentially cause dependency issues and add
>>> >> complication
>>> >> > to our release cycle. **Secondly, **majority of Airflow users run
>>> >> Airflow
>>> >> > with the UI; it's one of Airflow's best features. Separating UI out
>>> of
>>> >> > Airflow core will complicate setup and configuration, while making
>>> >> Airflow
>>> >> > core less complete.*
>>> >> >
>>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
>>> >> > development on `www`, eventually removing `www` completely when
>>> `www2`
>>> >> is
>>> >> > stable.
>>> >> > *This makes the most sense to me. Airflow Webserver is developed
>>> with
>>> >> the
>>> >> > goal of feature parity to the current UI, plus additional RBAC
>>> >> capability,
>>> >> > in hope to replace the old UI completely. Yes, this means there
>>> will be
>>> >> a
>>> >> > short period of having to maintain two UIs, but once we freeze
>>> >> development
>>> >> > on www, it shouldn't be a concern for long.*
>>> >> >
>>> >> > I'd love to hear everyone's thoughts on this! I'm excited about
>>> bringing
>>> >> > RBAC to airflow and I hope it's something others will find useful as
>>> >> well!
>>> >> >
>>> >> > Cheers,
>>> >> > Joy
>>> >> >
>>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
>>> >> >
>>> >> >> Thank you everyone for the active feedback so far, and thanks for
>>> >> setting
>>> >> >> up the demo Maxime!
>>> >> >>
>>> >> >> Going to work on pruning through the issues in the upcoming days.
>>> >> >>
>>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I
>>> can
>>> >> >> look into it? Otherwise I'll wait until it's down again =P
>>> >> >>
>>> >> >> Cheers,
>>> >> >>
>>> >> >> Joy
>>> >> >>
>>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
>>> >> >> maximebeauchemin@gmail.com> wrote:
>>> >> >>
>>> >> >>> I just restarted it, not sure how long it will take to get in a
>>> bad
>>> >> state
>>> >> >>> again...
>>> >> >>>
>>> >> >>> Max
>>> >> >>>
>>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
>>> >> <fokko@driesprong.frl
>>> >> >>> >
>>> >> >>> wrote:
>>> >> >>>
>>> >> >>> > Good morning,
>>> >> >>> >
>>> >> >>> > The demo provided by Max is down, it throws a
>>> SQLAlchemyexception
>>> >> :'(
>>> >> >>> >
>>> >> >>> > Cheers, Fokko
>>> >> >>> >
>>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
>>> criccomini@apache.org>:
>>> >> >>> >
>>> >> >>> > > @bolke, open issues on the Github repo, please.
>>> >> >>> > >
>>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
>>> >> bdbruin@gmail.com>
>>> >> >>> > > wrote:
>>> >> >>> > >
>>> >> >>> > > > Chris,
>>> >> >>> > > >
>>> >> >>> > > > Do you want us to report bugs somewhere (I have encountered
>>> a
>>> >> >>> few)? Or
>>> >> >>> > > > just generic user experiences posted here?
>>> >> >>> > > >
>>> >> >>> > > > Cheers
>>> >> >>> > > > Bolke
>>> >> >>> > > >
>>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
>>> >> criccomini@apache.org
>>> >> >>> >
>>> >> >>> > > wrote:
>>> >> >>> > > > >
>>> >> >>> > > > > Hey all,
>>> >> >>> > > > >
>>> >> >>> > > > > I know the weekend is coming up, and for those of us in
>>> the
>>> >> US,
>>> >> >>> next
>>> >> >>> > > week
>>> >> >>> > > > > is a bit of a slow holiday week. Would love to get some
>>> >> feedback
>>> >> >>> from
>>> >> >>> > > > > everyone on this. The goal would ideally to be to
>>> converge on
>>> >> >>> this
>>> >> >>> > and
>>> >> >>> > > > > eventually replace the existing Airflow UI with this one.
>>> >> >>> > > > >
>>> >> >>> > > > > Cheers,
>>> >> >>> > > > > Chris
>>> >> >>> > > > >
>>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com>
>>> >> wrote:
>>> >> >>> > > > >
>>> >> >>> > > > >> Hi guys.
>>> >> >>> > > > >>
>>> >> >>> > > > >> I've been working on moving airflow from Flask-Admin to
>>> >> >>> > > Flask-AppBuilder
>>> >> >>> > > > >> for RBAC
>>> >> >>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
>>> >> >>> > > > Airflow+RBAC+proposal
>>> >> >>> > > > >>> ,
>>> >> >>> > > > >> check it out at https://github.com/wepay/airfl
>>> ow-webserver.
>>> >> >>> > > > >>
>>> >> >>> > > > >> It's still a work-in-progress, but most features you see
>>> in
>>> >> the
>>> >> >>> > > > webserver
>>> >> >>> > > > >> UI today is available there. For those who are
>>> interested in
>>> >> >>> RBAC,
>>> >> >>> > I'd
>>> >> >>> > > > love
>>> >> >>> > > > >> to get some early feedback in terms of the following:
>>> >> >>> > > > >>
>>> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
>>> >> >>> > > > >> - Setup issues
>>> >> >>> > > > >> - Ease of integration with third party auth (i.e. LDAP,
>>> AD,
>>> >> >>> OAuth,
>>> >> >>> > > etc.)
>>> >> >>> > > > >> - Any other thoughts/concerns
>>> >> >>> > > > >>
>>> >> >>> > > > >> Thanks a lot!
>>> >> >>> > > > >>
>>> >> >>> > > > >> Cheers,
>>> >> >>> > > > >> Joy
>>> >> >>> > > > >>
>>> >> >>> > > >
>>> >> >>> > > >
>>> >> >>> > >
>>> >> >>> >
>>> >> >>>
>>> >> >>
>>> >> >>
>>> >> >
>>> >>
>>> >
>>> >
>>>
>>
>>
>
Re: RBAC Update
Posted by Joy Gao <jo...@wepay.com>.
Hi folks,
I have a PR <https://github.com/apache/incubator-airflow/pull/3015> out for
the new UI. I've included instructions on how to test it out in the PR
description. Looking forward to your feedbacks.
Cheers,
Joy
On Fri, Dec 1, 2017 at 6:18 PM, Joy Gao <jo...@wepay.com> wrote:
> Thanks for the background info. Would be really awesome for you to have
> PyPi access :D I'll make the change to have Airflow Webserver's FAB
> dependency pointing to my fork for the mean time.
>
> For folks who are interested in RBAC, I will be giving a talk/demo at the Airflow
> Meet-Up
> <https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/>
> next Monday. Happy to chat afterwards about it as well :)
>
> On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
>> A bit of related history here:
>> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
>>
>> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>> > Given I have merge rights on FAB I could probably do another round of
>> > review and get your PRs through. I would really like to get the main
>> > maintainer's input on things that touch the core (composite-key
>> support) as
>> > he might have concerns/intuitions that we can't know about.
>> >
>> > I do not have Pypi access though so I cannot push new releases out. I
>> > could ask for that.
>> >
>> > I've threatened to fork the project before, that's always an option.
>> I've
>> > noticed his involvement is sporadic and comes in bursts.
>> >
>> > In the meantime, you can have the dependency in Airflow Webserver
>> pointing
>> > straight to your fork.
>> >
>> > Max
>> >
>> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
>> >
>> >> I just created a new webserver instance if you haven't gotten a chance
>> to
>> >> fiddle around with the new web UI and the RBAC configurations (thanks
>> >> Maxime for getting started with this earlier!):
>> >>
>> >> http://104.209.38.171:8080/
>> >>
>> >> Admin Account
>> >> username: admin
>> >> password: admin
>> >>
>> >> Read-Only Account
>> >> username: viewer
>> >> password: password
>> >>
>> >>
>> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
>> >>
>> >> > Hi folks,
>> >> >
>> >> > Thanks for all the feedback regarding to the new Airflow Webserver UI
>> >> > <https://github.com/wepay/airflow-webserver/>! I've been actively
>> >> > addressing all the bugs that were raised on Github. So I want to take
>> >> this
>> >> > opportunity to discuss two issues coming up:
>> >> >
>> >> > The first issue is unaddressed PRs in FAB. If these PRs continue to
>> stay
>> >> > unaddressed, RBAC is blocked from making further progress. If this
>> >> continue
>> >> > to be an issue, I'm inclined to fork FAB, even though it's not
>> >> idealistic.
>> >> >
>> >> >
>> >> > - PR/631 <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>> >> Binary
>> >> > column support (merged, unreleased)
>> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>> >> > - PR/639 <https://github.com/dpgaspar/Flask-AppBuilder/pull/639>
>> >> Composite
>> >> > primary key support (unmerged)
>> >> > - PR/655 <https://github.com/dpgaspar/Flask-AppBuilder/pull/655>
>> >> Form
>> >> > prefill support (unmerged)
>> >> >
>> >> >
>> >> > The second issue is an open question about the next step of Airflow
>> >> > Webserver itself. Here are the 3 potential directions we could take,
>> and
>> >> > I've added my thought on each.
>> >> >
>> >> > 1. Permanently keep Airflow Webserver as a separated package from
>> >> Airflow,
>> >> > and treat it as another UI option. Keep `www` in Airflow. Allow
>> >> development
>> >> > on both UIs.
>> >> > *I'm not a fan of this. When there is an existing UI in Airflow, most
>> >> > contributors would prefer to maintain the official version that is
>> >> > installed out-of-the-box. **Having a second UI outside of Airflow
>> will
>> >> > make maintaining it very difficult, leading to an eventual death of
>> the
>> >> new
>> >> > UI :(*
>> >> >
>> >> > 2. Permanently keep Airflow Webserver as a separated package from
>> >> Airflow,
>> >> > but freeze all development on `www` and direct all future UI
>> >> development
>> >> > to Airflow Webserver, eventually removing `www` completely when
>> Airflow
>> >> > Webserver is stable.
>> >> > *I'm not a fan of this either. First of all, the views and models are
>> >> > tightly coupled in both old and new UI; until we have a full-fledged
>> >> REST
>> >> > API to build the UI (and cli) on top of it, separating them to a
>> >> separate
>> >> > package now will potentially cause dependency issues and add
>> >> complication
>> >> > to our release cycle. **Secondly, **majority of Airflow users run
>> >> Airflow
>> >> > with the UI; it's one of Airflow's best features. Separating UI out
>> of
>> >> > Airflow core will complicate setup and configuration, while making
>> >> Airflow
>> >> > core less complete.*
>> >> >
>> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
>> >> > development on `www`, eventually removing `www` completely when
>> `www2`
>> >> is
>> >> > stable.
>> >> > *This makes the most sense to me. Airflow Webserver is developed with
>> >> the
>> >> > goal of feature parity to the current UI, plus additional RBAC
>> >> capability,
>> >> > in hope to replace the old UI completely. Yes, this means there will
>> be
>> >> a
>> >> > short period of having to maintain two UIs, but once we freeze
>> >> development
>> >> > on www, it shouldn't be a concern for long.*
>> >> >
>> >> > I'd love to hear everyone's thoughts on this! I'm excited about
>> bringing
>> >> > RBAC to airflow and I hope it's something others will find useful as
>> >> well!
>> >> >
>> >> > Cheers,
>> >> > Joy
>> >> >
>> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
>> >> >
>> >> >> Thank you everyone for the active feedback so far, and thanks for
>> >> setting
>> >> >> up the demo Maxime!
>> >> >>
>> >> >> Going to work on pruning through the issues in the upcoming days.
>> >> >>
>> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I
>> can
>> >> >> look into it? Otherwise I'll wait until it's down again =P
>> >> >>
>> >> >> Cheers,
>> >> >>
>> >> >> Joy
>> >> >>
>> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
>> >> >> maximebeauchemin@gmail.com> wrote:
>> >> >>
>> >> >>> I just restarted it, not sure how long it will take to get in a bad
>> >> state
>> >> >>> again...
>> >> >>>
>> >> >>> Max
>> >> >>>
>> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
>> >> <fokko@driesprong.frl
>> >> >>> >
>> >> >>> wrote:
>> >> >>>
>> >> >>> > Good morning,
>> >> >>> >
>> >> >>> > The demo provided by Max is down, it throws a SQLAlchemyexception
>> >> :'(
>> >> >>> >
>> >> >>> > Cheers, Fokko
>> >> >>> >
>> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <
>> criccomini@apache.org>:
>> >> >>> >
>> >> >>> > > @bolke, open issues on the Github repo, please.
>> >> >>> > >
>> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
>> >> bdbruin@gmail.com>
>> >> >>> > > wrote:
>> >> >>> > >
>> >> >>> > > > Chris,
>> >> >>> > > >
>> >> >>> > > > Do you want us to report bugs somewhere (I have encountered a
>> >> >>> few)? Or
>> >> >>> > > > just generic user experiences posted here?
>> >> >>> > > >
>> >> >>> > > > Cheers
>> >> >>> > > > Bolke
>> >> >>> > > >
>> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
>> >> criccomini@apache.org
>> >> >>> >
>> >> >>> > > wrote:
>> >> >>> > > > >
>> >> >>> > > > > Hey all,
>> >> >>> > > > >
>> >> >>> > > > > I know the weekend is coming up, and for those of us in the
>> >> US,
>> >> >>> next
>> >> >>> > > week
>> >> >>> > > > > is a bit of a slow holiday week. Would love to get some
>> >> feedback
>> >> >>> from
>> >> >>> > > > > everyone on this. The goal would ideally to be to converge
>> on
>> >> >>> this
>> >> >>> > and
>> >> >>> > > > > eventually replace the existing Airflow UI with this one.
>> >> >>> > > > >
>> >> >>> > > > > Cheers,
>> >> >>> > > > > Chris
>> >> >>> > > > >
>> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com>
>> >> wrote:
>> >> >>> > > > >
>> >> >>> > > > >> Hi guys.
>> >> >>> > > > >>
>> >> >>> > > > >> I've been working on moving airflow from Flask-Admin to
>> >> >>> > > Flask-AppBuilder
>> >> >>> > > > >> for RBAC
>> >> >>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
>> >> >>> > > > Airflow+RBAC+proposal
>> >> >>> > > > >>> ,
>> >> >>> > > > >> check it out at https://github.com/wepay/airfl
>> ow-webserver.
>> >> >>> > > > >>
>> >> >>> > > > >> It's still a work-in-progress, but most features you see
>> in
>> >> the
>> >> >>> > > > webserver
>> >> >>> > > > >> UI today is available there. For those who are interested
>> in
>> >> >>> RBAC,
>> >> >>> > I'd
>> >> >>> > > > love
>> >> >>> > > > >> to get some early feedback in terms of the following:
>> >> >>> > > > >>
>> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
>> >> >>> > > > >> - Setup issues
>> >> >>> > > > >> - Ease of integration with third party auth (i.e. LDAP,
>> AD,
>> >> >>> OAuth,
>> >> >>> > > etc.)
>> >> >>> > > > >> - Any other thoughts/concerns
>> >> >>> > > > >>
>> >> >>> > > > >> Thanks a lot!
>> >> >>> > > > >>
>> >> >>> > > > >> Cheers,
>> >> >>> > > > >> Joy
>> >> >>> > > > >>
>> >> >>> > > >
>> >> >>> > > >
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >
>> >
>>
>
>
Re: RBAC Update
Posted by Joy Gao <jo...@wepay.com>.
Thanks for the background info. Would be really awesome for you to have
PyPi access :D I'll make the change to have Airflow Webserver's FAB
dependency pointing to my fork for the mean time.
For folks who are interested in RBAC, I will be giving a talk/demo at
the Airflow
Meet-Up
<https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/>
next Monday. Happy to chat afterwards about it as well :)
On Thu, Nov 30, 2017 at 8:36 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> A bit of related history here:
> https://github.com/dpgaspar/Flask-AppBuilder/issues/399
>
> On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > Given I have merge rights on FAB I could probably do another round of
> > review and get your PRs through. I would really like to get the main
> > maintainer's input on things that touch the core (composite-key support)
> as
> > he might have concerns/intuitions that we can't know about.
> >
> > I do not have Pypi access though so I cannot push new releases out. I
> > could ask for that.
> >
> > I've threatened to fork the project before, that's always an option. I've
> > noticed his involvement is sporadic and comes in bursts.
> >
> > In the meantime, you can have the dependency in Airflow Webserver
> pointing
> > straight to your fork.
> >
> > Max
> >
> > On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> >> I just created a new webserver instance if you haven't gotten a chance
> to
> >> fiddle around with the new web UI and the RBAC configurations (thanks
> >> Maxime for getting started with this earlier!):
> >>
> >> http://104.209.38.171:8080/
> >>
> >> Admin Account
> >> username: admin
> >> password: admin
> >>
> >> Read-Only Account
> >> username: viewer
> >> password: password
> >>
> >>
> >> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
> >>
> >> > Hi folks,
> >> >
> >> > Thanks for all the feedback regarding to the new Airflow Webserver UI
> >> > <https://github.com/wepay/airflow-webserver/>! I've been actively
> >> > addressing all the bugs that were raised on Github. So I want to take
> >> this
> >> > opportunity to discuss two issues coming up:
> >> >
> >> > The first issue is unaddressed PRs in FAB. If these PRs continue to
> stay
> >> > unaddressed, RBAC is blocked from making further progress. If this
> >> continue
> >> > to be an issue, I'm inclined to fork FAB, even though it's not
> >> idealistic.
> >> >
> >> >
> >> > - PR/631 <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> >> Binary
> >> > column support (merged, unreleased)
> >> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> >> > - PR/639 <https://github.com/dpgaspar/Flask-AppBuilder/pull/639>
> >> Composite
> >> > primary key support (unmerged)
> >> > - PR/655 <https://github.com/dpgaspar/Flask-AppBuilder/pull/655>
> >> Form
> >> > prefill support (unmerged)
> >> >
> >> >
> >> > The second issue is an open question about the next step of Airflow
> >> > Webserver itself. Here are the 3 potential directions we could take,
> and
> >> > I've added my thought on each.
> >> >
> >> > 1. Permanently keep Airflow Webserver as a separated package from
> >> Airflow,
> >> > and treat it as another UI option. Keep `www` in Airflow. Allow
> >> development
> >> > on both UIs.
> >> > *I'm not a fan of this. When there is an existing UI in Airflow, most
> >> > contributors would prefer to maintain the official version that is
> >> > installed out-of-the-box. **Having a second UI outside of Airflow will
> >> > make maintaining it very difficult, leading to an eventual death of
> the
> >> new
> >> > UI :(*
> >> >
> >> > 2. Permanently keep Airflow Webserver as a separated package from
> >> Airflow,
> >> > but freeze all development on `www` and direct all future UI
> >> development
> >> > to Airflow Webserver, eventually removing `www` completely when
> Airflow
> >> > Webserver is stable.
> >> > *I'm not a fan of this either. First of all, the views and models are
> >> > tightly coupled in both old and new UI; until we have a full-fledged
> >> REST
> >> > API to build the UI (and cli) on top of it, separating them to a
> >> separate
> >> > package now will potentially cause dependency issues and add
> >> complication
> >> > to our release cycle. **Secondly, **majority of Airflow users run
> >> Airflow
> >> > with the UI; it's one of Airflow's best features. Separating UI out of
> >> > Airflow core will complicate setup and configuration, while making
> >> Airflow
> >> > core less complete.*
> >> >
> >> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
> >> > development on `www`, eventually removing `www` completely when `www2`
> >> is
> >> > stable.
> >> > *This makes the most sense to me. Airflow Webserver is developed with
> >> the
> >> > goal of feature parity to the current UI, plus additional RBAC
> >> capability,
> >> > in hope to replace the old UI completely. Yes, this means there will
> be
> >> a
> >> > short period of having to maintain two UIs, but once we freeze
> >> development
> >> > on www, it shouldn't be a concern for long.*
> >> >
> >> > I'd love to hear everyone's thoughts on this! I'm excited about
> bringing
> >> > RBAC to airflow and I hope it's something others will find useful as
> >> well!
> >> >
> >> > Cheers,
> >> > Joy
> >> >
> >> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
> >> >
> >> >> Thank you everyone for the active feedback so far, and thanks for
> >> setting
> >> >> up the demo Maxime!
> >> >>
> >> >> Going to work on pruning through the issues in the upcoming days.
> >> >>
> >> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I can
> >> >> look into it? Otherwise I'll wait until it's down again =P
> >> >>
> >> >> Cheers,
> >> >>
> >> >> Joy
> >> >>
> >> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >> >> maximebeauchemin@gmail.com> wrote:
> >> >>
> >> >>> I just restarted it, not sure how long it will take to get in a bad
> >> state
> >> >>> again...
> >> >>>
> >> >>> Max
> >> >>>
> >> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> >> <fokko@driesprong.frl
> >> >>> >
> >> >>> wrote:
> >> >>>
> >> >>> > Good morning,
> >> >>> >
> >> >>> > The demo provided by Max is down, it throws a SQLAlchemyexception
> >> :'(
> >> >>> >
> >> >>> > Cheers, Fokko
> >> >>> >
> >> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <criccomini@apache.org
> >:
> >> >>> >
> >> >>> > > @bolke, open issues on the Github repo, please.
> >> >>> > >
> >> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> >> bdbruin@gmail.com>
> >> >>> > > wrote:
> >> >>> > >
> >> >>> > > > Chris,
> >> >>> > > >
> >> >>> > > > Do you want us to report bugs somewhere (I have encountered a
> >> >>> few)? Or
> >> >>> > > > just generic user experiences posted here?
> >> >>> > > >
> >> >>> > > > Cheers
> >> >>> > > > Bolke
> >> >>> > > >
> >> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> >> criccomini@apache.org
> >> >>> >
> >> >>> > > wrote:
> >> >>> > > > >
> >> >>> > > > > Hey all,
> >> >>> > > > >
> >> >>> > > > > I know the weekend is coming up, and for those of us in the
> >> US,
> >> >>> next
> >> >>> > > week
> >> >>> > > > > is a bit of a slow holiday week. Would love to get some
> >> feedback
> >> >>> from
> >> >>> > > > > everyone on this. The goal would ideally to be to converge
> on
> >> >>> this
> >> >>> > and
> >> >>> > > > > eventually replace the existing Airflow UI with this one.
> >> >>> > > > >
> >> >>> > > > > Cheers,
> >> >>> > > > > Chris
> >> >>> > > > >
> >> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com>
> >> wrote:
> >> >>> > > > >
> >> >>> > > > >> Hi guys.
> >> >>> > > > >>
> >> >>> > > > >> I've been working on moving airflow from Flask-Admin to
> >> >>> > > Flask-AppBuilder
> >> >>> > > > >> for RBAC
> >> >>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> >> >>> > > > Airflow+RBAC+proposal
> >> >>> > > > >>> ,
> >> >>> > > > >> check it out at https://github.com/wepay/airflow-webserver
> .
> >> >>> > > > >>
> >> >>> > > > >> It's still a work-in-progress, but most features you see in
> >> the
> >> >>> > > > webserver
> >> >>> > > > >> UI today is available there. For those who are interested
> in
> >> >>> RBAC,
> >> >>> > I'd
> >> >>> > > > love
> >> >>> > > > >> to get some early feedback in terms of the following:
> >> >>> > > > >>
> >> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> >> >>> > > > >> - Setup issues
> >> >>> > > > >> - Ease of integration with third party auth (i.e. LDAP, AD,
> >> >>> OAuth,
> >> >>> > > etc.)
> >> >>> > > > >> - Any other thoughts/concerns
> >> >>> > > > >>
> >> >>> > > > >> Thanks a lot!
> >> >>> > > > >>
> >> >>> > > > >> Cheers,
> >> >>> > > > >> Joy
> >> >>> > > > >>
> >> >>> > > >
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>>
> >> >>
> >> >>
> >> >
> >>
> >
> >
>
Re: RBAC Update
Posted by Maxime Beauchemin <ma...@gmail.com>.
A bit of related history here:
https://github.com/dpgaspar/Flask-AppBuilder/issues/399
On Thu, Nov 30, 2017 at 8:33 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> Given I have merge rights on FAB I could probably do another round of
> review and get your PRs through. I would really like to get the main
> maintainer's input on things that touch the core (composite-key support) as
> he might have concerns/intuitions that we can't know about.
>
> I do not have Pypi access though so I cannot push new releases out. I
> could ask for that.
>
> I've threatened to fork the project before, that's always an option. I've
> noticed his involvement is sporadic and comes in bursts.
>
> In the meantime, you can have the dependency in Airflow Webserver pointing
> straight to your fork.
>
> Max
>
> On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
>
>> I just created a new webserver instance if you haven't gotten a chance to
>> fiddle around with the new web UI and the RBAC configurations (thanks
>> Maxime for getting started with this earlier!):
>>
>> http://104.209.38.171:8080/
>>
>> Admin Account
>> username: admin
>> password: admin
>>
>> Read-Only Account
>> username: viewer
>> password: password
>>
>>
>> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
>>
>> > Hi folks,
>> >
>> > Thanks for all the feedback regarding to the new Airflow Webserver UI
>> > <https://github.com/wepay/airflow-webserver/>! I've been actively
>> > addressing all the bugs that were raised on Github. So I want to take
>> this
>> > opportunity to discuss two issues coming up:
>> >
>> > The first issue is unaddressed PRs in FAB. If these PRs continue to stay
>> > unaddressed, RBAC is blocked from making further progress. If this
>> continue
>> > to be an issue, I'm inclined to fork FAB, even though it's not
>> idealistic.
>> >
>> >
>> > - PR/631 <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>> Binary
>> > column support (merged, unreleased)
>> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
>> > - PR/639 <https://github.com/dpgaspar/Flask-AppBuilder/pull/639>
>> Composite
>> > primary key support (unmerged)
>> > - PR/655 <https://github.com/dpgaspar/Flask-AppBuilder/pull/655>
>> Form
>> > prefill support (unmerged)
>> >
>> >
>> > The second issue is an open question about the next step of Airflow
>> > Webserver itself. Here are the 3 potential directions we could take, and
>> > I've added my thought on each.
>> >
>> > 1. Permanently keep Airflow Webserver as a separated package from
>> Airflow,
>> > and treat it as another UI option. Keep `www` in Airflow. Allow
>> development
>> > on both UIs.
>> > *I'm not a fan of this. When there is an existing UI in Airflow, most
>> > contributors would prefer to maintain the official version that is
>> > installed out-of-the-box. **Having a second UI outside of Airflow will
>> > make maintaining it very difficult, leading to an eventual death of the
>> new
>> > UI :(*
>> >
>> > 2. Permanently keep Airflow Webserver as a separated package from
>> Airflow,
>> > but freeze all development on `www` and direct all future UI
>> development
>> > to Airflow Webserver, eventually removing `www` completely when Airflow
>> > Webserver is stable.
>> > *I'm not a fan of this either. First of all, the views and models are
>> > tightly coupled in both old and new UI; until we have a full-fledged
>> REST
>> > API to build the UI (and cli) on top of it, separating them to a
>> separate
>> > package now will potentially cause dependency issues and add
>> complication
>> > to our release cycle. **Secondly, **majority of Airflow users run
>> Airflow
>> > with the UI; it's one of Airflow's best features. Separating UI out of
>> > Airflow core will complicate setup and configuration, while making
>> Airflow
>> > core less complete.*
>> >
>> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
>> > development on `www`, eventually removing `www` completely when `www2`
>> is
>> > stable.
>> > *This makes the most sense to me. Airflow Webserver is developed with
>> the
>> > goal of feature parity to the current UI, plus additional RBAC
>> capability,
>> > in hope to replace the old UI completely. Yes, this means there will be
>> a
>> > short period of having to maintain two UIs, but once we freeze
>> development
>> > on www, it shouldn't be a concern for long.*
>> >
>> > I'd love to hear everyone's thoughts on this! I'm excited about bringing
>> > RBAC to airflow and I hope it's something others will find useful as
>> well!
>> >
>> > Cheers,
>> > Joy
>> >
>> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
>> >
>> >> Thank you everyone for the active feedback so far, and thanks for
>> setting
>> >> up the demo Maxime!
>> >>
>> >> Going to work on pruning through the issues in the upcoming days.
>> >>
>> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I can
>> >> look into it? Otherwise I'll wait until it's down again =P
>> >>
>> >> Cheers,
>> >>
>> >> Joy
>> >>
>> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
>> >> maximebeauchemin@gmail.com> wrote:
>> >>
>> >>> I just restarted it, not sure how long it will take to get in a bad
>> state
>> >>> again...
>> >>>
>> >>> Max
>> >>>
>> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
>> <fokko@driesprong.frl
>> >>> >
>> >>> wrote:
>> >>>
>> >>> > Good morning,
>> >>> >
>> >>> > The demo provided by Max is down, it throws a SQLAlchemyexception
>> :'(
>> >>> >
>> >>> > Cheers, Fokko
>> >>> >
>> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <cr...@apache.org>:
>> >>> >
>> >>> > > @bolke, open issues on the Github repo, please.
>> >>> > >
>> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
>> bdbruin@gmail.com>
>> >>> > > wrote:
>> >>> > >
>> >>> > > > Chris,
>> >>> > > >
>> >>> > > > Do you want us to report bugs somewhere (I have encountered a
>> >>> few)? Or
>> >>> > > > just generic user experiences posted here?
>> >>> > > >
>> >>> > > > Cheers
>> >>> > > > Bolke
>> >>> > > >
>> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
>> criccomini@apache.org
>> >>> >
>> >>> > > wrote:
>> >>> > > > >
>> >>> > > > > Hey all,
>> >>> > > > >
>> >>> > > > > I know the weekend is coming up, and for those of us in the
>> US,
>> >>> next
>> >>> > > week
>> >>> > > > > is a bit of a slow holiday week. Would love to get some
>> feedback
>> >>> from
>> >>> > > > > everyone on this. The goal would ideally to be to converge on
>> >>> this
>> >>> > and
>> >>> > > > > eventually replace the existing Airflow UI with this one.
>> >>> > > > >
>> >>> > > > > Cheers,
>> >>> > > > > Chris
>> >>> > > > >
>> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com>
>> wrote:
>> >>> > > > >
>> >>> > > > >> Hi guys.
>> >>> > > > >>
>> >>> > > > >> I've been working on moving airflow from Flask-Admin to
>> >>> > > Flask-AppBuilder
>> >>> > > > >> for RBAC
>> >>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
>> >>> > > > Airflow+RBAC+proposal
>> >>> > > > >>> ,
>> >>> > > > >> check it out at https://github.com/wepay/airflow-webserver.
>> >>> > > > >>
>> >>> > > > >> It's still a work-in-progress, but most features you see in
>> the
>> >>> > > > webserver
>> >>> > > > >> UI today is available there. For those who are interested in
>> >>> RBAC,
>> >>> > I'd
>> >>> > > > love
>> >>> > > > >> to get some early feedback in terms of the following:
>> >>> > > > >>
>> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
>> >>> > > > >> - Setup issues
>> >>> > > > >> - Ease of integration with third party auth (i.e. LDAP, AD,
>> >>> OAuth,
>> >>> > > etc.)
>> >>> > > > >> - Any other thoughts/concerns
>> >>> > > > >>
>> >>> > > > >> Thanks a lot!
>> >>> > > > >>
>> >>> > > > >> Cheers,
>> >>> > > > >> Joy
>> >>> > > > >>
>> >>> > > >
>> >>> > > >
>> >>> > >
>> >>> >
>> >>>
>> >>
>> >>
>> >
>>
>
>
Re: RBAC Update
Posted by Maxime Beauchemin <ma...@gmail.com>.
Given I have merge rights on FAB I could probably do another round of
review and get your PRs through. I would really like to get the main
maintainer's input on things that touch the core (composite-key support) as
he might have concerns/intuitions that we can't know about.
I do not have Pypi access though so I cannot push new releases out. I could
ask for that.
I've threatened to fork the project before, that's always an option. I've
noticed his involvement is sporadic and comes in bursts.
In the meantime, you can have the dependency in Airflow Webserver pointing
straight to your fork.
Max
On Wed, Nov 29, 2017 at 7:02 PM, Joy Gao <jo...@wepay.com> wrote:
> I just created a new webserver instance if you haven't gotten a chance to
> fiddle around with the new web UI and the RBAC configurations (thanks
> Maxime for getting started with this earlier!):
>
> http://104.209.38.171:8080/
>
> Admin Account
> username: admin
> password: admin
>
> Read-Only Account
> username: viewer
> password: password
>
>
> On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hi folks,
> >
> > Thanks for all the feedback regarding to the new Airflow Webserver UI
> > <https://github.com/wepay/airflow-webserver/>! I've been actively
> > addressing all the bugs that were raised on Github. So I want to take
> this
> > opportunity to discuss two issues coming up:
> >
> > The first issue is unaddressed PRs in FAB. If these PRs continue to stay
> > unaddressed, RBAC is blocked from making further progress. If this
> continue
> > to be an issue, I'm inclined to fork FAB, even though it's not
> idealistic.
> >
> >
> > - PR/631 <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> Binary
> > column support (merged, unreleased)
> > <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> > - PR/639 <https://github.com/dpgaspar/Flask-AppBuilder/pull/639>
> Composite
> > primary key support (unmerged)
> > - PR/655 <https://github.com/dpgaspar/Flask-AppBuilder/pull/655> Form
> > prefill support (unmerged)
> >
> >
> > The second issue is an open question about the next step of Airflow
> > Webserver itself. Here are the 3 potential directions we could take, and
> > I've added my thought on each.
> >
> > 1. Permanently keep Airflow Webserver as a separated package from
> Airflow,
> > and treat it as another UI option. Keep `www` in Airflow. Allow
> development
> > on both UIs.
> > *I'm not a fan of this. When there is an existing UI in Airflow, most
> > contributors would prefer to maintain the official version that is
> > installed out-of-the-box. **Having a second UI outside of Airflow will
> > make maintaining it very difficult, leading to an eventual death of the
> new
> > UI :(*
> >
> > 2. Permanently keep Airflow Webserver as a separated package from
> Airflow,
> > but freeze all development on `www` and direct all future UI development
> > to Airflow Webserver, eventually removing `www` completely when Airflow
> > Webserver is stable.
> > *I'm not a fan of this either. First of all, the views and models are
> > tightly coupled in both old and new UI; until we have a full-fledged REST
> > API to build the UI (and cli) on top of it, separating them to a separate
> > package now will potentially cause dependency issues and add complication
> > to our release cycle. **Secondly, **majority of Airflow users run Airflow
> > with the UI; it's one of Airflow's best features. Separating UI out of
> > Airflow core will complicate setup and configuration, while making
> Airflow
> > core less complete.*
> >
> > 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
> > development on `www`, eventually removing `www` completely when `www2` is
> > stable.
> > *This makes the most sense to me. Airflow Webserver is developed with the
> > goal of feature parity to the current UI, plus additional RBAC
> capability,
> > in hope to replace the old UI completely. Yes, this means there will be a
> > short period of having to maintain two UIs, but once we freeze
> development
> > on www, it shouldn't be a concern for long.*
> >
> > I'd love to hear everyone's thoughts on this! I'm excited about bringing
> > RBAC to airflow and I hope it's something others will find useful as
> well!
> >
> > Cheers,
> > Joy
> >
> > On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
> >
> >> Thank you everyone for the active feedback so far, and thanks for
> setting
> >> up the demo Maxime!
> >>
> >> Going to work on pruning through the issues in the upcoming days.
> >>
> >> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I can
> >> look into it? Otherwise I'll wait until it's down again =P
> >>
> >> Cheers,
> >>
> >> Joy
> >>
> >> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> >> maximebeauchemin@gmail.com> wrote:
> >>
> >>> I just restarted it, not sure how long it will take to get in a bad
> state
> >>> again...
> >>>
> >>> Max
> >>>
> >>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko
> <fokko@driesprong.frl
> >>> >
> >>> wrote:
> >>>
> >>> > Good morning,
> >>> >
> >>> > The demo provided by Max is down, it throws a SQLAlchemyexception :'(
> >>> >
> >>> > Cheers, Fokko
> >>> >
> >>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <cr...@apache.org>:
> >>> >
> >>> > > @bolke, open issues on the Github repo, please.
> >>> > >
> >>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <
> bdbruin@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Chris,
> >>> > > >
> >>> > > > Do you want us to report bugs somewhere (I have encountered a
> >>> few)? Or
> >>> > > > just generic user experiences posted here?
> >>> > > >
> >>> > > > Cheers
> >>> > > > Bolke
> >>> > > >
> >>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <
> criccomini@apache.org
> >>> >
> >>> > > wrote:
> >>> > > > >
> >>> > > > > Hey all,
> >>> > > > >
> >>> > > > > I know the weekend is coming up, and for those of us in the US,
> >>> next
> >>> > > week
> >>> > > > > is a bit of a slow holiday week. Would love to get some
> feedback
> >>> from
> >>> > > > > everyone on this. The goal would ideally to be to converge on
> >>> this
> >>> > and
> >>> > > > > eventually replace the existing Airflow UI with this one.
> >>> > > > >
> >>> > > > > Cheers,
> >>> > > > > Chris
> >>> > > > >
> >>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com>
> wrote:
> >>> > > > >
> >>> > > > >> Hi guys.
> >>> > > > >>
> >>> > > > >> I've been working on moving airflow from Flask-Admin to
> >>> > > Flask-AppBuilder
> >>> > > > >> for RBAC
> >>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> >>> > > > Airflow+RBAC+proposal
> >>> > > > >>> ,
> >>> > > > >> check it out at https://github.com/wepay/airflow-webserver.
> >>> > > > >>
> >>> > > > >> It's still a work-in-progress, but most features you see in
> the
> >>> > > > webserver
> >>> > > > >> UI today is available there. For those who are interested in
> >>> RBAC,
> >>> > I'd
> >>> > > > love
> >>> > > > >> to get some early feedback in terms of the following:
> >>> > > > >>
> >>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> >>> > > > >> - Setup issues
> >>> > > > >> - Ease of integration with third party auth (i.e. LDAP, AD,
> >>> OAuth,
> >>> > > etc.)
> >>> > > > >> - Any other thoughts/concerns
> >>> > > > >>
> >>> > > > >> Thanks a lot!
> >>> > > > >>
> >>> > > > >> Cheers,
> >>> > > > >> Joy
> >>> > > > >>
> >>> > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
Re: RBAC Update
Posted by Joy Gao <jo...@wepay.com>.
I just created a new webserver instance if you haven't gotten a chance to
fiddle around with the new web UI and the RBAC configurations (thanks
Maxime for getting started with this earlier!):
http://104.209.38.171:8080/
Admin Account
username: admin
password: admin
Read-Only Account
username: viewer
password: password
On Wed, Nov 29, 2017 at 2:58 PM, Joy Gao <jo...@wepay.com> wrote:
> Hi folks,
>
> Thanks for all the feedback regarding to the new Airflow Webserver UI
> <https://github.com/wepay/airflow-webserver/>! I've been actively
> addressing all the bugs that were raised on Github. So I want to take this
> opportunity to discuss two issues coming up:
>
> The first issue is unaddressed PRs in FAB. If these PRs continue to stay
> unaddressed, RBAC is blocked from making further progress. If this continue
> to be an issue, I'm inclined to fork FAB, even though it's not idealistic.
>
>
> - PR/631 <https://github.com/dpgaspar/Flask-AppBuilder/pull/631> Binary
> column support (merged, unreleased)
> <https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
> - PR/639 <https://github.com/dpgaspar/Flask-AppBuilder/pull/639> Composite
> primary key support (unmerged)
> - PR/655 <https://github.com/dpgaspar/Flask-AppBuilder/pull/655> Form
> prefill support (unmerged)
>
>
> The second issue is an open question about the next step of Airflow
> Webserver itself. Here are the 3 potential directions we could take, and
> I've added my thought on each.
>
> 1. Permanently keep Airflow Webserver as a separated package from Airflow,
> and treat it as another UI option. Keep `www` in Airflow. Allow development
> on both UIs.
> *I'm not a fan of this. When there is an existing UI in Airflow, most
> contributors would prefer to maintain the official version that is
> installed out-of-the-box. **Having a second UI outside of Airflow will
> make maintaining it very difficult, leading to an eventual death of the new
> UI :(*
>
> 2. Permanently keep Airflow Webserver as a separated package from Airflow,
> but freeze all development on `www` and direct all future UI development
> to Airflow Webserver, eventually removing `www` completely when Airflow
> Webserver is stable.
> *I'm not a fan of this either. First of all, the views and models are
> tightly coupled in both old and new UI; until we have a full-fledged REST
> API to build the UI (and cli) on top of it, separating them to a separate
> package now will potentially cause dependency issues and add complication
> to our release cycle. **Secondly, **majority of Airflow users run Airflow
> with the UI; it's one of Airflow's best features. Separating UI out of
> Airflow core will complicate setup and configuration, while making Airflow
> core less complete.*
>
> 3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
> development on `www`, eventually removing `www` completely when `www2` is
> stable.
> *This makes the most sense to me. Airflow Webserver is developed with the
> goal of feature parity to the current UI, plus additional RBAC capability,
> in hope to replace the old UI completely. Yes, this means there will be a
> short period of having to maintain two UIs, but once we freeze development
> on www, it shouldn't be a concern for long.*
>
> I'd love to hear everyone's thoughts on this! I'm excited about bringing
> RBAC to airflow and I hope it's something others will find useful as well!
>
> Cheers,
> Joy
>
> On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
>
>> Thank you everyone for the active feedback so far, and thanks for setting
>> up the demo Maxime!
>>
>> Going to work on pruning through the issues in the upcoming days.
>>
>> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I can
>> look into it? Otherwise I'll wait until it's down again =P
>>
>> Cheers,
>>
>> Joy
>>
>> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
>> maximebeauchemin@gmail.com> wrote:
>>
>>> I just restarted it, not sure how long it will take to get in a bad state
>>> again...
>>>
>>> Max
>>>
>>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko <fokko@driesprong.frl
>>> >
>>> wrote:
>>>
>>> > Good morning,
>>> >
>>> > The demo provided by Max is down, it throws a SQLAlchemyexception :'(
>>> >
>>> > Cheers, Fokko
>>> >
>>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <cr...@apache.org>:
>>> >
>>> > > @bolke, open issues on the Github repo, please.
>>> > >
>>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <bd...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Chris,
>>> > > >
>>> > > > Do you want us to report bugs somewhere (I have encountered a
>>> few)? Or
>>> > > > just generic user experiences posted here?
>>> > > >
>>> > > > Cheers
>>> > > > Bolke
>>> > > >
>>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <criccomini@apache.org
>>> >
>>> > > wrote:
>>> > > > >
>>> > > > > Hey all,
>>> > > > >
>>> > > > > I know the weekend is coming up, and for those of us in the US,
>>> next
>>> > > week
>>> > > > > is a bit of a slow holiday week. Would love to get some feedback
>>> from
>>> > > > > everyone on this. The goal would ideally to be to converge on
>>> this
>>> > and
>>> > > > > eventually replace the existing Airflow UI with this one.
>>> > > > >
>>> > > > > Cheers,
>>> > > > > Chris
>>> > > > >
>>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com> wrote:
>>> > > > >
>>> > > > >> Hi guys.
>>> > > > >>
>>> > > > >> I've been working on moving airflow from Flask-Admin to
>>> > > Flask-AppBuilder
>>> > > > >> for RBAC
>>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
>>> > > > Airflow+RBAC+proposal
>>> > > > >>> ,
>>> > > > >> check it out at https://github.com/wepay/airflow-webserver.
>>> > > > >>
>>> > > > >> It's still a work-in-progress, but most features you see in the
>>> > > > webserver
>>> > > > >> UI today is available there. For those who are interested in
>>> RBAC,
>>> > I'd
>>> > > > love
>>> > > > >> to get some early feedback in terms of the following:
>>> > > > >>
>>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
>>> > > > >> - Setup issues
>>> > > > >> - Ease of integration with third party auth (i.e. LDAP, AD,
>>> OAuth,
>>> > > etc.)
>>> > > > >> - Any other thoughts/concerns
>>> > > > >>
>>> > > > >> Thanks a lot!
>>> > > > >>
>>> > > > >> Cheers,
>>> > > > >> Joy
>>> > > > >>
>>> > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: RBAC Update
Posted by Joy Gao <jo...@wepay.com>.
Hi folks,
Thanks for all the feedback regarding to the new Airflow Webserver UI
<https://github.com/wepay/airflow-webserver/>! I've been actively
addressing all the bugs that were raised on Github. So I want to take this
opportunity to discuss two issues coming up:
The first issue is unaddressed PRs in FAB. If these PRs continue to stay
unaddressed, RBAC is blocked from making further progress. If this continue
to be an issue, I'm inclined to fork FAB, even though it's not idealistic.
- PR/631 <https://github.com/dpgaspar/Flask-AppBuilder/pull/631> Binary
column support (merged, unreleased)
<https://github.com/dpgaspar/Flask-AppBuilder/pull/631>
- PR/639 <https://github.com/dpgaspar/Flask-AppBuilder/pull/639> Composite
primary key support (unmerged)
- PR/655 <https://github.com/dpgaspar/Flask-AppBuilder/pull/655> Form
prefill support (unmerged)
The second issue is an open question about the next step of Airflow
Webserver itself. Here are the 3 potential directions we could take, and
I've added my thought on each.
1. Permanently keep Airflow Webserver as a separated package from Airflow,
and treat it as another UI option. Keep `www` in Airflow. Allow development
on both UIs.
*I'm not a fan of this. When there is an existing UI in Airflow, most
contributors would prefer to maintain the official version that is
installed out-of-the-box. **Having a second UI outside of Airflow will make
maintaining it very difficult, leading to an eventual death of the new UI
:(*
2. Permanently keep Airflow Webserver as a separated package from Airflow,
but freeze all development on `www` and direct all future UI development
to Airflow Webserver, eventually removing `www` completely when Airflow
Webserver is stable.
*I'm not a fan of this either. First of all, the views and models are
tightly coupled in both old and new UI; until we have a full-fledged REST
API to build the UI (and cli) on top of it, separating them to a separate
package now will potentially cause dependency issues and add complication
to our release cycle. **Secondly, **majority of Airflow users run Airflow
with the UI; it's one of Airflow's best features. Separating UI out of
Airflow core will complicate setup and configuration, while making Airflow
core less complete.*
3. Merge Airflow Webserver back into Airflow as `www2`, freeze all
development on `www`, eventually removing `www` completely when `www2` is
stable.
*This makes the most sense to me. Airflow Webserver is developed with the
goal of feature parity to the current UI, plus additional RBAC capability,
in hope to replace the old UI completely. Yes, this means there will be a
short period of having to maintain two UIs, but once we freeze development
on www, it shouldn't be a concern for long.*
I'd love to hear everyone's thoughts on this! I'm excited about bringing
RBAC to airflow and I hope it's something others will find useful as well!
Cheers,
Joy
On Mon, Nov 20, 2017 at 11:24 AM, Joy Gao <jo...@wepay.com> wrote:
> Thank you everyone for the active feedback so far, and thanks for setting
> up the demo Maxime!
>
> Going to work on pruning through the issues in the upcoming days.
>
> Fokko/Maxime, do you recall the SQLAlchemy Exception message so I can look
> into it? Otherwise I'll wait until it's down again =P
>
> Cheers,
>
> Joy
>
> On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
>> I just restarted it, not sure how long it will take to get in a bad state
>> again...
>>
>> Max
>>
>> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko <fokko@driesprong.frl
>> >
>> wrote:
>>
>> > Good morning,
>> >
>> > The demo provided by Max is down, it throws a SQLAlchemyexception :'(
>> >
>> > Cheers, Fokko
>> >
>> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <cr...@apache.org>:
>> >
>> > > @bolke, open issues on the Github repo, please.
>> > >
>> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <bd...@gmail.com>
>> > > wrote:
>> > >
>> > > > Chris,
>> > > >
>> > > > Do you want us to report bugs somewhere (I have encountered a few)?
>> Or
>> > > > just generic user experiences posted here?
>> > > >
>> > > > Cheers
>> > > > Bolke
>> > > >
>> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <cr...@apache.org>
>> > > wrote:
>> > > > >
>> > > > > Hey all,
>> > > > >
>> > > > > I know the weekend is coming up, and for those of us in the US,
>> next
>> > > week
>> > > > > is a bit of a slow holiday week. Would love to get some feedback
>> from
>> > > > > everyone on this. The goal would ideally to be to converge on this
>> > and
>> > > > > eventually replace the existing Airflow UI with this one.
>> > > > >
>> > > > > Cheers,
>> > > > > Chris
>> > > > >
>> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com> wrote:
>> > > > >
>> > > > >> Hi guys.
>> > > > >>
>> > > > >> I've been working on moving airflow from Flask-Admin to
>> > > Flask-AppBuilder
>> > > > >> for RBAC
>> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
>> > > > Airflow+RBAC+proposal
>> > > > >>> ,
>> > > > >> check it out at https://github.com/wepay/airflow-webserver.
>> > > > >>
>> > > > >> It's still a work-in-progress, but most features you see in the
>> > > > webserver
>> > > > >> UI today is available there. For those who are interested in
>> RBAC,
>> > I'd
>> > > > love
>> > > > >> to get some early feedback in terms of the following:
>> > > > >>
>> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
>> > > > >> - Setup issues
>> > > > >> - Ease of integration with third party auth (i.e. LDAP, AD,
>> OAuth,
>> > > etc.)
>> > > > >> - Any other thoughts/concerns
>> > > > >>
>> > > > >> Thanks a lot!
>> > > > >>
>> > > > >> Cheers,
>> > > > >> Joy
>> > > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
>
Re: RBAC Update
Posted by Joy Gao <jo...@wepay.com>.
Thank you everyone for the active feedback so far, and thanks for setting
up the demo Maxime!
Going to work on pruning through the issues in the upcoming days.
Fokko/Maxime, do you recall the SQLAlchemy Exception message so I can look
into it? Otherwise I'll wait until it's down again =P
Cheers,
Joy
On Mon, Nov 20, 2017 at 9:35 AM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> I just restarted it, not sure how long it will take to get in a bad state
> again...
>
> Max
>
> On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
> > Good morning,
> >
> > The demo provided by Max is down, it throws a SQLAlchemyexception :'(
> >
> > Cheers, Fokko
> >
> > 2017-11-18 19:14 GMT+01:00 Chris Riccomini <cr...@apache.org>:
> >
> > > @bolke, open issues on the Github repo, please.
> > >
> > > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <bd...@gmail.com>
> > > wrote:
> > >
> > > > Chris,
> > > >
> > > > Do you want us to report bugs somewhere (I have encountered a few)?
> Or
> > > > just generic user experiences posted here?
> > > >
> > > > Cheers
> > > > Bolke
> > > >
> > > > > On 18 Nov 2017, at 00:47, Chris Riccomini <cr...@apache.org>
> > > wrote:
> > > > >
> > > > > Hey all,
> > > > >
> > > > > I know the weekend is coming up, and for those of us in the US,
> next
> > > week
> > > > > is a bit of a slow holiday week. Would love to get some feedback
> from
> > > > > everyone on this. The goal would ideally to be to converge on this
> > and
> > > > > eventually replace the existing Airflow UI with this one.
> > > > >
> > > > > Cheers,
> > > > > Chris
> > > > >
> > > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com> wrote:
> > > > >
> > > > >> Hi guys.
> > > > >>
> > > > >> I've been working on moving airflow from Flask-Admin to
> > > Flask-AppBuilder
> > > > >> for RBAC
> > > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> > > > Airflow+RBAC+proposal
> > > > >>> ,
> > > > >> check it out at https://github.com/wepay/airflow-webserver.
> > > > >>
> > > > >> It's still a work-in-progress, but most features you see in the
> > > > webserver
> > > > >> UI today is available there. For those who are interested in RBAC,
> > I'd
> > > > love
> > > > >> to get some early feedback in terms of the following:
> > > > >>
> > > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> > > > >> - Setup issues
> > > > >> - Ease of integration with third party auth (i.e. LDAP, AD, OAuth,
> > > etc.)
> > > > >> - Any other thoughts/concerns
> > > > >>
> > > > >> Thanks a lot!
> > > > >>
> > > > >> Cheers,
> > > > >> Joy
> > > > >>
> > > >
> > > >
> > >
> >
>
Re: RBAC Update
Posted by Maxime Beauchemin <ma...@gmail.com>.
I just restarted it, not sure how long it will take to get in a bad state
again...
Max
On Sun, Nov 19, 2017 at 11:55 PM, Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Good morning,
>
> The demo provided by Max is down, it throws a SQLAlchemyexception :'(
>
> Cheers, Fokko
>
> 2017-11-18 19:14 GMT+01:00 Chris Riccomini <cr...@apache.org>:
>
> > @bolke, open issues on the Github repo, please.
> >
> > On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> >
> > > Chris,
> > >
> > > Do you want us to report bugs somewhere (I have encountered a few)? Or
> > > just generic user experiences posted here?
> > >
> > > Cheers
> > > Bolke
> > >
> > > > On 18 Nov 2017, at 00:47, Chris Riccomini <cr...@apache.org>
> > wrote:
> > > >
> > > > Hey all,
> > > >
> > > > I know the weekend is coming up, and for those of us in the US, next
> > week
> > > > is a bit of a slow holiday week. Would love to get some feedback from
> > > > everyone on this. The goal would ideally to be to converge on this
> and
> > > > eventually replace the existing Airflow UI with this one.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com> wrote:
> > > >
> > > >> Hi guys.
> > > >>
> > > >> I've been working on moving airflow from Flask-Admin to
> > Flask-AppBuilder
> > > >> for RBAC
> > > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> > > Airflow+RBAC+proposal
> > > >>> ,
> > > >> check it out at https://github.com/wepay/airflow-webserver.
> > > >>
> > > >> It's still a work-in-progress, but most features you see in the
> > > webserver
> > > >> UI today is available there. For those who are interested in RBAC,
> I'd
> > > love
> > > >> to get some early feedback in terms of the following:
> > > >>
> > > >> - New Flask-AppBuilder UI (any bugs/regressions)
> > > >> - Setup issues
> > > >> - Ease of integration with third party auth (i.e. LDAP, AD, OAuth,
> > etc.)
> > > >> - Any other thoughts/concerns
> > > >>
> > > >> Thanks a lot!
> > > >>
> > > >> Cheers,
> > > >> Joy
> > > >>
> > >
> > >
> >
>
Re: RBAC Update
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Good morning,
The demo provided by Max is down, it throws a SQLAlchemyexception :'(
Cheers, Fokko
2017-11-18 19:14 GMT+01:00 Chris Riccomini <cr...@apache.org>:
> @bolke, open issues on the Github repo, please.
>
> On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
>
> > Chris,
> >
> > Do you want us to report bugs somewhere (I have encountered a few)? Or
> > just generic user experiences posted here?
> >
> > Cheers
> > Bolke
> >
> > > On 18 Nov 2017, at 00:47, Chris Riccomini <cr...@apache.org>
> wrote:
> > >
> > > Hey all,
> > >
> > > I know the weekend is coming up, and for those of us in the US, next
> week
> > > is a bit of a slow holiday week. Would love to get some feedback from
> > > everyone on this. The goal would ideally to be to converge on this and
> > > eventually replace the existing Airflow UI with this one.
> > >
> > > Cheers,
> > > Chris
> > >
> > > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com> wrote:
> > >
> > >> Hi guys.
> > >>
> > >> I've been working on moving airflow from Flask-Admin to
> Flask-AppBuilder
> > >> for RBAC
> > >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> > Airflow+RBAC+proposal
> > >>> ,
> > >> check it out at https://github.com/wepay/airflow-webserver.
> > >>
> > >> It's still a work-in-progress, but most features you see in the
> > webserver
> > >> UI today is available there. For those who are interested in RBAC, I'd
> > love
> > >> to get some early feedback in terms of the following:
> > >>
> > >> - New Flask-AppBuilder UI (any bugs/regressions)
> > >> - Setup issues
> > >> - Ease of integration with third party auth (i.e. LDAP, AD, OAuth,
> etc.)
> > >> - Any other thoughts/concerns
> > >>
> > >> Thanks a lot!
> > >>
> > >> Cheers,
> > >> Joy
> > >>
> >
> >
>
Re: RBAC Update
Posted by Chris Riccomini <cr...@apache.org>.
@bolke, open issues on the Github repo, please.
On Sat, Nov 18, 2017 at 10:13 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> Chris,
>
> Do you want us to report bugs somewhere (I have encountered a few)? Or
> just generic user experiences posted here?
>
> Cheers
> Bolke
>
> > On 18 Nov 2017, at 00:47, Chris Riccomini <cr...@apache.org> wrote:
> >
> > Hey all,
> >
> > I know the weekend is coming up, and for those of us in the US, next week
> > is a bit of a slow holiday week. Would love to get some feedback from
> > everyone on this. The goal would ideally to be to converge on this and
> > eventually replace the existing Airflow UI with this one.
> >
> > Cheers,
> > Chris
> >
> > On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> >> Hi guys.
> >>
> >> I've been working on moving airflow from Flask-Admin to Flask-AppBuilder
> >> for RBAC
> >> <https://cwiki.apache.org/confluence/display/AIRFLOW/
> Airflow+RBAC+proposal
> >>> ,
> >> check it out at https://github.com/wepay/airflow-webserver.
> >>
> >> It's still a work-in-progress, but most features you see in the
> webserver
> >> UI today is available there. For those who are interested in RBAC, I'd
> love
> >> to get some early feedback in terms of the following:
> >>
> >> - New Flask-AppBuilder UI (any bugs/regressions)
> >> - Setup issues
> >> - Ease of integration with third party auth (i.e. LDAP, AD, OAuth, etc.)
> >> - Any other thoughts/concerns
> >>
> >> Thanks a lot!
> >>
> >> Cheers,
> >> Joy
> >>
>
>
Re: RBAC Update
Posted by Bolke de Bruin <bd...@gmail.com>.
Chris,
Do you want us to report bugs somewhere (I have encountered a few)? Or just generic user experiences posted here?
Cheers
Bolke
> On 18 Nov 2017, at 00:47, Chris Riccomini <cr...@apache.org> wrote:
>
> Hey all,
>
> I know the weekend is coming up, and for those of us in the US, next week
> is a bit of a slow holiday week. Would love to get some feedback from
> everyone on this. The goal would ideally to be to converge on this and
> eventually replace the existing Airflow UI with this one.
>
> Cheers,
> Chris
>
> On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com> wrote:
>
>> Hi guys.
>>
>> I've been working on moving airflow from Flask-Admin to Flask-AppBuilder
>> for RBAC
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+RBAC+proposal
>>> ,
>> check it out at https://github.com/wepay/airflow-webserver.
>>
>> It's still a work-in-progress, but most features you see in the webserver
>> UI today is available there. For those who are interested in RBAC, I'd love
>> to get some early feedback in terms of the following:
>>
>> - New Flask-AppBuilder UI (any bugs/regressions)
>> - Setup issues
>> - Ease of integration with third party auth (i.e. LDAP, AD, OAuth, etc.)
>> - Any other thoughts/concerns
>>
>> Thanks a lot!
>>
>> Cheers,
>> Joy
>>
Re: RBAC Update
Posted by Chris Riccomini <cr...@apache.org>.
Hey all,
I know the weekend is coming up, and for those of us in the US, next week
is a bit of a slow holiday week. Would love to get some feedback from
everyone on this. The goal would ideally to be to converge on this and
eventually replace the existing Airflow UI with this one.
Cheers,
Chris
On Fri, Nov 17, 2017 at 1:44 PM, Joy Gao <jo...@wepay.com> wrote:
> Hi guys.
>
> I've been working on moving airflow from Flask-Admin to Flask-AppBuilder
> for RBAC
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+RBAC+proposal
> >,
> check it out at https://github.com/wepay/airflow-webserver.
>
> It's still a work-in-progress, but most features you see in the webserver
> UI today is available there. For those who are interested in RBAC, I'd love
> to get some early feedback in terms of the following:
>
> - New Flask-AppBuilder UI (any bugs/regressions)
> - Setup issues
> - Ease of integration with third party auth (i.e. LDAP, AD, OAuth, etc.)
> - Any other thoughts/concerns
>
> Thanks a lot!
>
> Cheers,
> Joy
>