You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Jonathan Hung <jy...@gmail.com> on 2017/09/28 18:32:29 UTC
[DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Hi everyone,
Starting this thread to discuss merging API-based scheduler configuration
to trunk/branch-2. The feature adds the framework for allowing users to
modify scheduler configuration via REST or CLI using a configurable backend
(leveldb/zk are currently supported), and adds capacity scheduler support
for this. The umbrella JIRA is YARN-5734. All the required work for this
feature is done and committed to branch YARN-5734, and a full diff has been
generated at YARN-7241.
Regarding compatibility, this feature is configurable and turned off by
default.
The feature has been tested locally on a couple RMs (since it is an RM only
change), with queue addition/removal/updates tested on single RM (leveldb)
and two RMs (zk). Also we verified the original configuration update
mechanism (via refreshQueues) is unaffected when the feature is off/not
configured.
Our original plan was to merge this to trunk (which is what the YARN-7241
diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
what are your thoughts on also merging this to branch-3.0?
Thanks!
Jonathan Hung
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Thanks Andrew and Larry for the feedback. I was hoping to start a merge
vote early next week, because of the 2.9 deadline. (I suppose meeting this
deadline depends on the outcome of this DISCUSS thread.) Appreciate any
questions you have on the JIRA.
To answer your questions Larry:
*Is this feature extending the existing YARM RM REST API?*
Yes, this feature adds another endpoint to the YARN RM REST API, for users
to send their configuration change requests.
*When it isn't enabled what is the API behavior?*
When disabled and API is called, nothing happens, it will return HTTP 400
bad request.
*Does it implement the trusted proxy pattern for proxies to be able to
impersonate users and most importantly to dictate what proxies would be
allowed to impersonate an admin for this API - which I assume will be
required?*
Right now there's a pluggable policy which controls which users can make
which configuration changes (see YARN-5949). The default policy is to only
allow YARN admins (i.e. users in yarn.admin.acl) to make changes. There's
also an implementation of a more relaxed policy which allows admins of
queues to make configuration modifications to their own queue. Not sure if
this answers your question.
Thanks,
Jonathan Hung
On Fri, Sep 29, 2017 at 12:01 PM, larry mccay <la...@gmail.com> wrote:
> Hi Jonathan -
>
> Thank you for bringing this up for discussion!
>
> I would personally like to see a specific security review of features like
> this - especially ones that allow for remote access to configuration.
> I'll take a look at the JIRA and see whether I can come up with any
> concerns or questions and I would urge others to give it a pass from a
> security perspective as well.
>
> In addition, here are a couple questions of the top of my head:
>
> Is this feature extending the existing YARM RM REST API?
> When it isn't enabled what is the API behavior?
> Does it implement the trusted proxy pattern for proxies to be able to
> impersonate users and most importantly to dictate what proxies would be
> allowed to impersonate an admin for this API - which I assume will be
> required?
>
> --larry
>
> On Fri, Sep 29, 2017 at 2:44 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
>> Hi Jonathan,
>>
>> I'm okay with putting this into branch-3.0 for GA if it can be merged
>> within the next two weeks. Even though beta1 has slipped by a month, I
>> want
>> to stick to the targeted GA data of Nov 1st as much as possible. Of
>> course,
>> let's not sacrifice quality or stability for speed; if something's not
>> ready, let's defer it to 3.1.0.
>>
>> Subru, have you been able to review this feature from the 2.9.0
>> perspective? It'd add confidence if you think it's immediately ready for
>> merging to branch-2 for 2.9.0.
>>
>> Thanks,
>> Andrew
>>
>> On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > Starting this thread to discuss merging API-based scheduler
>> configuration
>> > to trunk/branch-2. The feature adds the framework for allowing users to
>> > modify scheduler configuration via REST or CLI using a configurable
>> backend
>> > (leveldb/zk are currently supported), and adds capacity scheduler
>> support
>> > for this. The umbrella JIRA is YARN-5734. All the required work for this
>> > feature is done and committed to branch YARN-5734, and a full diff has
>> been
>> > generated at YARN-7241.
>> >
>> > Regarding compatibility, this feature is configurable and turned off by
>> > default.
>> >
>> > The feature has been tested locally on a couple RMs (since it is an RM
>> > only change), with queue addition/removal/updates tested on single RM
>> > (leveldb) and two RMs (zk). Also we verified the original configuration
>> > update mechanism (via refreshQueues) is unaffected when the feature is
>> > off/not configured.
>> >
>> > Our original plan was to merge this to trunk (which is what the
>> YARN-7241
>> > diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
>> > what are your thoughts on also merging this to branch-3.0?
>> >
>> > Thanks!
>> >
>> > Jonathan Hung
>> >
>>
>
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Thanks Andrew and Larry for the feedback. I was hoping to start a merge
vote early next week, because of the 2.9 deadline. (I suppose meeting this
deadline depends on the outcome of this DISCUSS thread.) Appreciate any
questions you have on the JIRA.
To answer your questions Larry:
*Is this feature extending the existing YARM RM REST API?*
Yes, this feature adds another endpoint to the YARN RM REST API, for users
to send their configuration change requests.
*When it isn't enabled what is the API behavior?*
When disabled and API is called, nothing happens, it will return HTTP 400
bad request.
*Does it implement the trusted proxy pattern for proxies to be able to
impersonate users and most importantly to dictate what proxies would be
allowed to impersonate an admin for this API - which I assume will be
required?*
Right now there's a pluggable policy which controls which users can make
which configuration changes (see YARN-5949). The default policy is to only
allow YARN admins (i.e. users in yarn.admin.acl) to make changes. There's
also an implementation of a more relaxed policy which allows admins of
queues to make configuration modifications to their own queue. Not sure if
this answers your question.
Thanks,
Jonathan Hung
On Fri, Sep 29, 2017 at 12:01 PM, larry mccay <la...@gmail.com> wrote:
> Hi Jonathan -
>
> Thank you for bringing this up for discussion!
>
> I would personally like to see a specific security review of features like
> this - especially ones that allow for remote access to configuration.
> I'll take a look at the JIRA and see whether I can come up with any
> concerns or questions and I would urge others to give it a pass from a
> security perspective as well.
>
> In addition, here are a couple questions of the top of my head:
>
> Is this feature extending the existing YARM RM REST API?
> When it isn't enabled what is the API behavior?
> Does it implement the trusted proxy pattern for proxies to be able to
> impersonate users and most importantly to dictate what proxies would be
> allowed to impersonate an admin for this API - which I assume will be
> required?
>
> --larry
>
> On Fri, Sep 29, 2017 at 2:44 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
>> Hi Jonathan,
>>
>> I'm okay with putting this into branch-3.0 for GA if it can be merged
>> within the next two weeks. Even though beta1 has slipped by a month, I
>> want
>> to stick to the targeted GA data of Nov 1st as much as possible. Of
>> course,
>> let's not sacrifice quality or stability for speed; if something's not
>> ready, let's defer it to 3.1.0.
>>
>> Subru, have you been able to review this feature from the 2.9.0
>> perspective? It'd add confidence if you think it's immediately ready for
>> merging to branch-2 for 2.9.0.
>>
>> Thanks,
>> Andrew
>>
>> On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > Starting this thread to discuss merging API-based scheduler
>> configuration
>> > to trunk/branch-2. The feature adds the framework for allowing users to
>> > modify scheduler configuration via REST or CLI using a configurable
>> backend
>> > (leveldb/zk are currently supported), and adds capacity scheduler
>> support
>> > for this. The umbrella JIRA is YARN-5734. All the required work for this
>> > feature is done and committed to branch YARN-5734, and a full diff has
>> been
>> > generated at YARN-7241.
>> >
>> > Regarding compatibility, this feature is configurable and turned off by
>> > default.
>> >
>> > The feature has been tested locally on a couple RMs (since it is an RM
>> > only change), with queue addition/removal/updates tested on single RM
>> > (leveldb) and two RMs (zk). Also we verified the original configuration
>> > update mechanism (via refreshQueues) is unaffected when the feature is
>> > off/not configured.
>> >
>> > Our original plan was to merge this to trunk (which is what the
>> YARN-7241
>> > diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
>> > what are your thoughts on also merging this to branch-3.0?
>> >
>> > Thanks!
>> >
>> > Jonathan Hung
>> >
>>
>
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Thanks Andrew and Larry for the feedback. I was hoping to start a merge
vote early next week, because of the 2.9 deadline. (I suppose meeting this
deadline depends on the outcome of this DISCUSS thread.) Appreciate any
questions you have on the JIRA.
To answer your questions Larry:
*Is this feature extending the existing YARM RM REST API?*
Yes, this feature adds another endpoint to the YARN RM REST API, for users
to send their configuration change requests.
*When it isn't enabled what is the API behavior?*
When disabled and API is called, nothing happens, it will return HTTP 400
bad request.
*Does it implement the trusted proxy pattern for proxies to be able to
impersonate users and most importantly to dictate what proxies would be
allowed to impersonate an admin for this API - which I assume will be
required?*
Right now there's a pluggable policy which controls which users can make
which configuration changes (see YARN-5949). The default policy is to only
allow YARN admins (i.e. users in yarn.admin.acl) to make changes. There's
also an implementation of a more relaxed policy which allows admins of
queues to make configuration modifications to their own queue. Not sure if
this answers your question.
Thanks,
Jonathan Hung
On Fri, Sep 29, 2017 at 12:01 PM, larry mccay <la...@gmail.com> wrote:
> Hi Jonathan -
>
> Thank you for bringing this up for discussion!
>
> I would personally like to see a specific security review of features like
> this - especially ones that allow for remote access to configuration.
> I'll take a look at the JIRA and see whether I can come up with any
> concerns or questions and I would urge others to give it a pass from a
> security perspective as well.
>
> In addition, here are a couple questions of the top of my head:
>
> Is this feature extending the existing YARM RM REST API?
> When it isn't enabled what is the API behavior?
> Does it implement the trusted proxy pattern for proxies to be able to
> impersonate users and most importantly to dictate what proxies would be
> allowed to impersonate an admin for this API - which I assume will be
> required?
>
> --larry
>
> On Fri, Sep 29, 2017 at 2:44 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
>> Hi Jonathan,
>>
>> I'm okay with putting this into branch-3.0 for GA if it can be merged
>> within the next two weeks. Even though beta1 has slipped by a month, I
>> want
>> to stick to the targeted GA data of Nov 1st as much as possible. Of
>> course,
>> let's not sacrifice quality or stability for speed; if something's not
>> ready, let's defer it to 3.1.0.
>>
>> Subru, have you been able to review this feature from the 2.9.0
>> perspective? It'd add confidence if you think it's immediately ready for
>> merging to branch-2 for 2.9.0.
>>
>> Thanks,
>> Andrew
>>
>> On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > Starting this thread to discuss merging API-based scheduler
>> configuration
>> > to trunk/branch-2. The feature adds the framework for allowing users to
>> > modify scheduler configuration via REST or CLI using a configurable
>> backend
>> > (leveldb/zk are currently supported), and adds capacity scheduler
>> support
>> > for this. The umbrella JIRA is YARN-5734. All the required work for this
>> > feature is done and committed to branch YARN-5734, and a full diff has
>> been
>> > generated at YARN-7241.
>> >
>> > Regarding compatibility, this feature is configurable and turned off by
>> > default.
>> >
>> > The feature has been tested locally on a couple RMs (since it is an RM
>> > only change), with queue addition/removal/updates tested on single RM
>> > (leveldb) and two RMs (zk). Also we verified the original configuration
>> > update mechanism (via refreshQueues) is unaffected when the feature is
>> > off/not configured.
>> >
>> > Our original plan was to merge this to trunk (which is what the
>> YARN-7241
>> > diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
>> > what are your thoughts on also merging this to branch-3.0?
>> >
>> > Thanks!
>> >
>> > Jonathan Hung
>> >
>>
>
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Thanks Andrew and Larry for the feedback. I was hoping to start a merge
vote early next week, because of the 2.9 deadline. (I suppose meeting this
deadline depends on the outcome of this DISCUSS thread.) Appreciate any
questions you have on the JIRA.
To answer your questions Larry:
*Is this feature extending the existing YARM RM REST API?*
Yes, this feature adds another endpoint to the YARN RM REST API, for users
to send their configuration change requests.
*When it isn't enabled what is the API behavior?*
When disabled and API is called, nothing happens, it will return HTTP 400
bad request.
*Does it implement the trusted proxy pattern for proxies to be able to
impersonate users and most importantly to dictate what proxies would be
allowed to impersonate an admin for this API - which I assume will be
required?*
Right now there's a pluggable policy which controls which users can make
which configuration changes (see YARN-5949). The default policy is to only
allow YARN admins (i.e. users in yarn.admin.acl) to make changes. There's
also an implementation of a more relaxed policy which allows admins of
queues to make configuration modifications to their own queue. Not sure if
this answers your question.
Thanks,
Jonathan Hung
On Fri, Sep 29, 2017 at 12:01 PM, larry mccay <la...@gmail.com> wrote:
> Hi Jonathan -
>
> Thank you for bringing this up for discussion!
>
> I would personally like to see a specific security review of features like
> this - especially ones that allow for remote access to configuration.
> I'll take a look at the JIRA and see whether I can come up with any
> concerns or questions and I would urge others to give it a pass from a
> security perspective as well.
>
> In addition, here are a couple questions of the top of my head:
>
> Is this feature extending the existing YARM RM REST API?
> When it isn't enabled what is the API behavior?
> Does it implement the trusted proxy pattern for proxies to be able to
> impersonate users and most importantly to dictate what proxies would be
> allowed to impersonate an admin for this API - which I assume will be
> required?
>
> --larry
>
> On Fri, Sep 29, 2017 at 2:44 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
>> Hi Jonathan,
>>
>> I'm okay with putting this into branch-3.0 for GA if it can be merged
>> within the next two weeks. Even though beta1 has slipped by a month, I
>> want
>> to stick to the targeted GA data of Nov 1st as much as possible. Of
>> course,
>> let's not sacrifice quality or stability for speed; if something's not
>> ready, let's defer it to 3.1.0.
>>
>> Subru, have you been able to review this feature from the 2.9.0
>> perspective? It'd add confidence if you think it's immediately ready for
>> merging to branch-2 for 2.9.0.
>>
>> Thanks,
>> Andrew
>>
>> On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
>> wrote:
>>
>> > Hi everyone,
>> >
>> > Starting this thread to discuss merging API-based scheduler
>> configuration
>> > to trunk/branch-2. The feature adds the framework for allowing users to
>> > modify scheduler configuration via REST or CLI using a configurable
>> backend
>> > (leveldb/zk are currently supported), and adds capacity scheduler
>> support
>> > for this. The umbrella JIRA is YARN-5734. All the required work for this
>> > feature is done and committed to branch YARN-5734, and a full diff has
>> been
>> > generated at YARN-7241.
>> >
>> > Regarding compatibility, this feature is configurable and turned off by
>> > default.
>> >
>> > The feature has been tested locally on a couple RMs (since it is an RM
>> > only change), with queue addition/removal/updates tested on single RM
>> > (leveldb) and two RMs (zk). Also we verified the original configuration
>> > update mechanism (via refreshQueues) is unaffected when the feature is
>> > off/not configured.
>> >
>> > Our original plan was to merge this to trunk (which is what the
>> YARN-7241
>> > diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
>> > what are your thoughts on also merging this to branch-3.0?
>> >
>> > Thanks!
>> >
>> > Jonathan Hung
>> >
>>
>
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by larry mccay <la...@gmail.com>.
Hi Jonathan -
Thank you for bringing this up for discussion!
I would personally like to see a specific security review of features like
this - especially ones that allow for remote access to configuration.
I'll take a look at the JIRA and see whether I can come up with any
concerns or questions and I would urge others to give it a pass from a
security perspective as well.
In addition, here are a couple questions of the top of my head:
Is this feature extending the existing YARM RM REST API?
When it isn't enabled what is the API behavior?
Does it implement the trusted proxy pattern for proxies to be able to
impersonate users and most importantly to dictate what proxies would be
allowed to impersonate an admin for this API - which I assume will be
required?
--larry
On Fri, Sep 29, 2017 at 2:44 PM, Andrew Wang <an...@cloudera.com>
wrote:
> Hi Jonathan,
>
> I'm okay with putting this into branch-3.0 for GA if it can be merged
> within the next two weeks. Even though beta1 has slipped by a month, I want
> to stick to the targeted GA data of Nov 1st as much as possible. Of course,
> let's not sacrifice quality or stability for speed; if something's not
> ready, let's defer it to 3.1.0.
>
> Subru, have you been able to review this feature from the 2.9.0
> perspective? It'd add confidence if you think it's immediately ready for
> merging to branch-2 for 2.9.0.
>
> Thanks,
> Andrew
>
> On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > Starting this thread to discuss merging API-based scheduler configuration
> > to trunk/branch-2. The feature adds the framework for allowing users to
> > modify scheduler configuration via REST or CLI using a configurable
> backend
> > (leveldb/zk are currently supported), and adds capacity scheduler support
> > for this. The umbrella JIRA is YARN-5734. All the required work for this
> > feature is done and committed to branch YARN-5734, and a full diff has
> been
> > generated at YARN-7241.
> >
> > Regarding compatibility, this feature is configurable and turned off by
> > default.
> >
> > The feature has been tested locally on a couple RMs (since it is an RM
> > only change), with queue addition/removal/updates tested on single RM
> > (leveldb) and two RMs (zk). Also we verified the original configuration
> > update mechanism (via refreshQueues) is unaffected when the feature is
> > off/not configured.
> >
> > Our original plan was to merge this to trunk (which is what the YARN-7241
> > diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
> > what are your thoughts on also merging this to branch-3.0?
> >
> > Thanks!
> >
> > Jonathan Hung
> >
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by larry mccay <la...@gmail.com>.
Hi Jonathan -
Thank you for bringing this up for discussion!
I would personally like to see a specific security review of features like
this - especially ones that allow for remote access to configuration.
I'll take a look at the JIRA and see whether I can come up with any
concerns or questions and I would urge others to give it a pass from a
security perspective as well.
In addition, here are a couple questions of the top of my head:
Is this feature extending the existing YARM RM REST API?
When it isn't enabled what is the API behavior?
Does it implement the trusted proxy pattern for proxies to be able to
impersonate users and most importantly to dictate what proxies would be
allowed to impersonate an admin for this API - which I assume will be
required?
--larry
On Fri, Sep 29, 2017 at 2:44 PM, Andrew Wang <an...@cloudera.com>
wrote:
> Hi Jonathan,
>
> I'm okay with putting this into branch-3.0 for GA if it can be merged
> within the next two weeks. Even though beta1 has slipped by a month, I want
> to stick to the targeted GA data of Nov 1st as much as possible. Of course,
> let's not sacrifice quality or stability for speed; if something's not
> ready, let's defer it to 3.1.0.
>
> Subru, have you been able to review this feature from the 2.9.0
> perspective? It'd add confidence if you think it's immediately ready for
> merging to branch-2 for 2.9.0.
>
> Thanks,
> Andrew
>
> On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > Starting this thread to discuss merging API-based scheduler configuration
> > to trunk/branch-2. The feature adds the framework for allowing users to
> > modify scheduler configuration via REST or CLI using a configurable
> backend
> > (leveldb/zk are currently supported), and adds capacity scheduler support
> > for this. The umbrella JIRA is YARN-5734. All the required work for this
> > feature is done and committed to branch YARN-5734, and a full diff has
> been
> > generated at YARN-7241.
> >
> > Regarding compatibility, this feature is configurable and turned off by
> > default.
> >
> > The feature has been tested locally on a couple RMs (since it is an RM
> > only change), with queue addition/removal/updates tested on single RM
> > (leveldb) and two RMs (zk). Also we verified the original configuration
> > update mechanism (via refreshQueues) is unaffected when the feature is
> > off/not configured.
> >
> > Our original plan was to merge this to trunk (which is what the YARN-7241
> > diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
> > what are your thoughts on also merging this to branch-3.0?
> >
> > Thanks!
> >
> > Jonathan Hung
> >
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by larry mccay <la...@gmail.com>.
Hi Jonathan -
Thank you for bringing this up for discussion!
I would personally like to see a specific security review of features like
this - especially ones that allow for remote access to configuration.
I'll take a look at the JIRA and see whether I can come up with any
concerns or questions and I would urge others to give it a pass from a
security perspective as well.
In addition, here are a couple questions of the top of my head:
Is this feature extending the existing YARM RM REST API?
When it isn't enabled what is the API behavior?
Does it implement the trusted proxy pattern for proxies to be able to
impersonate users and most importantly to dictate what proxies would be
allowed to impersonate an admin for this API - which I assume will be
required?
--larry
On Fri, Sep 29, 2017 at 2:44 PM, Andrew Wang <an...@cloudera.com>
wrote:
> Hi Jonathan,
>
> I'm okay with putting this into branch-3.0 for GA if it can be merged
> within the next two weeks. Even though beta1 has slipped by a month, I want
> to stick to the targeted GA data of Nov 1st as much as possible. Of course,
> let's not sacrifice quality or stability for speed; if something's not
> ready, let's defer it to 3.1.0.
>
> Subru, have you been able to review this feature from the 2.9.0
> perspective? It'd add confidence if you think it's immediately ready for
> merging to branch-2 for 2.9.0.
>
> Thanks,
> Andrew
>
> On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > Starting this thread to discuss merging API-based scheduler configuration
> > to trunk/branch-2. The feature adds the framework for allowing users to
> > modify scheduler configuration via REST or CLI using a configurable
> backend
> > (leveldb/zk are currently supported), and adds capacity scheduler support
> > for this. The umbrella JIRA is YARN-5734. All the required work for this
> > feature is done and committed to branch YARN-5734, and a full diff has
> been
> > generated at YARN-7241.
> >
> > Regarding compatibility, this feature is configurable and turned off by
> > default.
> >
> > The feature has been tested locally on a couple RMs (since it is an RM
> > only change), with queue addition/removal/updates tested on single RM
> > (leveldb) and two RMs (zk). Also we verified the original configuration
> > update mechanism (via refreshQueues) is unaffected when the feature is
> > off/not configured.
> >
> > Our original plan was to merge this to trunk (which is what the YARN-7241
> > diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
> > what are your thoughts on also merging this to branch-3.0?
> >
> > Thanks!
> >
> > Jonathan Hung
> >
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by larry mccay <la...@gmail.com>.
Hi Jonathan -
Thank you for bringing this up for discussion!
I would personally like to see a specific security review of features like
this - especially ones that allow for remote access to configuration.
I'll take a look at the JIRA and see whether I can come up with any
concerns or questions and I would urge others to give it a pass from a
security perspective as well.
In addition, here are a couple questions of the top of my head:
Is this feature extending the existing YARM RM REST API?
When it isn't enabled what is the API behavior?
Does it implement the trusted proxy pattern for proxies to be able to
impersonate users and most importantly to dictate what proxies would be
allowed to impersonate an admin for this API - which I assume will be
required?
--larry
On Fri, Sep 29, 2017 at 2:44 PM, Andrew Wang <an...@cloudera.com>
wrote:
> Hi Jonathan,
>
> I'm okay with putting this into branch-3.0 for GA if it can be merged
> within the next two weeks. Even though beta1 has slipped by a month, I want
> to stick to the targeted GA data of Nov 1st as much as possible. Of course,
> let's not sacrifice quality or stability for speed; if something's not
> ready, let's defer it to 3.1.0.
>
> Subru, have you been able to review this feature from the 2.9.0
> perspective? It'd add confidence if you think it's immediately ready for
> merging to branch-2 for 2.9.0.
>
> Thanks,
> Andrew
>
> On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
> wrote:
>
> > Hi everyone,
> >
> > Starting this thread to discuss merging API-based scheduler configuration
> > to trunk/branch-2. The feature adds the framework for allowing users to
> > modify scheduler configuration via REST or CLI using a configurable
> backend
> > (leveldb/zk are currently supported), and adds capacity scheduler support
> > for this. The umbrella JIRA is YARN-5734. All the required work for this
> > feature is done and committed to branch YARN-5734, and a full diff has
> been
> > generated at YARN-7241.
> >
> > Regarding compatibility, this feature is configurable and turned off by
> > default.
> >
> > The feature has been tested locally on a couple RMs (since it is an RM
> > only change), with queue addition/removal/updates tested on single RM
> > (leveldb) and two RMs (zk). Also we verified the original configuration
> > update mechanism (via refreshQueues) is unaffected when the feature is
> > off/not configured.
> >
> > Our original plan was to merge this to trunk (which is what the YARN-7241
> > diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
> > what are your thoughts on also merging this to branch-3.0?
> >
> > Thanks!
> >
> > Jonathan Hung
> >
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by Andrew Wang <an...@cloudera.com>.
Hi Jonathan,
I'm okay with putting this into branch-3.0 for GA if it can be merged
within the next two weeks. Even though beta1 has slipped by a month, I want
to stick to the targeted GA data of Nov 1st as much as possible. Of course,
let's not sacrifice quality or stability for speed; if something's not
ready, let's defer it to 3.1.0.
Subru, have you been able to review this feature from the 2.9.0
perspective? It'd add confidence if you think it's immediately ready for
merging to branch-2 for 2.9.0.
Thanks,
Andrew
On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
wrote:
> Hi everyone,
>
> Starting this thread to discuss merging API-based scheduler configuration
> to trunk/branch-2. The feature adds the framework for allowing users to
> modify scheduler configuration via REST or CLI using a configurable backend
> (leveldb/zk are currently supported), and adds capacity scheduler support
> for this. The umbrella JIRA is YARN-5734. All the required work for this
> feature is done and committed to branch YARN-5734, and a full diff has been
> generated at YARN-7241.
>
> Regarding compatibility, this feature is configurable and turned off by
> default.
>
> The feature has been tested locally on a couple RMs (since it is an RM
> only change), with queue addition/removal/updates tested on single RM
> (leveldb) and two RMs (zk). Also we verified the original configuration
> update mechanism (via refreshQueues) is unaffected when the feature is
> off/not configured.
>
> Our original plan was to merge this to trunk (which is what the YARN-7241
> diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
> what are your thoughts on also merging this to branch-3.0?
>
> Thanks!
>
> Jonathan Hung
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by Andrew Wang <an...@cloudera.com>.
Hi Jonathan,
I'm okay with putting this into branch-3.0 for GA if it can be merged
within the next two weeks. Even though beta1 has slipped by a month, I want
to stick to the targeted GA data of Nov 1st as much as possible. Of course,
let's not sacrifice quality or stability for speed; if something's not
ready, let's defer it to 3.1.0.
Subru, have you been able to review this feature from the 2.9.0
perspective? It'd add confidence if you think it's immediately ready for
merging to branch-2 for 2.9.0.
Thanks,
Andrew
On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
wrote:
> Hi everyone,
>
> Starting this thread to discuss merging API-based scheduler configuration
> to trunk/branch-2. The feature adds the framework for allowing users to
> modify scheduler configuration via REST or CLI using a configurable backend
> (leveldb/zk are currently supported), and adds capacity scheduler support
> for this. The umbrella JIRA is YARN-5734. All the required work for this
> feature is done and committed to branch YARN-5734, and a full diff has been
> generated at YARN-7241.
>
> Regarding compatibility, this feature is configurable and turned off by
> default.
>
> The feature has been tested locally on a couple RMs (since it is an RM
> only change), with queue addition/removal/updates tested on single RM
> (leveldb) and two RMs (zk). Also we verified the original configuration
> update mechanism (via refreshQueues) is unaffected when the feature is
> off/not configured.
>
> Our original plan was to merge this to trunk (which is what the YARN-7241
> diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
> what are your thoughts on also merging this to branch-3.0?
>
> Thanks!
>
> Jonathan Hung
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by Andrew Wang <an...@cloudera.com>.
Hi Jonathan,
I'm okay with putting this into branch-3.0 for GA if it can be merged
within the next two weeks. Even though beta1 has slipped by a month, I want
to stick to the targeted GA data of Nov 1st as much as possible. Of course,
let's not sacrifice quality or stability for speed; if something's not
ready, let's defer it to 3.1.0.
Subru, have you been able to review this feature from the 2.9.0
perspective? It'd add confidence if you think it's immediately ready for
merging to branch-2 for 2.9.0.
Thanks,
Andrew
On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
wrote:
> Hi everyone,
>
> Starting this thread to discuss merging API-based scheduler configuration
> to trunk/branch-2. The feature adds the framework for allowing users to
> modify scheduler configuration via REST or CLI using a configurable backend
> (leveldb/zk are currently supported), and adds capacity scheduler support
> for this. The umbrella JIRA is YARN-5734. All the required work for this
> feature is done and committed to branch YARN-5734, and a full diff has been
> generated at YARN-7241.
>
> Regarding compatibility, this feature is configurable and turned off by
> default.
>
> The feature has been tested locally on a couple RMs (since it is an RM
> only change), with queue addition/removal/updates tested on single RM
> (leveldb) and two RMs (zk). Also we verified the original configuration
> update mechanism (via refreshQueues) is unaffected when the feature is
> off/not configured.
>
> Our original plan was to merge this to trunk (which is what the YARN-7241
> diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
> what are your thoughts on also merging this to branch-3.0?
>
> Thanks!
>
> Jonathan Hung
>
Re: [DISCUSS] Merging API-based scheduler configuration to trunk/branch-2
Posted by Andrew Wang <an...@cloudera.com>.
Hi Jonathan,
I'm okay with putting this into branch-3.0 for GA if it can be merged
within the next two weeks. Even though beta1 has slipped by a month, I want
to stick to the targeted GA data of Nov 1st as much as possible. Of course,
let's not sacrifice quality or stability for speed; if something's not
ready, let's defer it to 3.1.0.
Subru, have you been able to review this feature from the 2.9.0
perspective? It'd add confidence if you think it's immediately ready for
merging to branch-2 for 2.9.0.
Thanks,
Andrew
On Thu, Sep 28, 2017 at 11:32 AM, Jonathan Hung <jy...@gmail.com>
wrote:
> Hi everyone,
>
> Starting this thread to discuss merging API-based scheduler configuration
> to trunk/branch-2. The feature adds the framework for allowing users to
> modify scheduler configuration via REST or CLI using a configurable backend
> (leveldb/zk are currently supported), and adds capacity scheduler support
> for this. The umbrella JIRA is YARN-5734. All the required work for this
> feature is done and committed to branch YARN-5734, and a full diff has been
> generated at YARN-7241.
>
> Regarding compatibility, this feature is configurable and turned off by
> default.
>
> The feature has been tested locally on a couple RMs (since it is an RM
> only change), with queue addition/removal/updates tested on single RM
> (leveldb) and two RMs (zk). Also we verified the original configuration
> update mechanism (via refreshQueues) is unaffected when the feature is
> off/not configured.
>
> Our original plan was to merge this to trunk (which is what the YARN-7241
> diff is based on), and port to branch-2 before the 2.9 release. @Andrew,
> what are your thoughts on also merging this to branch-3.0?
>
> Thanks!
>
> Jonathan Hung
>