You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Jonathan Hung <jy...@gmail.com> on 2017/10/02 18:09:13 UTC
[VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Hi all,
From discussion at [1], I'd like to start a vote to merge feature branch
YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
Monday Oct 9 at 11:00AM PDT.
This branch adds a framework to the scheduler to allow scheduler
configuration mutation on the fly, including a REST and CLI interface, and
an interface for the scheduler configuration backing store. Currently the
capacity scheduler implements this framework.
Umbrella is here (YARN-5734
<https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here (
YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
tasks for this feature are committed. Since this feature changes RM only,
we have tested this on a local RM setup with a suite of configuration
changes with no issue so far.
Key points:
- The feature is turned off by default, and must be explicitly configured
to turn on. When turned off, the behavior reverts back to the original file
based mechanism for changing scheduler configuration (i.e. yarn rmadmin
-refreshQueues).
- The framework was designed in a way to be extendable to other schedulers
(most notably FairScheduler).
- A pluggable ACL policy (YARN-5949
<https://issues.apache.org/jira/browse/YARN-5949>) allows admins
fine-grained control for who can change what configurations.
- The configuration storage backend is also pluggable. Currently an
in-memory, leveldb, and zookeeper implementation are supported.
There were 15 subtasks completed for this feature.
Huge thanks to everyone who helped with reviews, commits, guidance, and
technical discussion/design, including Carlo Curino, Xuan Gong, Subru
Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
[1]
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
Jonathan Hung
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Wangda Tan <wh...@gmail.com>.
+1 (Binding), huge thanks to all the other folks spent time to design,
development, test and review this feature.
In addition to what Jonathan mentioned, all exposed APIs / CLIs are marked
as unstable so there should not have backward-compatibility issues.
Participated initial design and review all patches related to RM/scheduler,
CLI/Rest API and documentations. Considering scope of this patch is
relatively small (slightly over 200k) and Jonathan and folks from Linkedin
has done lots of validations to new logics, I'm supportive to merge this
into branch-2/branch-3.0 and trunk.
Thanks,
Wangda
On Mon, Oct 2, 2017 at 11:23 AM, Zhe Zhang <zh...@apache.org> wrote:
> +1 (binding)
>
> LinkedIn has been running an internal version of OrgQueue for almost 2
> years. It is an essential feature for running YARN scheduler on clusters
> shared by multiple internal organizations.
>
> I have participated in the YARN-5734 design and have regularly synced with
> Jonathan on development and implementation details. I believe the feature
> is of decent quality now and is ready to merge into trunk.
>
> On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jy...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > From discussion at [1], I'd like to start a vote to merge feature branch
> > YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> > Monday Oct 9 at 11:00AM PDT.
> >
> > This branch adds a framework to the scheduler to allow scheduler
> > configuration mutation on the fly, including a REST and CLI interface,
> and
> > an interface for the scheduler configuration backing store. Currently the
> > capacity scheduler implements this framework.
> >
> > Umbrella is here (YARN-5734
> > <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
> here
> > (
> > YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
> required
> > tasks for this feature are committed. Since this feature changes RM only,
> > we have tested this on a local RM setup with a suite of configuration
> > changes with no issue so far.
> >
> > Key points:
> > - The feature is turned off by default, and must be explicitly configured
> > to turn on. When turned off, the behavior reverts back to the original
> file
> > based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> > -refreshQueues).
> > - The framework was designed in a way to be extendable to other
> schedulers
> > (most notably FairScheduler).
> > - A pluggable ACL policy (YARN-5949
> > <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> > fine-grained control for who can change what configurations.
> > - The configuration storage backend is also pluggable. Currently an
> > in-memory, leveldb, and zookeeper implementation are supported.
> >
> > There were 15 subtasks completed for this feature.
> >
> > Huge thanks to everyone who helped with reviews, commits, guidance, and
> > technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> > Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
> Vinod
> > Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
> >
> > [1]
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
> >
> > Jonathan Hung
> >
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Wangda Tan <wh...@gmail.com>.
+1 (Binding), huge thanks to all the other folks spent time to design,
development, test and review this feature.
In addition to what Jonathan mentioned, all exposed APIs / CLIs are marked
as unstable so there should not have backward-compatibility issues.
Participated initial design and review all patches related to RM/scheduler,
CLI/Rest API and documentations. Considering scope of this patch is
relatively small (slightly over 200k) and Jonathan and folks from Linkedin
has done lots of validations to new logics, I'm supportive to merge this
into branch-2/branch-3.0 and trunk.
Thanks,
Wangda
On Mon, Oct 2, 2017 at 11:23 AM, Zhe Zhang <zh...@apache.org> wrote:
> +1 (binding)
>
> LinkedIn has been running an internal version of OrgQueue for almost 2
> years. It is an essential feature for running YARN scheduler on clusters
> shared by multiple internal organizations.
>
> I have participated in the YARN-5734 design and have regularly synced with
> Jonathan on development and implementation details. I believe the feature
> is of decent quality now and is ready to merge into trunk.
>
> On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jy...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > From discussion at [1], I'd like to start a vote to merge feature branch
> > YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> > Monday Oct 9 at 11:00AM PDT.
> >
> > This branch adds a framework to the scheduler to allow scheduler
> > configuration mutation on the fly, including a REST and CLI interface,
> and
> > an interface for the scheduler configuration backing store. Currently the
> > capacity scheduler implements this framework.
> >
> > Umbrella is here (YARN-5734
> > <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
> here
> > (
> > YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
> required
> > tasks for this feature are committed. Since this feature changes RM only,
> > we have tested this on a local RM setup with a suite of configuration
> > changes with no issue so far.
> >
> > Key points:
> > - The feature is turned off by default, and must be explicitly configured
> > to turn on. When turned off, the behavior reverts back to the original
> file
> > based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> > -refreshQueues).
> > - The framework was designed in a way to be extendable to other
> schedulers
> > (most notably FairScheduler).
> > - A pluggable ACL policy (YARN-5949
> > <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> > fine-grained control for who can change what configurations.
> > - The configuration storage backend is also pluggable. Currently an
> > in-memory, leveldb, and zookeeper implementation are supported.
> >
> > There were 15 subtasks completed for this feature.
> >
> > Huge thanks to everyone who helped with reviews, commits, guidance, and
> > technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> > Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
> Vinod
> > Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
> >
> > [1]
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
> >
> > Jonathan Hung
> >
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Wangda Tan <wh...@gmail.com>.
+1 (Binding), huge thanks to all the other folks spent time to design,
development, test and review this feature.
In addition to what Jonathan mentioned, all exposed APIs / CLIs are marked
as unstable so there should not have backward-compatibility issues.
Participated initial design and review all patches related to RM/scheduler,
CLI/Rest API and documentations. Considering scope of this patch is
relatively small (slightly over 200k) and Jonathan and folks from Linkedin
has done lots of validations to new logics, I'm supportive to merge this
into branch-2/branch-3.0 and trunk.
Thanks,
Wangda
On Mon, Oct 2, 2017 at 11:23 AM, Zhe Zhang <zh...@apache.org> wrote:
> +1 (binding)
>
> LinkedIn has been running an internal version of OrgQueue for almost 2
> years. It is an essential feature for running YARN scheduler on clusters
> shared by multiple internal organizations.
>
> I have participated in the YARN-5734 design and have regularly synced with
> Jonathan on development and implementation details. I believe the feature
> is of decent quality now and is ready to merge into trunk.
>
> On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jy...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > From discussion at [1], I'd like to start a vote to merge feature branch
> > YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> > Monday Oct 9 at 11:00AM PDT.
> >
> > This branch adds a framework to the scheduler to allow scheduler
> > configuration mutation on the fly, including a REST and CLI interface,
> and
> > an interface for the scheduler configuration backing store. Currently the
> > capacity scheduler implements this framework.
> >
> > Umbrella is here (YARN-5734
> > <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
> here
> > (
> > YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
> required
> > tasks for this feature are committed. Since this feature changes RM only,
> > we have tested this on a local RM setup with a suite of configuration
> > changes with no issue so far.
> >
> > Key points:
> > - The feature is turned off by default, and must be explicitly configured
> > to turn on. When turned off, the behavior reverts back to the original
> file
> > based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> > -refreshQueues).
> > - The framework was designed in a way to be extendable to other
> schedulers
> > (most notably FairScheduler).
> > - A pluggable ACL policy (YARN-5949
> > <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> > fine-grained control for who can change what configurations.
> > - The configuration storage backend is also pluggable. Currently an
> > in-memory, leveldb, and zookeeper implementation are supported.
> >
> > There were 15 subtasks completed for this feature.
> >
> > Huge thanks to everyone who helped with reviews, commits, guidance, and
> > technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> > Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
> Vinod
> > Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
> >
> > [1]
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
> >
> > Jonathan Hung
> >
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Wangda Tan <wh...@gmail.com>.
+1 (Binding), huge thanks to all the other folks spent time to design,
development, test and review this feature.
In addition to what Jonathan mentioned, all exposed APIs / CLIs are marked
as unstable so there should not have backward-compatibility issues.
Participated initial design and review all patches related to RM/scheduler,
CLI/Rest API and documentations. Considering scope of this patch is
relatively small (slightly over 200k) and Jonathan and folks from Linkedin
has done lots of validations to new logics, I'm supportive to merge this
into branch-2/branch-3.0 and trunk.
Thanks,
Wangda
On Mon, Oct 2, 2017 at 11:23 AM, Zhe Zhang <zh...@apache.org> wrote:
> +1 (binding)
>
> LinkedIn has been running an internal version of OrgQueue for almost 2
> years. It is an essential feature for running YARN scheduler on clusters
> shared by multiple internal organizations.
>
> I have participated in the YARN-5734 design and have regularly synced with
> Jonathan on development and implementation details. I believe the feature
> is of decent quality now and is ready to merge into trunk.
>
> On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jy...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > From discussion at [1], I'd like to start a vote to merge feature branch
> > YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> > Monday Oct 9 at 11:00AM PDT.
> >
> > This branch adds a framework to the scheduler to allow scheduler
> > configuration mutation on the fly, including a REST and CLI interface,
> and
> > an interface for the scheduler configuration backing store. Currently the
> > capacity scheduler implements this framework.
> >
> > Umbrella is here (YARN-5734
> > <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
> here
> > (
> > YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
> required
> > tasks for this feature are committed. Since this feature changes RM only,
> > we have tested this on a local RM setup with a suite of configuration
> > changes with no issue so far.
> >
> > Key points:
> > - The feature is turned off by default, and must be explicitly configured
> > to turn on. When turned off, the behavior reverts back to the original
> file
> > based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> > -refreshQueues).
> > - The framework was designed in a way to be extendable to other
> schedulers
> > (most notably FairScheduler).
> > - A pluggable ACL policy (YARN-5949
> > <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> > fine-grained control for who can change what configurations.
> > - The configuration storage backend is also pluggable. Currently an
> > in-memory, leveldb, and zookeeper implementation are supported.
> >
> > There were 15 subtasks completed for this feature.
> >
> > Huge thanks to everyone who helped with reviews, commits, guidance, and
> > technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> > Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
> Vinod
> > Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
> >
> > [1]
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
> >
> > Jonathan Hung
> >
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Zhe Zhang <zh...@apache.org>.
+1 (binding)
LinkedIn has been running an internal version of OrgQueue for almost 2
years. It is an essential feature for running YARN scheduler on clusters
shared by multiple internal organizations.
I have participated in the YARN-5734 design and have regularly synced with
Jonathan on development and implementation details. I believe the feature
is of decent quality now and is ready to merge into trunk.
On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jy...@gmail.com> wrote:
> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>
> Jonathan Hung
>
--
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Thanks for the votes and discussion. It is now past Monday Oct 9 11:00AM
PDT so the vote has ended. There were 4 +1 and no -1, so vote passes. This
feature will be merged to trunk, branch-3.0, and branch-2 shortly (16
subtasks).
Thanks everyone!
Jonathan Hung
On Mon, Oct 9, 2017 at 9:18 AM, Xuan Gong <xg...@hortonworks.com> wrote:
> +1 (binding)
>
> Xuan Gong
>
>
> >
> >On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com>
> >wrote:
> >
> >> Hi all,
> >>
> >> From discussion at [1], I'd like to start a vote to merge feature branch
> >> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days,
> >>ending
> >> Monday Oct 9 at 11:00AM PDT.
> >>
> >> This branch adds a framework to the scheduler to allow scheduler
> >> configuration mutation on the fly, including a REST and CLI interface,
> >>and
> >> an interface for the scheduler configuration backing store. Currently
> >>the
> >> capacity scheduler implements this framework.
> >>
> >> Umbrella is here (YARN-5734
> >> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
> >>here
> >> (
> >> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
> >>required
> >> tasks for this feature are committed. Since this feature changes RM
> >>only,
> >> we have tested this on a local RM setup with a suite of configuration
> >> changes with no issue so far.
> >>
> >> Key points:
> >> - The feature is turned off by default, and must be explicitly
> >>configured
> >> to turn on. When turned off, the behavior reverts back to the original
> >>file
> >> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> >> -refreshQueues).
> >> - The framework was designed in a way to be extendable to other
> >>schedulers
> >> (most notably FairScheduler).
> >> - A pluggable ACL policy (YARN-5949
> >> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> >> fine-grained control for who can change what configurations.
> >> - The configuration storage backend is also pluggable. Currently an
> >> in-memory, leveldb, and zookeeper implementation are supported.
> >>
> >> There were 15 subtasks completed for this feature.
> >>
> >> Huge thanks to everyone who helped with reviews, commits, guidance, and
> >> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> >> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
> >>Vinod
> >> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
> >>
> >> [1]
> >> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> >> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
> >>
> >> Jonathan Hung
> >>
>
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Thanks for the votes and discussion. It is now past Monday Oct 9 11:00AM
PDT so the vote has ended. There were 4 +1 and no -1, so vote passes. This
feature will be merged to trunk, branch-3.0, and branch-2 shortly (16
subtasks).
Thanks everyone!
Jonathan Hung
On Mon, Oct 9, 2017 at 9:18 AM, Xuan Gong <xg...@hortonworks.com> wrote:
> +1 (binding)
>
> Xuan Gong
>
>
> >
> >On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com>
> >wrote:
> >
> >> Hi all,
> >>
> >> From discussion at [1], I'd like to start a vote to merge feature branch
> >> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days,
> >>ending
> >> Monday Oct 9 at 11:00AM PDT.
> >>
> >> This branch adds a framework to the scheduler to allow scheduler
> >> configuration mutation on the fly, including a REST and CLI interface,
> >>and
> >> an interface for the scheduler configuration backing store. Currently
> >>the
> >> capacity scheduler implements this framework.
> >>
> >> Umbrella is here (YARN-5734
> >> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
> >>here
> >> (
> >> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
> >>required
> >> tasks for this feature are committed. Since this feature changes RM
> >>only,
> >> we have tested this on a local RM setup with a suite of configuration
> >> changes with no issue so far.
> >>
> >> Key points:
> >> - The feature is turned off by default, and must be explicitly
> >>configured
> >> to turn on. When turned off, the behavior reverts back to the original
> >>file
> >> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> >> -refreshQueues).
> >> - The framework was designed in a way to be extendable to other
> >>schedulers
> >> (most notably FairScheduler).
> >> - A pluggable ACL policy (YARN-5949
> >> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> >> fine-grained control for who can change what configurations.
> >> - The configuration storage backend is also pluggable. Currently an
> >> in-memory, leveldb, and zookeeper implementation are supported.
> >>
> >> There were 15 subtasks completed for this feature.
> >>
> >> Huge thanks to everyone who helped with reviews, commits, guidance, and
> >> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> >> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
> >>Vinod
> >> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
> >>
> >> [1]
> >> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> >> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
> >>
> >> Jonathan Hung
> >>
>
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Thanks for the votes and discussion. It is now past Monday Oct 9 11:00AM
PDT so the vote has ended. There were 4 +1 and no -1, so vote passes. This
feature will be merged to trunk, branch-3.0, and branch-2 shortly (16
subtasks).
Thanks everyone!
Jonathan Hung
On Mon, Oct 9, 2017 at 9:18 AM, Xuan Gong <xg...@hortonworks.com> wrote:
> +1 (binding)
>
> Xuan Gong
>
>
> >
> >On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com>
> >wrote:
> >
> >> Hi all,
> >>
> >> From discussion at [1], I'd like to start a vote to merge feature branch
> >> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days,
> >>ending
> >> Monday Oct 9 at 11:00AM PDT.
> >>
> >> This branch adds a framework to the scheduler to allow scheduler
> >> configuration mutation on the fly, including a REST and CLI interface,
> >>and
> >> an interface for the scheduler configuration backing store. Currently
> >>the
> >> capacity scheduler implements this framework.
> >>
> >> Umbrella is here (YARN-5734
> >> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
> >>here
> >> (
> >> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
> >>required
> >> tasks for this feature are committed. Since this feature changes RM
> >>only,
> >> we have tested this on a local RM setup with a suite of configuration
> >> changes with no issue so far.
> >>
> >> Key points:
> >> - The feature is turned off by default, and must be explicitly
> >>configured
> >> to turn on. When turned off, the behavior reverts back to the original
> >>file
> >> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> >> -refreshQueues).
> >> - The framework was designed in a way to be extendable to other
> >>schedulers
> >> (most notably FairScheduler).
> >> - A pluggable ACL policy (YARN-5949
> >> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> >> fine-grained control for who can change what configurations.
> >> - The configuration storage backend is also pluggable. Currently an
> >> in-memory, leveldb, and zookeeper implementation are supported.
> >>
> >> There were 15 subtasks completed for this feature.
> >>
> >> Huge thanks to everyone who helped with reviews, commits, guidance, and
> >> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> >> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
> >>Vinod
> >> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
> >>
> >> [1]
> >> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> >> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
> >>
> >> Jonathan Hung
> >>
>
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Jonathan Hung <jy...@gmail.com>.
Thanks for the votes and discussion. It is now past Monday Oct 9 11:00AM
PDT so the vote has ended. There were 4 +1 and no -1, so vote passes. This
feature will be merged to trunk, branch-3.0, and branch-2 shortly (16
subtasks).
Thanks everyone!
Jonathan Hung
On Mon, Oct 9, 2017 at 9:18 AM, Xuan Gong <xg...@hortonworks.com> wrote:
> +1 (binding)
>
> Xuan Gong
>
>
> >
> >On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com>
> >wrote:
> >
> >> Hi all,
> >>
> >> From discussion at [1], I'd like to start a vote to merge feature branch
> >> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days,
> >>ending
> >> Monday Oct 9 at 11:00AM PDT.
> >>
> >> This branch adds a framework to the scheduler to allow scheduler
> >> configuration mutation on the fly, including a REST and CLI interface,
> >>and
> >> an interface for the scheduler configuration backing store. Currently
> >>the
> >> capacity scheduler implements this framework.
> >>
> >> Umbrella is here (YARN-5734
> >> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
> >>here
> >> (
> >> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
> >>required
> >> tasks for this feature are committed. Since this feature changes RM
> >>only,
> >> we have tested this on a local RM setup with a suite of configuration
> >> changes with no issue so far.
> >>
> >> Key points:
> >> - The feature is turned off by default, and must be explicitly
> >>configured
> >> to turn on. When turned off, the behavior reverts back to the original
> >>file
> >> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> >> -refreshQueues).
> >> - The framework was designed in a way to be extendable to other
> >>schedulers
> >> (most notably FairScheduler).
> >> - A pluggable ACL policy (YARN-5949
> >> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> >> fine-grained control for who can change what configurations.
> >> - The configuration storage backend is also pluggable. Currently an
> >> in-memory, leveldb, and zookeeper implementation are supported.
> >>
> >> There were 15 subtasks completed for this feature.
> >>
> >> Huge thanks to everyone who helped with reviews, commits, guidance, and
> >> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> >> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
> >>Vinod
> >> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
> >>
> >> [1]
> >> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> >> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
> >>
> >> Jonathan Hung
> >>
>
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Xuan Gong <xg...@hortonworks.com>.
+1 (binding)
Xuan Gong
>
>On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com>
>wrote:
>
>> Hi all,
>>
>> From discussion at [1], I'd like to start a vote to merge feature branch
>> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days,
>>ending
>> Monday Oct 9 at 11:00AM PDT.
>>
>> This branch adds a framework to the scheduler to allow scheduler
>> configuration mutation on the fly, including a REST and CLI interface,
>>and
>> an interface for the scheduler configuration backing store. Currently
>>the
>> capacity scheduler implements this framework.
>>
>> Umbrella is here (YARN-5734
>> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
>>here
>> (
>> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
>>required
>> tasks for this feature are committed. Since this feature changes RM
>>only,
>> we have tested this on a local RM setup with a suite of configuration
>> changes with no issue so far.
>>
>> Key points:
>> - The feature is turned off by default, and must be explicitly
>>configured
>> to turn on. When turned off, the behavior reverts back to the original
>>file
>> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
>> -refreshQueues).
>> - The framework was designed in a way to be extendable to other
>>schedulers
>> (most notably FairScheduler).
>> - A pluggable ACL policy (YARN-5949
>> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
>> fine-grained control for who can change what configurations.
>> - The configuration storage backend is also pluggable. Currently an
>> in-memory, leveldb, and zookeeper implementation are supported.
>>
>> There were 15 subtasks completed for this feature.
>>
>> Huge thanks to everyone who helped with reviews, commits, guidance, and
>> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
>> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
>>Vinod
>> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
>> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>>
>> Jonathan Hung
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Xuan Gong <xg...@hortonworks.com>.
+1 (binding)
Xuan Gong
>
>On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com>
>wrote:
>
>> Hi all,
>>
>> From discussion at [1], I'd like to start a vote to merge feature branch
>> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days,
>>ending
>> Monday Oct 9 at 11:00AM PDT.
>>
>> This branch adds a framework to the scheduler to allow scheduler
>> configuration mutation on the fly, including a REST and CLI interface,
>>and
>> an interface for the scheduler configuration backing store. Currently
>>the
>> capacity scheduler implements this framework.
>>
>> Umbrella is here (YARN-5734
>> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
>>here
>> (
>> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
>>required
>> tasks for this feature are committed. Since this feature changes RM
>>only,
>> we have tested this on a local RM setup with a suite of configuration
>> changes with no issue so far.
>>
>> Key points:
>> - The feature is turned off by default, and must be explicitly
>>configured
>> to turn on. When turned off, the behavior reverts back to the original
>>file
>> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
>> -refreshQueues).
>> - The framework was designed in a way to be extendable to other
>>schedulers
>> (most notably FairScheduler).
>> - A pluggable ACL policy (YARN-5949
>> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
>> fine-grained control for who can change what configurations.
>> - The configuration storage backend is also pluggable. Currently an
>> in-memory, leveldb, and zookeeper implementation are supported.
>>
>> There were 15 subtasks completed for this feature.
>>
>> Huge thanks to everyone who helped with reviews, commits, guidance, and
>> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
>> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
>>Vinod
>> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
>> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>>
>> Jonathan Hung
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Xuan Gong <xg...@hortonworks.com>.
+1 (binding)
Xuan Gong
>
>On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com>
>wrote:
>
>> Hi all,
>>
>> From discussion at [1], I'd like to start a vote to merge feature branch
>> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days,
>>ending
>> Monday Oct 9 at 11:00AM PDT.
>>
>> This branch adds a framework to the scheduler to allow scheduler
>> configuration mutation on the fly, including a REST and CLI interface,
>>and
>> an interface for the scheduler configuration backing store. Currently
>>the
>> capacity scheduler implements this framework.
>>
>> Umbrella is here (YARN-5734
>> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
>>here
>> (
>> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
>>required
>> tasks for this feature are committed. Since this feature changes RM
>>only,
>> we have tested this on a local RM setup with a suite of configuration
>> changes with no issue so far.
>>
>> Key points:
>> - The feature is turned off by default, and must be explicitly
>>configured
>> to turn on. When turned off, the behavior reverts back to the original
>>file
>> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
>> -refreshQueues).
>> - The framework was designed in a way to be extendable to other
>>schedulers
>> (most notably FairScheduler).
>> - A pluggable ACL policy (YARN-5949
>> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
>> fine-grained control for who can change what configurations.
>> - The configuration storage backend is also pluggable. Currently an
>> in-memory, leveldb, and zookeeper implementation are supported.
>>
>> There were 15 subtasks completed for this feature.
>>
>> Huge thanks to everyone who helped with reviews, commits, guidance, and
>> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
>> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
>>Vinod
>> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
>> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>>
>> Jonathan Hung
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Xuan Gong <xg...@hortonworks.com>.
+1 (binding)
Xuan Gong
>
>On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com>
>wrote:
>
>> Hi all,
>>
>> From discussion at [1], I'd like to start a vote to merge feature branch
>> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days,
>>ending
>> Monday Oct 9 at 11:00AM PDT.
>>
>> This branch adds a framework to the scheduler to allow scheduler
>> configuration mutation on the fly, including a REST and CLI interface,
>>and
>> an interface for the scheduler configuration backing store. Currently
>>the
>> capacity scheduler implements this framework.
>>
>> Umbrella is here (YARN-5734
>> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is
>>here
>> (
>> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All
>>required
>> tasks for this feature are committed. Since this feature changes RM
>>only,
>> we have tested this on a local RM setup with a suite of configuration
>> changes with no issue so far.
>>
>> Key points:
>> - The feature is turned off by default, and must be explicitly
>>configured
>> to turn on. When turned off, the behavior reverts back to the original
>>file
>> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
>> -refreshQueues).
>> - The framework was designed in a way to be extendable to other
>>schedulers
>> (most notably FairScheduler).
>> - A pluggable ACL policy (YARN-5949
>> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
>> fine-grained control for who can change what configurations.
>> - The configuration storage backend is also pluggable. Currently an
>> in-memory, leveldb, and zookeeper implementation are supported.
>>
>> There were 15 subtasks completed for this feature.
>>
>> Huge thanks to everyone who helped with reviews, commits, guidance, and
>> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
>> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan,
>>Vinod
>> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
>> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>>
>> Jonathan Hung
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Konstantin Shvachko <sh...@gmail.com>.
+1 (binding)
I was following this effort closely from the side.
The feature proved to be very useful based on our internal use of it.
I think it will be beneficial for the community as well.
Thanks for working on this Jonathan, Wangda, and everybody involved.
--Konstantin
On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com> wrote:
> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>
> Jonathan Hung
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Konstantin Shvachko <sh...@gmail.com>.
+1 (binding)
I was following this effort closely from the side.
The feature proved to be very useful based on our internal use of it.
I think it will be beneficial for the community as well.
Thanks for working on this Jonathan, Wangda, and everybody involved.
--Konstantin
On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com> wrote:
> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>
> Jonathan Hung
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Zhe Zhang <zh...@apache.org>.
+1 (binding)
LinkedIn has been running an internal version of OrgQueue for almost 2
years. It is an essential feature for running YARN scheduler on clusters
shared by multiple internal organizations.
I have participated in the YARN-5734 design and have regularly synced with
Jonathan on development and implementation details. I believe the feature
is of decent quality now and is ready to merge into trunk.
On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jy...@gmail.com> wrote:
> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>
> Jonathan Hung
>
--
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Zhe Zhang <zh...@apache.org>.
+1 (binding)
LinkedIn has been running an internal version of OrgQueue for almost 2
years. It is an essential feature for running YARN scheduler on clusters
shared by multiple internal organizations.
I have participated in the YARN-5734 design and have regularly synced with
Jonathan on development and implementation details. I believe the feature
is of decent quality now and is ready to merge into trunk.
On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jy...@gmail.com> wrote:
> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>
> Jonathan Hung
>
--
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Konstantin Shvachko <sh...@gmail.com>.
+1 (binding)
I was following this effort closely from the side.
The feature proved to be very useful based on our internal use of it.
I think it will be beneficial for the community as well.
Thanks for working on this Jonathan, Wangda, and everybody involved.
--Konstantin
On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com> wrote:
> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>
> Jonathan Hung
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Konstantin Shvachko <sh...@gmail.com>.
+1 (binding)
I was following this effort closely from the side.
The feature proved to be very useful based on our internal use of it.
I think it will be beneficial for the community as well.
Thanks for working on this Jonathan, Wangda, and everybody involved.
--Konstantin
On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung <jy...@gmail.com> wrote:
> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>
> Jonathan Hung
>
Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler
configuration) to trunk, branch-3.0, branch-2
Posted by Zhe Zhang <zh...@apache.org>.
+1 (binding)
LinkedIn has been running an internal version of OrgQueue for almost 2
years. It is an essential feature for running YARN scheduler on clusters
shared by multiple internal organizations.
I have participated in the YARN-5734 design and have regularly synced with
Jonathan on development and implementation details. I believe the feature
is of decent quality now and is ready to merge into trunk.
On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jy...@gmail.com> wrote:
> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=PKEc9dPGGYX9rZF3NvDw@mail.gmail.com%3E
>
> Jonathan Hung
>
--
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap