You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by Sunil G <su...@apache.org> on 2017/11/30 01:56:20 UTC

[VOTE] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

Hi All,

Based on the discussion at [1], I'd like to start a vote to merge feature
branch

YARN-5881 to trunk. Vote will run for 7 days, ending Wednesday Dec 6 at
6:00PM PDT.

This branch adds support to configure queue capacity as absolute resource in

capacity scheduler. This will help admins who want fine control of
resources of queues.

Feature development is done at YARN-5881 [2], jenkins build is here
(YARN-7510 [3]).

All required tasks for this feature are committed. This feature changes
RM’s Capacity Scheduler only,

and we did extensive tests for the feature in the last couple of months
including performance tests.

Key points:

- The feature is turned off by default, and have to configure absolute
resource to enable same.

- Detailed documentation about how to use this feature is done as part of
[4].

- No major performance degradation is observed with this branch work. SLS
and UT performance

tests are done.

There were 11 subtasks completed for this feature.

Huge thanks to everyone who helped with reviews, commits, guidance, and

technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
Rohith Sharma K S, Eric Payne .

[1] :
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%3CCACYiTuhKhF1JCtR7ZFuZSEKQ4sBvN_n_tV5GHsbJ3YeyJP%2BP4Q%40mail.gmail.com%3E

[2] : https://issues.apache.org/jira/browse/YARN-5881

[3] : https://issues.apache.org/jira/browse/YARN-7510

[4] : https://issues.apache.org/jira/browse/YARN-7533

Regards

Sunil and Wangda

Re: [VOTE] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

Posted by Wangda Tan <wh...@gmail.com>.

+1, thanks Sunil!

Wangda Tan

On Wed, Nov 29, 2017 at 5:56 PM, Sunil G <su...@apache.org> wrote:

> Hi All,
>
>
> Based on the discussion at [1], I'd like to start a vote to merge feature
> branch
>
> YARN-5881 to trunk. Vote will run for 7 days, ending Wednesday Dec 6 at
> 6:00PM PDT.
>
>
> This branch adds support to configure queue capacity as absolute resource
> in
>
> capacity scheduler. This will help admins who want fine control of
> resources of queues.
>
>
> Feature development is done at YARN-5881 [2], jenkins build is here
> (YARN-7510 [3]).
>
> All required tasks for this feature are committed. This feature changes
> RM’s Capacity Scheduler only,
>
> and we did extensive tests for the feature in the last couple of months
> including performance tests.
>
>
> Key points:
>
> - The feature is turned off by default, and have to configure absolute
> resource to enable same.
>
> - Detailed documentation about how to use this feature is done as part of
> [4].
>
> - No major performance degradation is observed with this branch work. SLS
> and UT performance
>
> tests are done.
>
>
> There were 11 subtasks completed for this feature.
>
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
>
> technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
> Rohith Sharma K S, Eric Payne .
>
>
> [1] :
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%
> 3CCACYiTuhKhF1JCtR7ZFuZSEKQ4sBvN_n_tV5GHsbJ3YeyJP%2BP4Q%
> 40mail.gmail.com%3E
>
> [2] : https://issues.apache.org/jira/browse/YARN-5881
>
> [3] : https://issues.apache.org/jira/browse/YARN-7510
>
> [4] : https://issues.apache.org/jira/browse/YARN-7533
>
>
> Regards
>
> Sunil and Wangda
>

Re: [VOTE] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

Posted by Jian He <jh...@hortonworks.com>.

My late +1 

> On Dec 7, 2017, at 7:38 PM, Sunil G <su...@apache.org> wrote:
> 
> Thank You all.
> 
> We merged the branch to trunk and updated jiras accordingly. Thanks
> everyone who helped in this feature.
> 
> - Sunil and Wangda
> 
> 
> On Thu, Dec 7, 2017 at 10:13 PM Sunil G <su...@apache.org> wrote:
> 
>> And lastly +1 (binding) from myself.
>> Vote passes with 6 (+1) bindings by considering Weiwei's vote as binding
>> itself.
>> 
>> Thank you very much for all who voted. I’ll merge to trunk by the end of
>> today.
>> 
>> 
>> - Sunil
>> 
>> 
>> 
>> On Thu, Dec 7, 2017 at 8:08 AM Subramaniam V K <su...@gmail.com> wrote:
>> 
>>> +1.
>>> 
>>> Skimmed through the design doc and uber patch and seems to be reasonable.
>>> 
>>> This is a welcome addition especially w.r.t. cloud deployments so thanks
>>> to everyone who worked on this.
>>> 
>>> On Mon, Dec 4, 2017 at 8:18 PM, Rohith Sharma K S <
>>> rohithsharmaks@apache.org> wrote:
>>> 
>>>> +1
>>>> 
>>>> On Nov 30, 2017 7:26 AM, "Sunil G" <su...@apache.org> wrote:
>>>> 
>>>>> Hi All,
>>>>> 
>>>>> 
>>>>> Based on the discussion at [1], I'd like to start a vote to merge
>>>> feature
>>>>> branch
>>>>> 
>>>>> YARN-5881 to trunk. Vote will run for 7 days, ending Wednesday Dec 6 at
>>>>> 6:00PM PDT.
>>>>> 
>>>>> 
>>>>> This branch adds support to configure queue capacity as absolute
>>>> resource
>>>>> in
>>>>> 
>>>>> capacity scheduler. This will help admins who want fine control of
>>>>> resources of queues.
>>>>> 
>>>>> 
>>>>> Feature development is done at YARN-5881 [2], jenkins build is here
>>>>> (YARN-7510 [3]).
>>>>> 
>>>>> All required tasks for this feature are committed. This feature changes
>>>>> RM’s Capacity Scheduler only,
>>>>> 
>>>>> and we did extensive tests for the feature in the last couple of months
>>>>> including performance tests.
>>>>> 
>>>>> 
>>>>> Key points:
>>>>> 
>>>>> - The feature is turned off by default, and have to configure absolute
>>>>> resource to enable same.
>>>>> 
>>>>> - Detailed documentation about how to use this feature is done as part
>>>> of
>>>>> [4].
>>>>> 
>>>>> - No major performance degradation is observed with this branch work.
>>>> SLS
>>>>> and UT performance
>>>>> 
>>>>> tests are done.
>>>>> 
>>>>> 
>>>>> There were 11 subtasks completed for this feature.
>>>>> 
>>>>> 
>>>>> Huge thanks to everyone who helped with reviews, commits, guidance, and
>>>>> 
>>>>> technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
>>>>> Rohith Sharma K S, Eric Payne .
>>>>> 
>>>>> 
>>>>> [1] :
>>>>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%
>>>>> 3CCACYiTuhKhF1JCtR7ZFuZSEKQ4sBvN_n_tV5GHsbJ3YeyJP%2BP4Q%
>>>>> 40mail.gmail.com%3E
>>>>> 
>>>>> [2] : https://issues.apache.org/jira/browse/YARN-5881
>>>>> 
>>>>> [3] : https://issues.apache.org/jira/browse/YARN-7510
>>>>> 
>>>>> [4] : https://issues.apache.org/jira/browse/YARN-7533
>>>>> 
>>>>> 
>>>>> Regards
>>>>> 
>>>>> Sunil and Wangda
>>>>> 
>>>> 
>>> 
>>>

Re: [VOTE] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

Posted by Jian He <jh...@hortonworks.com>.

My late +1 

> On Dec 7, 2017, at 7:38 PM, Sunil G <su...@apache.org> wrote:
> 
> Thank You all.
> 
> We merged the branch to trunk and updated jiras accordingly. Thanks
> everyone who helped in this feature.
> 
> - Sunil and Wangda
> 
> 
> On Thu, Dec 7, 2017 at 10:13 PM Sunil G <su...@apache.org> wrote:
> 
>> And lastly +1 (binding) from myself.
>> Vote passes with 6 (+1) bindings by considering Weiwei's vote as binding
>> itself.
>> 
>> Thank you very much for all who voted. I’ll merge to trunk by the end of
>> today.
>> 
>> 
>> - Sunil
>> 
>> 
>> 
>> On Thu, Dec 7, 2017 at 8:08 AM Subramaniam V K <su...@gmail.com> wrote:
>> 
>>> +1.
>>> 
>>> Skimmed through the design doc and uber patch and seems to be reasonable.
>>> 
>>> This is a welcome addition especially w.r.t. cloud deployments so thanks
>>> to everyone who worked on this.
>>> 
>>> On Mon, Dec 4, 2017 at 8:18 PM, Rohith Sharma K S <
>>> rohithsharmaks@apache.org> wrote:
>>> 
>>>> +1
>>>> 
>>>> On Nov 30, 2017 7:26 AM, "Sunil G" <su...@apache.org> wrote:
>>>> 
>>>>> Hi All,
>>>>> 
>>>>> 
>>>>> Based on the discussion at [1], I'd like to start a vote to merge
>>>> feature
>>>>> branch
>>>>> 
>>>>> YARN-5881 to trunk. Vote will run for 7 days, ending Wednesday Dec 6 at
>>>>> 6:00PM PDT.
>>>>> 
>>>>> 
>>>>> This branch adds support to configure queue capacity as absolute
>>>> resource
>>>>> in
>>>>> 
>>>>> capacity scheduler. This will help admins who want fine control of
>>>>> resources of queues.
>>>>> 
>>>>> 
>>>>> Feature development is done at YARN-5881 [2], jenkins build is here
>>>>> (YARN-7510 [3]).
>>>>> 
>>>>> All required tasks for this feature are committed. This feature changes
>>>>> RM’s Capacity Scheduler only,
>>>>> 
>>>>> and we did extensive tests for the feature in the last couple of months
>>>>> including performance tests.
>>>>> 
>>>>> 
>>>>> Key points:
>>>>> 
>>>>> - The feature is turned off by default, and have to configure absolute
>>>>> resource to enable same.
>>>>> 
>>>>> - Detailed documentation about how to use this feature is done as part
>>>> of
>>>>> [4].
>>>>> 
>>>>> - No major performance degradation is observed with this branch work.
>>>> SLS
>>>>> and UT performance
>>>>> 
>>>>> tests are done.
>>>>> 
>>>>> 
>>>>> There were 11 subtasks completed for this feature.
>>>>> 
>>>>> 
>>>>> Huge thanks to everyone who helped with reviews, commits, guidance, and
>>>>> 
>>>>> technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
>>>>> Rohith Sharma K S, Eric Payne .
>>>>> 
>>>>> 
>>>>> [1] :
>>>>> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%
>>>>> 3CCACYiTuhKhF1JCtR7ZFuZSEKQ4sBvN_n_tV5GHsbJ3YeyJP%2BP4Q%
>>>>> 40mail.gmail.com%3E
>>>>> 
>>>>> [2] : https://issues.apache.org/jira/browse/YARN-5881
>>>>> 
>>>>> [3] : https://issues.apache.org/jira/browse/YARN-7510
>>>>> 
>>>>> [4] : https://issues.apache.org/jira/browse/YARN-7533
>>>>> 
>>>>> 
>>>>> Regards
>>>>> 
>>>>> Sunil and Wangda
>>>>> 
>>>> 
>>> 
>>>