You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Swapan Shridhar <ss...@hortonworks.com> on 2017/01/27 00:09:35 UTC
Review Request 56003: AMBARI-19737. Add a validation check in HSI
about Capacity Scheduler preemption requirements.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56003/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
Bugs: AMBARI-19737
https://issues.apache.org/jira/browse/AMBARI-19737
Repository: ambari
Description
-------
Capacity Scheduler preemption is not enabled by default in HDP. Enabling it affects the entire cluster and LLAP is of concern in the bug.
**Requirement:**
- Add a validation check about CS preemption configuration, while LLAP is not enabled.
**Message:**
Enabling YARN config 'yarn.resourcemanager.scheduler.monitor.enable' is recommended if enabling LLAP, to ensure it is able to get cluster capacity.
If CS preemption is enabled, no validation check required.
-> Also, uncommenting and fixing the test code in test_stack_advisor::test_validateHiveInteractiveSiteConfigurations().
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 70da914
ambari-server/src/test/python/stacks/2.5/common/services-normal-his-2-hosts.json 46a6f4d
ambari-server/src/test/python/stacks/2.5/common/services-normal-his-valid.json 7955333
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py a53cb25
Diff: https://reviews.apache.org/r/56003/diff/
Testing
-------
Yes.
Python UT added. Passes
Tested on cluster.
Screenshots
===========
When preemption off: **WARN msg.jpg**
When preemption off: **No WARN msg.jpg**
File Attachments
----------------
WARN msg
https://reviews.apache.org/media/uploaded/files/2017/01/27/d8eb1277-da18-4ed0-b821-8ddbc42256ce__WARN_msg.png
No WARN msg
https://reviews.apache.org/media/uploaded/files/2017/01/27/81673b84-212e-4cbe-b6d0-d6772ab51c19__No_WARN_msg.jpg
Thanks,
Swapan Shridhar
Re: Review Request 56003: AMBARI-19737. Add a validation check in HSI
about Capacity Scheduler preemption requirements.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
> On Jan. 27, 2017, 12:14 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, line 420
> > <https://reviews.apache.org/r/56003/diff/1/?file=1617155#file1617155line420>
> >
> > Nit: Change to - When enabling LLAP, set yarn.resourcemanager.scheduler.monitor.enable to true to ensure that LLAP gets the full allocated capacity.
Done.
- Swapan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56003/#review163207
-----------------------------------------------------------
On Jan. 27, 2017, 12:19 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56003/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2017, 12:19 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19737
> https://issues.apache.org/jira/browse/AMBARI-19737
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Capacity Scheduler preemption is not enabled by default in HDP. Enabling it affects the entire cluster and LLAP is of concern in the bug.
>
> **Requirement:**
> - Add a validation check about CS preemption configuration, while LLAP is not enabled.
>
> **Message:**
>
> Enabling YARN config 'yarn.resourcemanager.scheduler.monitor.enable' is recommended if enabling LLAP, to ensure it is able to get cluster capacity.
>
> If CS preemption is enabled, no validation check required.
>
>
> -> Also, uncommenting and fixing the test code in test_stack_advisor::test_validateHiveInteractiveSiteConfigurations().
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 70da914
> ambari-server/src/test/python/stacks/2.5/common/services-normal-his-2-hosts.json 46a6f4d
> ambari-server/src/test/python/stacks/2.5/common/services-normal-his-valid.json 7955333
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py a53cb25
>
> Diff: https://reviews.apache.org/r/56003/diff/
>
>
> Testing
> -------
>
> Yes.
> Python UT added. Passes
> Tested on cluster.
>
>
> Screenshots
> ===========
>
> When preemption off: **WARN msg.jpg**
> When preemption off: **No WARN msg.jpg**
>
>
> File Attachments
> ----------------
>
> WARN msg
> https://reviews.apache.org/media/uploaded/files/2017/01/27/d8eb1277-da18-4ed0-b821-8ddbc42256ce__WARN_msg.png
> No WARN msg
> https://reviews.apache.org/media/uploaded/files/2017/01/27/81673b84-212e-4cbe-b6d0-d6772ab51c19__No_WARN_msg.jpg
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56003: AMBARI-19737. Add a validation check in HSI
about Capacity Scheduler preemption requirements.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56003/#review163207
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 420)
<https://reviews.apache.org/r/56003/#comment234669>
Nit: Change to - When enabling LLAP, set yarn.resourcemanager.scheduler.monitor.enable to true to ensure that LLAP gets the full allocated capacity.
- Sumit Mohanty
On Jan. 27, 2017, 12:09 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56003/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2017, 12:09 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19737
> https://issues.apache.org/jira/browse/AMBARI-19737
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Capacity Scheduler preemption is not enabled by default in HDP. Enabling it affects the entire cluster and LLAP is of concern in the bug.
>
> **Requirement:**
> - Add a validation check about CS preemption configuration, while LLAP is not enabled.
>
> **Message:**
>
> Enabling YARN config 'yarn.resourcemanager.scheduler.monitor.enable' is recommended if enabling LLAP, to ensure it is able to get cluster capacity.
>
> If CS preemption is enabled, no validation check required.
>
>
> -> Also, uncommenting and fixing the test code in test_stack_advisor::test_validateHiveInteractiveSiteConfigurations().
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 70da914
> ambari-server/src/test/python/stacks/2.5/common/services-normal-his-2-hosts.json 46a6f4d
> ambari-server/src/test/python/stacks/2.5/common/services-normal-his-valid.json 7955333
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py a53cb25
>
> Diff: https://reviews.apache.org/r/56003/diff/
>
>
> Testing
> -------
>
> Yes.
> Python UT added. Passes
> Tested on cluster.
>
>
> Screenshots
> ===========
>
> When preemption off: **WARN msg.jpg**
> When preemption off: **No WARN msg.jpg**
>
>
> File Attachments
> ----------------
>
> WARN msg
> https://reviews.apache.org/media/uploaded/files/2017/01/27/d8eb1277-da18-4ed0-b821-8ddbc42256ce__WARN_msg.png
> No WARN msg
> https://reviews.apache.org/media/uploaded/files/2017/01/27/81673b84-212e-4cbe-b6d0-d6772ab51c19__No_WARN_msg.jpg
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56003: AMBARI-19737. Add a validation check in HSI
about Capacity Scheduler preemption requirements.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56003/#review163209
-----------------------------------------------------------
Ship it!
Fix it then ship it
- Sumit Mohanty
On Jan. 27, 2017, 12:09 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56003/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2017, 12:09 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19737
> https://issues.apache.org/jira/browse/AMBARI-19737
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Capacity Scheduler preemption is not enabled by default in HDP. Enabling it affects the entire cluster and LLAP is of concern in the bug.
>
> **Requirement:**
> - Add a validation check about CS preemption configuration, while LLAP is not enabled.
>
> **Message:**
>
> Enabling YARN config 'yarn.resourcemanager.scheduler.monitor.enable' is recommended if enabling LLAP, to ensure it is able to get cluster capacity.
>
> If CS preemption is enabled, no validation check required.
>
>
> -> Also, uncommenting and fixing the test code in test_stack_advisor::test_validateHiveInteractiveSiteConfigurations().
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 70da914
> ambari-server/src/test/python/stacks/2.5/common/services-normal-his-2-hosts.json 46a6f4d
> ambari-server/src/test/python/stacks/2.5/common/services-normal-his-valid.json 7955333
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py a53cb25
>
> Diff: https://reviews.apache.org/r/56003/diff/
>
>
> Testing
> -------
>
> Yes.
> Python UT added. Passes
> Tested on cluster.
>
>
> Screenshots
> ===========
>
> When preemption off: **WARN msg.jpg**
> When preemption off: **No WARN msg.jpg**
>
>
> File Attachments
> ----------------
>
> WARN msg
> https://reviews.apache.org/media/uploaded/files/2017/01/27/d8eb1277-da18-4ed0-b821-8ddbc42256ce__WARN_msg.png
> No WARN msg
> https://reviews.apache.org/media/uploaded/files/2017/01/27/81673b84-212e-4cbe-b6d0-d6772ab51c19__No_WARN_msg.jpg
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56003: AMBARI-19737. Add a validation check in HSI
about Capacity Scheduler preemption requirements.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56003/#review163214
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On Jan. 27, 2017, 12:19 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56003/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2017, 12:19 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19737
> https://issues.apache.org/jira/browse/AMBARI-19737
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Capacity Scheduler preemption is not enabled by default in HDP. Enabling it affects the entire cluster and LLAP is of concern in the bug.
>
> **Requirement:**
> - Add a validation check about CS preemption configuration, while LLAP is not enabled.
>
> **Message:**
>
> Enabling YARN config 'yarn.resourcemanager.scheduler.monitor.enable' is recommended if enabling LLAP, to ensure it is able to get cluster capacity.
>
> If CS preemption is enabled, no validation check required.
>
>
> -> Also, uncommenting and fixing the test code in test_stack_advisor::test_validateHiveInteractiveSiteConfigurations().
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 70da914
> ambari-server/src/test/python/stacks/2.5/common/services-normal-his-2-hosts.json 46a6f4d
> ambari-server/src/test/python/stacks/2.5/common/services-normal-his-valid.json 7955333
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py a53cb25
>
> Diff: https://reviews.apache.org/r/56003/diff/
>
>
> Testing
> -------
>
> Yes.
> Python UT added. Passes
> Tested on cluster.
>
>
> Screenshots
> ===========
>
> When preemption off: **WARN msg.jpg**
> When preemption off: **No WARN msg.jpg**
>
>
> File Attachments
> ----------------
>
> WARN msg
> https://reviews.apache.org/media/uploaded/files/2017/01/27/d8eb1277-da18-4ed0-b821-8ddbc42256ce__WARN_msg.png
> No WARN msg
> https://reviews.apache.org/media/uploaded/files/2017/01/27/81673b84-212e-4cbe-b6d0-d6772ab51c19__No_WARN_msg.jpg
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56003: AMBARI-19737. Add a validation check in HSI
about Capacity Scheduler preemption requirements.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56003/
-----------------------------------------------------------
(Updated Jan. 27, 2017, 12:19 a.m.)
Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
Changes
-------
Changed as per review suggestions.
Bugs: AMBARI-19737
https://issues.apache.org/jira/browse/AMBARI-19737
Repository: ambari
Description
-------
Capacity Scheduler preemption is not enabled by default in HDP. Enabling it affects the entire cluster and LLAP is of concern in the bug.
**Requirement:**
- Add a validation check about CS preemption configuration, while LLAP is not enabled.
**Message:**
Enabling YARN config 'yarn.resourcemanager.scheduler.monitor.enable' is recommended if enabling LLAP, to ensure it is able to get cluster capacity.
If CS preemption is enabled, no validation check required.
-> Also, uncommenting and fixing the test code in test_stack_advisor::test_validateHiveInteractiveSiteConfigurations().
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 70da914
ambari-server/src/test/python/stacks/2.5/common/services-normal-his-2-hosts.json 46a6f4d
ambari-server/src/test/python/stacks/2.5/common/services-normal-his-valid.json 7955333
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py a53cb25
Diff: https://reviews.apache.org/r/56003/diff/
Testing
-------
Yes.
Python UT added. Passes
Tested on cluster.
Screenshots
===========
When preemption off: **WARN msg.jpg**
When preemption off: **No WARN msg.jpg**
File Attachments
----------------
WARN msg
https://reviews.apache.org/media/uploaded/files/2017/01/27/d8eb1277-da18-4ed0-b821-8ddbc42256ce__WARN_msg.png
No WARN msg
https://reviews.apache.org/media/uploaded/files/2017/01/27/81673b84-212e-4cbe-b6d0-d6772ab51c19__No_WARN_msg.jpg
Thanks,
Swapan Shridhar