You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Swapan Shridhar <ss...@hortonworks.com> on 2016/04/15 11:24:45 UTC
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/
-----------------------------------------------------------
(Updated April 15, 2016, 9:24 a.m.)
Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
Changes
-------
Modified the code as updated in Description:
===========================================
-- When Hive Server Interactive is turned ON:
Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
- if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
- if 2 queues exists at root level ('llap' and 'default') :
- 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
- 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
Modifies 'llap' queue capacity to 'llap_queue_capacity'.
-- When Hive Server Interactive is turned OFF:
- Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
and (2). 'llap' is in RUNNING state.
If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
(2). 'llap' queue capacity set to 0 %,
(3). 'default' queue capacity set to 100 %
TESTING : - PYTHON UT passes.
- Manual testing done. Various scenarios tested.
- Test cases added to test the creation and disabling logic.
Summary (updated)
-----------------
AMBARI-15876. Stack advisor work for adding and disabling LLAP queue when Interactive query is turned on and off respectively.
Bugs: AMBARI-15876
https://issues.apache.org/jira/browse/AMBARI-15876
Repository: ambari
Description (updated)
-------
-- When Hive Server Interactive is turned ON:
Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
- if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
- if 2 queues exists at root level ('llap' and 'default') :
- 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
- 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
Modifies 'llap' queue capacity to 'llap_queue_capacity'.
-- When Hive Server Interactive is turned OFF:
- Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
and (2). 'llap' is in RUNNING state.
If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
(2). 'llap' queue capacity set to 0 %,
(3). 'default' queue capacity set to 100 %
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
Diff: https://reviews.apache.org/r/46183/diff/
Testing (updated)
-------
- Python UT passes.
- Manual testing done. Various scenarios tested.
- Test cases added to test the creation and disabling logic.
- Screenshots attached.
File Attachments
----------------
LLAP queue created
https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
HSI On and llap and default queues shown in dropdown
https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
YARn queue manager with llap and default queues
https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
Thanks,
Swapan Shridhar
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review129168
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On April 15, 2016, 9:24 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 15, 2016, 9:24 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> -- When Hive Server Interactive is turned ON:
> Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
> - if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
> - if 2 queues exists at root level ('llap' and 'default') :
> - 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
> - 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
> Modifies 'llap' queue capacity to 'llap_queue_capacity'.
>
> -- When Hive Server Interactive is turned OFF:
> - Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
> and (2). 'llap' is in RUNNING state.
>
> If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
> (2). 'llap' queue capacity set to 0 %,
> (3). 'default' queue capacity set to 100 %
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and disabling logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review129231
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On April 15, 2016, 10:29 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 15, 2016, 10:29 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> -- When Hive Server Interactive is turned ON:
> Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
> - if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
> - if 2 queues exists at root level ('llap' and 'default') :
> - 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
> - 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
> Modifies 'llap' queue capacity to 'llap_queue_capacity'.
>
> -- When Hive Server Interactive is turned OFF:
> - Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
> and (2). 'llap' is in RUNNING state.
>
> If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
> (2). 'llap' queue capacity set to 0 %,
> (3). 'default' queue capacity set to 100 %
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and disabling logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/
-----------------------------------------------------------
(Updated April 15, 2016, 10:29 p.m.)
Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
Changes
-------
Alejandro's suggestion incorporated.
Bugs: AMBARI-15876
https://issues.apache.org/jira/browse/AMBARI-15876
Repository: ambari
Description
-------
-- When Hive Server Interactive is turned ON:
Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
- if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
- if 2 queues exists at root level ('llap' and 'default') :
- 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
- 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
Modifies 'llap' queue capacity to 'llap_queue_capacity'.
-- When Hive Server Interactive is turned OFF:
- Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
and (2). 'llap' is in RUNNING state.
If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
(2). 'llap' queue capacity set to 0 %,
(3). 'default' queue capacity set to 100 %
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
Diff: https://reviews.apache.org/r/46183/diff/
Testing
-------
- Python UT passes.
- Manual testing done. Various scenarios tested.
- Test cases added to test the creation and disabling logic.
- Screenshots attached.
File Attachments
----------------
LLAP queue created
https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
HSI On and llap and default queues shown in dropdown
https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
YARn queue manager with llap and default queues
https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
Thanks,
Swapan Shridhar
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
> On April 15, 2016, 6:30 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, line 24
> > <https://reviews.apache.org/r/46183/diff/3/?file=1346309#file1346309line24>
> >
> > Can we do this in the constructor?
Done.
- Swapan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review129170
-----------------------------------------------------------
On April 15, 2016, 10:29 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 15, 2016, 10:29 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> -- When Hive Server Interactive is turned ON:
> Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
> - if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
> - if 2 queues exists at root level ('llap' and 'default') :
> - 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
> - 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
> Modifies 'llap' queue capacity to 'llap_queue_capacity'.
>
> -- When Hive Server Interactive is turned OFF:
> - Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
> and (2). 'llap' is in RUNNING state.
>
> If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
> (2). 'llap' queue capacity set to 0 %,
> (3). 'default' queue capacity set to 100 %
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and disabling logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review129170
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 24)
<https://reviews.apache.org/r/46183/#comment192630>
Can we do this in the constructor?
- Alejandro Fernandez
On April 15, 2016, 9:24 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 15, 2016, 9:24 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> -- When Hive Server Interactive is turned ON:
> Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
> - if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
> - if 2 queues exists at root level ('llap' and 'default') :
> - 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
> - 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
> Modifies 'llap' queue capacity to 'llap_queue_capacity'.
>
> -- When Hive Server Interactive is turned OFF:
> - Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
> and (2). 'llap' is in RUNNING state.
>
> If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
> (2). 'llap' queue capacity set to 0 %,
> (3). 'default' queue capacity set to 100 %
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and disabling logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>