You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Swapan Shridhar <ss...@hortonworks.com> on 2016/04/14 05:06:24 UTC
Review Request 46183: AMBARI-15876. Stack advisor work for adding and
removing LLAP queue when Interactive query is turned on and off
respectively.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
Bugs: AMBARI-15876
https://issues.apache.org/jira/browse/AMBARI-15876
Repository: ambari
Description
-------
- When Hive Server Interactive is turned ON, Ambari will create 'llap' named YARN queue if only 'default' queue exists at root level and is consuming 100% capacity.
- When Hive Server Interactive is turned OFF, Ambari will delete 'llap' queue only if 2 queues exist at root level namely, 'llap' and 'default'.
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
Diff: https://reviews.apache.org/r/46183/diff/
Testing
-------
- Python UT passes.
- Manual testing done. Various scenarios tested.
- Test cases added to test the creation and deletion logic.
- Screenshots attached.
File Attachments
----------------
LLAP queue created
https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
HSI On and llap and default queues shown in dropdown
https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
YARn queue manager with llap and default queues
https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
Thanks,
Swapan Shridhar
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and
removing LLAP queue when Interactive query is turned on and off respectively.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
> On April 14, 2016, 6:03 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml, line 360
> > <https://reviews.apache.org/r/46183/diff/2/?file=1344004#file1344004line360>
> >
> > Is this correct, cluster_name?
Yes. Suggested by Sid.
> On April 14, 2016, 6:03 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml, line 28
> > <https://reviews.apache.org/r/46183/diff/2/?file=1344005#file1344005line28>
> >
> > This means that ambari upgrade will not add this property, but because it is new, then stack upgrade from HDP 2.4 to 2.5 will add it. Is that correct?
> > DOes it need a value?
It doesn't need a value. Used for the reaosn that initially capacity-scheduler reads props from its xml and later in lifecycle Ambari UI and backend talks in terms of all properties coalesced as a bag of string (key=value\nkey=value\nkey=value\n....). So, this becomes a placeholder for setting values at backend and be passed back to UI.
- Swapan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review128936
-----------------------------------------------------------
On April 14, 2016, 9:01 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 9:01 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> - When Hive Server Interactive is turned ON, Ambari will create 'llap' named YARN queue if only 'default' queue exists at root level and is consuming 100% capacity.
> - When Hive Server Interactive is turned OFF, Ambari will delete 'llap' queue only if 2 queues exist at root level namely, 'llap' and 'default'.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and deletion logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and
removing LLAP queue when Interactive query is turned on and off respectively.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review128936
-----------------------------------------------------------
Fix it, then Ship it!
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml (line 360)
<https://reviews.apache.org/r/46183/#comment192384>
Is this correct, cluster_name?
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml (line 28)
<https://reviews.apache.org/r/46183/#comment192386>
This means that ambari upgrade will not add this property, but because it is new, then stack upgrade from HDP 2.4 to 2.5 will add it. Is that correct?
DOes it need a value?
- Alejandro Fernandez
On April 14, 2016, 9:01 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 9:01 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> - When Hive Server Interactive is turned ON, Ambari will create 'llap' named YARN queue if only 'default' queue exists at root level and is consuming 100% capacity.
> - When Hive Server Interactive is turned OFF, Ambari will delete 'llap' queue only if 2 queues exist at root level namely, 'llap' and 'default'.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and deletion logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review129168
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On April 15, 2016, 9:24 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 15, 2016, 9:24 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> -- When Hive Server Interactive is turned ON:
> Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
> - if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
> - if 2 queues exists at root level ('llap' and 'default') :
> - 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
> - 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
> Modifies 'llap' queue capacity to 'llap_queue_capacity'.
>
> -- When Hive Server Interactive is turned OFF:
> - Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
> and (2). 'llap' is in RUNNING state.
>
> If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
> (2). 'llap' queue capacity set to 0 %,
> (3). 'default' queue capacity set to 100 %
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and disabling logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review129231
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On April 15, 2016, 10:29 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 15, 2016, 10:29 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> -- When Hive Server Interactive is turned ON:
> Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
> - if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
> - if 2 queues exists at root level ('llap' and 'default') :
> - 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
> - 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
> Modifies 'llap' queue capacity to 'llap_queue_capacity'.
>
> -- When Hive Server Interactive is turned OFF:
> - Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
> and (2). 'llap' is in RUNNING state.
>
> If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
> (2). 'llap' queue capacity set to 0 %,
> (3). 'default' queue capacity set to 100 %
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and disabling logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/
-----------------------------------------------------------
(Updated April 15, 2016, 10:29 p.m.)
Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
Changes
-------
Alejandro's suggestion incorporated.
Bugs: AMBARI-15876
https://issues.apache.org/jira/browse/AMBARI-15876
Repository: ambari
Description
-------
-- When Hive Server Interactive is turned ON:
Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
- if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
- if 2 queues exists at root level ('llap' and 'default') :
- 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
- 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
Modifies 'llap' queue capacity to 'llap_queue_capacity'.
-- When Hive Server Interactive is turned OFF:
- Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
and (2). 'llap' is in RUNNING state.
If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
(2). 'llap' queue capacity set to 0 %,
(3). 'default' queue capacity set to 100 %
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
Diff: https://reviews.apache.org/r/46183/diff/
Testing
-------
- Python UT passes.
- Manual testing done. Various scenarios tested.
- Test cases added to test the creation and disabling logic.
- Screenshots attached.
File Attachments
----------------
LLAP queue created
https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
HSI On and llap and default queues shown in dropdown
https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
YARn queue manager with llap and default queues
https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
Thanks,
Swapan Shridhar
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
> On April 15, 2016, 6:30 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, line 24
> > <https://reviews.apache.org/r/46183/diff/3/?file=1346309#file1346309line24>
> >
> > Can we do this in the constructor?
Done.
- Swapan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review129170
-----------------------------------------------------------
On April 15, 2016, 10:29 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 15, 2016, 10:29 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> -- When Hive Server Interactive is turned ON:
> Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
> - if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
> - if 2 queues exists at root level ('llap' and 'default') :
> - 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
> - 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
> Modifies 'llap' queue capacity to 'llap_queue_capacity'.
>
> -- When Hive Server Interactive is turned OFF:
> - Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
> and (2). 'llap' is in RUNNING state.
>
> If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
> (2). 'llap' queue capacity set to 0 %,
> (3). 'default' queue capacity set to 100 %
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and disabling logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review129170
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 24)
<https://reviews.apache.org/r/46183/#comment192630>
Can we do this in the constructor?
- Alejandro Fernandez
On April 15, 2016, 9:24 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 15, 2016, 9:24 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> -- When Hive Server Interactive is turned ON:
> Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
> - if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
> - if 2 queues exists at root level ('llap' and 'default') :
> - 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
> - 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
> Modifies 'llap' queue capacity to 'llap_queue_capacity'.
>
> -- When Hive Server Interactive is turned OFF:
> - Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
> and (2). 'llap' is in RUNNING state.
>
> If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
> (2). 'llap' queue capacity set to 0 %,
> (3). 'default' queue capacity set to 100 %
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and disabling logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and disabling LLAP queue when Interactive query is turned on and off
respectively.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/
-----------------------------------------------------------
(Updated April 15, 2016, 9:24 a.m.)
Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
Changes
-------
Modified the code as updated in Description:
===========================================
-- When Hive Server Interactive is turned ON:
Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
- if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
- if 2 queues exists at root level ('llap' and 'default') :
- 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
- 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
Modifies 'llap' queue capacity to 'llap_queue_capacity'.
-- When Hive Server Interactive is turned OFF:
- Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
and (2). 'llap' is in RUNNING state.
If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
(2). 'llap' queue capacity set to 0 %,
(3). 'default' queue capacity set to 100 %
TESTING : - PYTHON UT passes.
- Manual testing done. Various scenarios tested.
- Test cases added to test the creation and disabling logic.
Summary (updated)
-----------------
AMBARI-15876. Stack advisor work for adding and disabling LLAP queue when Interactive query is turned on and off respectively.
Bugs: AMBARI-15876
https://issues.apache.org/jira/browse/AMBARI-15876
Repository: ambari
Description (updated)
-------
-- When Hive Server Interactive is turned ON:
Ambari creates OR "modifies 'llap' queue 'state and/or capacity' " based on below conditions:
- if only 1 queue exists at root level and is 'default' queue and has 100% cap -> Creates 'llap' queue, OR
- if 2 queues exists at root level ('llap' and 'default') :
- 'llap' queue state is STOPPED -> Modifies 'llap' queue state to RUNNING, adjusts capacity, OR
- 'llap' queue state is RUNNING and 'llap_queue_capacity' prop != 'llap' queue current running capacity ->
Modifies 'llap' queue capacity to 'llap_queue_capacity'.
-- When Hive Server Interactive is turned OFF:
- Checks and sees (1). If only two leaf queues exist at root level, namely: 'default' and 'llap',
and (2). 'llap' is in RUNNING state.
If yes, performs the following actions: (1). 'llap' queue state set to STOPPED,
(2). 'llap' queue capacity set to 0 %,
(3). 'default' queue capacity set to 100 %
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
Diff: https://reviews.apache.org/r/46183/diff/
Testing (updated)
-------
- Python UT passes.
- Manual testing done. Various scenarios tested.
- Test cases added to test the creation and disabling logic.
- Screenshots attached.
File Attachments
----------------
LLAP queue created
https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
HSI On and llap and default queues shown in dropdown
https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
YARn queue manager with llap and default queues
https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
Thanks,
Swapan Shridhar
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and
removing LLAP queue when Interactive query is turned on and off respectively.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/
-----------------------------------------------------------
(Updated April 14, 2016, 9:01 a.m.)
Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
Changes
-------
- Suggestions incorporated as per Sumit's review.
- Testing : Python UT passes.
Bugs: AMBARI-15876
https://issues.apache.org/jira/browse/AMBARI-15876
Repository: ambari
Description
-------
- When Hive Server Interactive is turned ON, Ambari will create 'llap' named YARN queue if only 'default' queue exists at root level and is consuming 100% capacity.
- When Hive Server Interactive is turned OFF, Ambari will delete 'llap' queue only if 2 queues exist at root level namely, 'llap' and 'default'.
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
Diff: https://reviews.apache.org/r/46183/diff/
Testing
-------
- Python UT passes.
- Manual testing done. Various scenarios tested.
- Test cases added to test the creation and deletion logic.
- Screenshots attached.
File Attachments
----------------
LLAP queue created
https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
HSI On and llap and default queues shown in dropdown
https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
YARn queue manager with llap and default queues
https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
Thanks,
Swapan Shridhar
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and
removing LLAP queue when Interactive query is turned on and off respectively.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
> On April 14, 2016, 4:12 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, line 128
> > <https://reviews.apache.org/r/46183/diff/1/?file=1343728#file1343728line128>
> >
> > Lets protect the whole logic based on if HIVE server exists and HSI is mapped to a host. Its possible to add hive-interactive-site without adding service/component.
Done.
> On April 14, 2016, 4:12 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, line 145
> > <https://reviews.apache.org/r/46183/diff/1/?file=1343728#file1343728line145>
> >
> > Similar to above - check that service exists, component exists and then configuration exists. Perhaps a helper method that can be used in multiple places.
Done.
> On April 14, 2016, 4:12 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, line 154
> > <https://reviews.apache.org/r/46183/diff/1/?file=1343728#file1343728line154>
> >
> > This may change based on what gets decided about RM delete. For now, you can keep the code as is.
Sure. Will keep that in mind, based on delete decision for 'llap' queue.
> On April 14, 2016, 4:12 a.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml, line 21
> > <https://reviews.apache.org/r/46183/diff/1/?file=1343727#file1343727line21>
> >
> > Anyway to avoid it? This is not a valid property for capacity-scheduler.
True. given that we are setting things as a bag. This is the only way I could submit capacity-scheduler configs as bag back, as part of recommendation.
- Swapan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review128828
-----------------------------------------------------------
On April 14, 2016, 3:06 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 3:06 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> - When Hive Server Interactive is turned ON, Ambari will create 'llap' named YARN queue if only 'default' queue exists at root level and is consuming 100% capacity.
> - When Hive Server Interactive is turned OFF, Ambari will delete 'llap' queue only if 2 queues exist at root level namely, 'llap' and 'default'.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and deletion logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 46183: AMBARI-15876. Stack advisor work for adding
and
removing LLAP queue when Interactive query is turned on and off respectively.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46183/#review128828
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml (line 21)
<https://reviews.apache.org/r/46183/#comment192284>
Anyway to avoid it? This is not a valid property for capacity-scheduler.
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 124)
<https://reviews.apache.org/r/46183/#comment192280>
Lets protect the whole logic based on if HIVE server exists and HSI is mapped to a host. Its possible to add hive-interactive-site without adding service/component.
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 140)
<https://reviews.apache.org/r/46183/#comment192281>
Similar to above - check that service exists, component exists and then configuration exists. Perhaps a helper method that can be used in multiple places.
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 149)
<https://reviews.apache.org/r/46183/#comment192282>
This may change based on what gets decided about RM delete. For now, you can keep the code as is.
- Sumit Mohanty
On April 14, 2016, 3:06 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46183/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 3:06 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15876
> https://issues.apache.org/jira/browse/AMBARI-15876
>
>
> Repository: ambari
>
>
> Description
> -------
>
> - When Hive Server Interactive is turned ON, Ambari will create 'llap' named YARN queue if only 'default' queue exists at root level and is consuming 100% capacity.
> - When Hive Server Interactive is turned OFF, Ambari will delete 'llap' queue only if 2 queues exist at root level namely, 'llap' and 'default'.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-env.xml d518c8b
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml ba46f00
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/capacity-scheduler.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 7407da3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py c21bad7
>
> Diff: https://reviews.apache.org/r/46183/diff/
>
>
> Testing
> -------
>
> - Python UT passes.
> - Manual testing done. Various scenarios tested.
> - Test cases added to test the creation and deletion logic.
>
> - Screenshots attached.
>
>
> File Attachments
> ----------------
>
> LLAP queue created
> https://reviews.apache.org/media/uploaded/files/2016/04/14/4304fa89-fcfa-47b2-9240-d4ffa221cb76__Screen_Shot_2016-04-13_at_4.51.35_PM.png
> HSI On and llap and default queues shown in dropdown
> https://reviews.apache.org/media/uploaded/files/2016/04/14/622db04f-ec39-4a63-920d-dffc3a269917__Screen_Shot_2016-04-13_at_4.51.03_PM.png
> YARn queue manager with llap and default queues
> https://reviews.apache.org/media/uploaded/files/2016/04/14/29ee3e5c-9c49-4e73-be39-46e6ec786d21__Screen_Shot_2016-04-13_at_4.48.46_PM.png
>
>
> Thanks,
>
> Swapan Shridhar
>
>