You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Swapan Shridhar <ss...@hortonworks.com> on 2017/02/02 00:41:27 UTC
Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
Bugs: AMBARI-19827
https://issues.apache.org/jira/browse/AMBARI-19827
Repository: ambari
Description
-------
HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py fe9737a
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py fe58217
Diff: https://reviews.apache.org/r/56189/diff/
Testing
-------
Python UT tests fixed and Passes.
Tested on cluster.
Thanks,
Swapan Shridhar
Re: Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
> On Feb. 2, 2017, 3:19 p.m., Sumit Mohanty wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, line 931
> > <https://reviews.apache.org/r/56189/diff/1/?file=1621862#file1621862line931>
> >
> > Lets add a test that goes through this condition. This one has low chance of coverage from system tests as hosts used are fairly powerful. So we should have a test with hosts in the order of 3-4 GB ram and 1-2 cores and verify the output from stack advisor.
Done.
> On Feb. 2, 2017, 3:19 p.m., Sumit Mohanty wrote:
> > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py, line 3585
> > <https://reviews.apache.org/r/56189/diff/1/?file=1621863#file1621863line3585>
> >
> > Why did these values change - were they not being set before?
Were not getting recomemnded before. Hence default values getting set.
- Swapan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/#review163975
-----------------------------------------------------------
On Feb. 4, 2017, 1 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56189/
> -----------------------------------------------------------
>
> (Updated Feb. 4, 2017, 1 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19827
> https://issues.apache.org/jira/browse/AMBARI-19827
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 22d50e3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 9029dbb
>
> Diff: https://reviews.apache.org/r/56189/diff/
>
>
> Testing
> -------
>
> Python UT tests fixed and Passes.
> Tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/#review163975
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 930)
<https://reviews.apache.org/r/56189/#comment235480>
Lets add a test that goes through this condition. This one has low chance of coverage from system tests as hosts used are fairly powerful. So we should have a test with hosts in the order of 3-4 GB ram and 1-2 cores and verify the output from stack advisor.
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py (line 3584)
<https://reviews.apache.org/r/56189/#comment235479>
Why did these values change - were they not being set before?
- Sumit Mohanty
On Feb. 2, 2017, 12:41 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56189/
> -----------------------------------------------------------
>
> (Updated Feb. 2, 2017, 12:41 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19827
> https://issues.apache.org/jira/browse/AMBARI-19827
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py fe9737a
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py fe58217
>
> Diff: https://reviews.apache.org/r/56189/diff/
>
>
> Testing
> -------
>
> Python UT tests fixed and Passes.
> Tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/#review164811
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Feb. 8, 2017, 11:57 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56189/
> -----------------------------------------------------------
>
> (Updated Feb. 8, 2017, 11:57 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19827
> https://issues.apache.org/jira/browse/AMBARI-19827
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 22d50e3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 9029dbb
>
> Diff: https://reviews.apache.org/r/56189/diff/
>
>
> Testing
> -------
>
> Python UT tests fixed and Passes.
> Tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
> On Feb. 9, 2017, 7:22 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py
> > Lines 919 (patched)
> > <https://reviews.apache.org/r/56189/diff/3/?file=1627962#file1627962line920>
> >
> > This change is also needed for YARN's Service Advisor in 3.0.0.3.0
Done as part of "AMBARI-20281" for trunk.
- Swapan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/#review164985
-----------------------------------------------------------
On Feb. 8, 2017, 11:57 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56189/
> -----------------------------------------------------------
>
> (Updated Feb. 8, 2017, 11:57 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19827
> https://issues.apache.org/jira/browse/AMBARI-19827
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 22d50e3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 9029dbb
>
>
> Diff: https://reviews.apache.org/r/56189/diff/3/
>
>
> Testing
> -------
>
> Python UT tests fixed and Passes.
> Tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/#review164985
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 919)
<https://reviews.apache.org/r/56189/#comment236821>
This change is also needed for YARN's Service Advisor in 3.0.0.3.0
- Alejandro Fernandez
On Feb. 8, 2017, 11:57 p.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56189/
> -----------------------------------------------------------
>
> (Updated Feb. 8, 2017, 11:57 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19827
> https://issues.apache.org/jira/browse/AMBARI-19827
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 22d50e3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 9029dbb
>
> Diff: https://reviews.apache.org/r/56189/diff/
>
>
> Testing
> -------
>
> Python UT tests fixed and Passes.
> Tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/
-----------------------------------------------------------
(Updated Feb. 8, 2017, 11:57 p.m.)
Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
Changes
-------
Fixed the following issue:
=============================================================================
Error occured in stack advisor.
Error details: invalid literal for float(): SET_ON_FIRST_INVOCATION
08 Feb 2017 02:24:19,299 INFO [pool-18-thread-1] StackAdvisorRunner:71 - advisor script stderr: Traceback (most recent call last):
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 166, in <module>
main(sys.argv)
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 116, in main
result = stackAdvisor.recommendConfigurations(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 775, in recommendConfigurations
calculation(configurations, clusterSummary, services, hosts)
File "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py", line 687, in recommendHIVEConfigurations
super(HDP25StackAdvisor, self).recommendHIVEConfigurations(configurations, clusterData, services, hosts)
File "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.3/services/stack_advisor.py", line 222, in recommendHIVEConfigurations
super(HDP23StackAdvisor, self).recommendHIVEConfigurations(configurations, clusterData, services, hosts)
File "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.2/services/stack_advisor.py", line 375, in recommendHIVEConfigurations
self.recommendYARNConfigurations(configurations, clusterData, services, hosts)
File "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py", line 734, in recommendYARNConfigurations
self.updateLlapConfigs(configurations, services, hosts, LLAP_QUEUE_NAME)
File "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py", line 910, in updateLlapConfigs
mem_per_thread_for_llap = self.calculate_mem_per_thread_for_llap(services, yarn_nm_mem_in_mb_normalized, cpu_per_nm_host)
File "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py", line 1320, in calculate_mem_per_thread_for_llap
hive_tez_container_size = self.get_hive_tez_container_size(services)
File "/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.5/services/stack_advisor.py", line 1347, in get_hive_tez_container_size
hive_container_size = float(hsi_site['hive.tez.container.size'])
ValueError: invalid literal for float(): SET_ON_FIRST_INVOCATION
=============================================================================
Bugs: AMBARI-19827
https://issues.apache.org/jira/browse/AMBARI-19827
Repository: ambari
Description
-------
HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 22d50e3
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 9029dbb
Diff: https://reviews.apache.org/r/56189/diff/
Testing
-------
Python UT tests fixed and Passes.
Tested on cluster.
Thanks,
Swapan Shridhar
Re: Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/#review164214
-----------------------------------------------------------
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py (line 1988)
<https://reviews.apache.org/r/56189/#comment235867>
Added Test case for specifically testing small cluster, representing a small GCE cluster.
- 3 nodes having 'yarn.nodemanager.resource.memory-mb' : 2046 and 'yarn.scheduler.minimum-allocation-mb' : 682
- Swapan Shridhar
On Feb. 4, 2017, 1 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56189/
> -----------------------------------------------------------
>
> (Updated Feb. 4, 2017, 1 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19827
> https://issues.apache.org/jira/browse/AMBARI-19827
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 22d50e3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 9029dbb
>
> Diff: https://reviews.apache.org/r/56189/diff/
>
>
> Testing
> -------
>
> Python UT tests fixed and Passes.
> Tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/#review164563
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Feb. 4, 2017, 1 a.m., Swapan Shridhar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56189/
> -----------------------------------------------------------
>
> (Updated Feb. 4, 2017, 1 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
>
>
> Bugs: AMBARI-19827
> https://issues.apache.org/jira/browse/AMBARI-19827
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 22d50e3
> ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 9029dbb
>
> Diff: https://reviews.apache.org/r/56189/diff/
>
>
> Testing
> -------
>
> Python UT tests fixed and Passes.
> Tested on cluster.
>
>
> Thanks,
>
> Swapan Shridhar
>
>
Re: Review Request 56189: AMBARI-19827. HiveServer2 Interactive won't
start in clusters with less memory.
Posted by Swapan Shridhar <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56189/
-----------------------------------------------------------
(Updated Feb. 4, 2017, 1 a.m.)
Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
Changes
-------
Added Test case for specifically testing small cluster, representing a small GCE cluster.
- 3 nodes having 'yarn.nodemanager.resource.memory-mb' : 2046 and 'yarn.scheduler.minimum-allocation-mb' : 682
Bugs: AMBARI-19827
https://issues.apache.org/jira/browse/AMBARI-19827
Repository: ambari
Description
-------
HSI caluclations fails on cluster with small memory. Fixed the calulations for that.
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 22d50e3
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 9029dbb
Diff: https://reviews.apache.org/r/56189/diff/
Testing
-------
Python UT tests fixed and Passes.
Tested on cluster.
Thanks,
Swapan Shridhar