You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Dmytro Sen <ds...@hortonworks.com> on 2015/01/28 18:45:47 UTC

Review Request 30378: Configs: Ambari support for HBase bucketcache

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30378/
-----------------------------------------------------------

Review request for Ambari, Mahadev Konar, Nate Cole, nick dimiduk, and Tom Beerbower.


Bugs: AMBARI-9383
    https://issues.apache.org/jira/browse/AMBARI-9383


Repository: ambari


Description
-------

Off-heap caches in HBase have been implemented. Ambari should expose the necessary configurations for a user to enable this feature. We're interested only in the BucketCache off-heap mode.


Diffs
-----

  ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params.py 74cc0a4 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml f9c5885 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml e5b893f 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 3d0ace2 
  ambari-web/app/data/HDP2.2/site_properties.js efc6cb9 

Diff: https://reviews.apache.org/r/30378/diff/


Testing
-------

All tests passed


Thanks,

Dmytro Sen

Re: Review Request 30378: Configs: Ambari support for HBase bucketcache

Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30378/#review74963
-----------------------------------------------------------


What is the status of this issue?  If it's pushed, please close it.

- Nate Cole


On Feb. 9, 2015, 5:59 a.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30378/
> -----------------------------------------------------------
> 
> (Updated Feb. 9, 2015, 5:59 a.m.)
> 
> 
> Review request for Ambari, Mahadev Konar, Nate Cole, nick dimiduk, and Tom Beerbower.
> 
> 
> Bugs: AMBARI-9383
>     https://issues.apache.org/jira/browse/AMBARI-9383
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Off-heap caches in HBase have been implemented. Ambari should expose the necessary configurations for a user to enable this feature. We're interested only in the BucketCache off-heap mode.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params.py a50b985 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml f9c5885 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml e5b893f 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 6a5aab5 
>   ambari-web/app/data/HDP2.2/site_properties.js 2e8f27a 
> 
> Diff: https://reviews.apache.org/r/30378/diff/
> 
> 
> Testing
> -------
> 
> All tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 30378: Configs: Ambari support for HBase bucketcache

Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30378/#review71816
-----------------------------------------------------------

Ship it!


Ship It!

- Nate Cole


On Feb. 9, 2015, 5:59 a.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30378/
> -----------------------------------------------------------
> 
> (Updated Feb. 9, 2015, 5:59 a.m.)
> 
> 
> Review request for Ambari, Mahadev Konar, Nate Cole, nick dimiduk, and Tom Beerbower.
> 
> 
> Bugs: AMBARI-9383
>     https://issues.apache.org/jira/browse/AMBARI-9383
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Off-heap caches in HBase have been implemented. Ambari should expose the necessary configurations for a user to enable this feature. We're interested only in the BucketCache off-heap mode.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params.py a50b985 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml f9c5885 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml e5b893f 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 6a5aab5 
>   ambari-web/app/data/HDP2.2/site_properties.js 2e8f27a 
> 
> Diff: https://reviews.apache.org/r/30378/diff/
> 
> 
> Testing
> -------
> 
> All tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 30378: Configs: Ambari support for HBase bucketcache

Posted by Tom Beerbower <tb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30378/#review71830
-----------------------------------------------------------

Ship it!


Ship It!

- Tom Beerbower


On Feb. 9, 2015, 10:59 a.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30378/
> -----------------------------------------------------------
> 
> (Updated Feb. 9, 2015, 10:59 a.m.)
> 
> 
> Review request for Ambari, Mahadev Konar, Nate Cole, nick dimiduk, and Tom Beerbower.
> 
> 
> Bugs: AMBARI-9383
>     https://issues.apache.org/jira/browse/AMBARI-9383
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Off-heap caches in HBase have been implemented. Ambari should expose the necessary configurations for a user to enable this feature. We're interested only in the BucketCache off-heap mode.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params.py a50b985 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml f9c5885 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml e5b893f 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 6a5aab5 
>   ambari-web/app/data/HDP2.2/site_properties.js 2e8f27a 
> 
> Diff: https://reviews.apache.org/r/30378/diff/
> 
> 
> Testing
> -------
> 
> All tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 30378: Configs: Ambari support for HBase bucketcache

Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30378/
-----------------------------------------------------------

(Updated Фев. 9, 2015, 10:59 д.п.)


Review request for Ambari, Mahadev Konar, Nate Cole, nick dimiduk, and Tom Beerbower.


Bugs: AMBARI-9383
    https://issues.apache.org/jira/browse/AMBARI-9383


Repository: ambari


Description
-------

Off-heap caches in HBase have been implemented. Ambari should expose the necessary configurations for a user to enable this feature. We're interested only in the BucketCache off-heap mode.


Diffs (updated)
-----

  ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params.py a50b985 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml f9c5885 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml e5b893f 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 6a5aab5 
  ambari-web/app/data/HDP2.2/site_properties.js 2e8f27a 

Diff: https://reviews.apache.org/r/30378/diff/


Testing
-------

All tests passed


Thanks,

Dmytro Sen

Re: Review Request 30378: Configs: Ambari support for HBase bucketcache

Posted by Dmytro Sen <ds...@hortonworks.com>.

> On Янв. 30, 2015, 6:39 п.п., nick dimiduk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py, line 60
> > <https://reviews.apache.org/r/30378/diff/1/?file=839012#file839012line60>
> >
> >     Not sure what this has to do with hbase configs.

This is not related to hbase. It was a hive fix near. Moved out from the patch.


> On Янв. 30, 2015, 6:39 п.п., nick dimiduk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py, line 88
> > <https://reviews.apache.org/r/30378/diff/1/?file=839012#file839012line88>
> >
> >     Is 23 a calculated value? better to show as basevalue + modifier.
> >     
> >     Where does this value come from?

23 = 20 + 2 + 1; 20 - regionserver_heap_size (-Xmx); 2 - reserved off-heap (DFSClient, etc); 1 - actual bucket cache.
If we set here 22, we will have 0 for bucket cache in case of 22GB RAM. If we set less than 22 (20), the amount of off-heap memory for BucketCache can be negative.
	bucketcache_offheap_memory = regionserver_total_ram - regionserver_heap_size - reserved_offheap_memory = X - 20 - 2.
Please suggest a number.


> On Янв. 30, 2015, 6:39 п.п., nick dimiduk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py, line 107
> > <https://reviews.apache.org/r/30378/diff/1/?file=839012#file839012line107>
> >
> >     where does the gloabal memstore size value come from?

We are not creating a wizard; we want to recommend default settings to work from scratch. So 0.4 is a default value recommended by Ambari. User can change it if he wants.


> On Янв. 30, 2015, 6:39 п.п., nick dimiduk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py, line 536
> > <https://reviews.apache.org/r/30378/diff/1/?file=839012#file839012line536>
> >
> >     pct.in.combinedcache ships with a default of 0.9.

In the case of disabled bucketcache, we set it to '' (putHbaseProperty('hbase.bucketcache.percentage.in.combinedcache', '')). Do we need to leave a default value?


> On Янв. 30, 2015, 6:39 п.п., nick dimiduk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py, line 551
> > <https://reviews.apache.org/r/30378/diff/1/?file=839012#file839012line551>
> >
> >     should be `if hbase_site_properties[prop_name1] == "offheap"`

Changed in the patch.


- Dmytro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30378/#review70386
-----------------------------------------------------------


On Фев. 9, 2015, 10:59 д.п., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30378/
> -----------------------------------------------------------
> 
> (Updated Фев. 9, 2015, 10:59 д.п.)
> 
> 
> Review request for Ambari, Mahadev Konar, Nate Cole, nick dimiduk, and Tom Beerbower.
> 
> 
> Bugs: AMBARI-9383
>     https://issues.apache.org/jira/browse/AMBARI-9383
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Off-heap caches in HBase have been implemented. Ambari should expose the necessary configurations for a user to enable this feature. We're interested only in the BucketCache off-heap mode.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params.py a50b985 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml f9c5885 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml e5b893f 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 6a5aab5 
>   ambari-web/app/data/HDP2.2/site_properties.js 2e8f27a 
> 
> Diff: https://reviews.apache.org/r/30378/diff/
> 
> 
> Testing
> -------
> 
> All tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 30378: Configs: Ambari support for HBase bucketcache

Posted by nick dimiduk <nd...@gmail.com>.

> On Jan. 30, 2015, 10:39 a.m., nick dimiduk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py, line 536
> > <https://reviews.apache.org/r/30378/diff/1/?file=839012#file839012line536>
> >
> >     pct.in.combinedcache ships with a default of 0.9.
> 
> Dmytro Sen wrote:
>     In the case of disabled bucketcache, we set it to '' (putHbaseProperty('hbase.bucketcache.percentage.in.combinedcache', '')). Do we need to leave a default value?

The value will not be considered when bucketcache is disabled. It shouldn't matter what value is set (or any)  in this case.


> On Jan. 30, 2015, 10:39 a.m., nick dimiduk wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py, line 88
> > <https://reviews.apache.org/r/30378/diff/1/?file=839012#file839012line88>
> >
> >     Is 23 a calculated value? better to show as basevalue + modifier.
> >     
> >     Where does this value come from?
> 
> Dmytro Sen wrote:
>     23 = 20 + 2 + 1; 20 - regionserver_heap_size (-Xmx); 2 - reserved off-heap (DFSClient, etc); 1 - actual bucket cache.
>     If we set here 22, we will have 0 for bucket cache in case of 22GB RAM. If we set less than 22 (20), the amount of off-heap memory for BucketCache can be negative.
>     	bucketcache_offheap_memory = regionserver_total_ram - regionserver_heap_size - reserved_offheap_memory = X - 20 - 2.
>     Please suggest a number.

Okay, makes sense. This is find for a first pass. In the future, probably 1g offheap isn't worth it for bucketcache. Would be better to have a minimum offheap size of 4G or maybe 6.


- nick


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30378/#review70386
-----------------------------------------------------------


On Feb. 9, 2015, 2:59 a.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30378/
> -----------------------------------------------------------
> 
> (Updated Feb. 9, 2015, 2:59 a.m.)
> 
> 
> Review request for Ambari, Mahadev Konar, Nate Cole, nick dimiduk, and Tom Beerbower.
> 
> 
> Bugs: AMBARI-9383
>     https://issues.apache.org/jira/browse/AMBARI-9383
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Off-heap caches in HBase have been implemented. Ambari should expose the necessary configurations for a user to enable this feature. We're interested only in the BucketCache off-heap mode.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params.py a50b985 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml f9c5885 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml e5b893f 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 6a5aab5 
>   ambari-web/app/data/HDP2.2/site_properties.js 2e8f27a 
> 
> Diff: https://reviews.apache.org/r/30378/diff/
> 
> 
> Testing
> -------
> 
> All tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 30378: Configs: Ambari support for HBase bucketcache

Posted by nick dimiduk <nd...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30378/#review70386
-----------------------------------------------------------



ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py
<https://reviews.apache.org/r/30378/#comment115532>

    Not sure what this has to do with hbase configs.



ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py
<https://reviews.apache.org/r/30378/#comment115536>

    Is 23 a calculated value? better to show as basevalue + modifier.
    
    Where does this value come from?



ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py
<https://reviews.apache.org/r/30378/#comment115542>

    where does the gloabal memstore size value come from?



ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py
<https://reviews.apache.org/r/30378/#comment115546>

    pct.in.combinedcache ships with a default of 0.9.



ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py
<https://reviews.apache.org/r/30378/#comment115549>

    should be `if hbase_site_properties[prop_name1] == "offheap"`


- nick dimiduk


On Jan. 28, 2015, 9:45 a.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30378/
> -----------------------------------------------------------
> 
> (Updated Jan. 28, 2015, 9:45 a.m.)
> 
> 
> Review request for Ambari, Mahadev Konar, Nate Cole, nick dimiduk, and Tom Beerbower.
> 
> 
> Bugs: AMBARI-9383
>     https://issues.apache.org/jira/browse/AMBARI-9383
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Off-heap caches in HBase have been implemented. Ambari should expose the necessary configurations for a user to enable this feature. We're interested only in the BucketCache off-heap mode.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params.py 74cc0a4 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-env.xml f9c5885 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/configuration/hbase-site.xml e5b893f 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 3d0ace2 
>   ambari-web/app/data/HDP2.2/site_properties.js efc6cb9 
> 
> Diff: https://reviews.apache.org/r/30378/diff/
> 
> 
> Testing
> -------
> 
> All tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>