Bucket cache configuration

[gabriela montiel <ga...@oracle.com>]

Hello,

We are working on performance tunning on HBase 0.98 CDH 5.1 using bucket 
cache. We were able to set up the bucket cache configuration on all 
storage nodes but we wanted to know further details on how the 
MaxDirectMemorySize is related to the bucket cache size and the Java 
heap size. How does the MaxDirectMemorySize affects performance in HBase?

Our current configuration is the following:

<property>
<name>hbase.bucketcache.ioengine</name>
<value>offheap</value>
<source>hbase-site.xml</source>
</property>
<property>
<name>hbase.bucketcache.percentage.in.combinedcache</name>
<value>0.8</value>
<source>hbase-site.xml</source>
</property>
<property>
<name>hbase.bucketcache.size</name>
<value>10000</value>
<source>hbase-site.xml</source>
</property>

-XX:MaxDirectMemorySize=12g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC 
-XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 
-XX:+CMSParallelRemarkEnabled
Java Heap Size: 12G


Thanks,

Gaby

Re: Bucket cache configuration

[Stack <st...@duboce.net>]

You need https://issues.apache.org/jira/browse/HBASE-11678

Just make sure the setting for MaxDirectMemorySize is > your offheap
bucketcache size and then some -- see blockcache section in refguide for
more detail (it looks like you have it set properly in the above).
Regards MaxDirectMemorySize
relationship to bucketcache size, it is a little messy.  hbase.bucketcache.
percentage.in.combinedcache has actually been removed in later versions
because it tended to confuse.  See HBASE-11520 for why and how it
works/worked.

Come back with more questions if still not clear.

St.Ack


On Fri, Aug 29, 2014 at 2:04 PM, gabriela montiel <
gabriela.montiel@oracle.com> wrote:

> Hello,
>
> We are working on performance tunning on HBase 0.98 CDH 5.1 using bucket
> cache. We were able to set up the bucket cache configuration on all storage
> nodes but we wanted to know further details on how the MaxDirectMemorySize
> is related to the bucket cache size and the Java heap size. How does the
> MaxDirectMemorySize affects performance in HBase?
>
> Our current configuration is the following:
>
> <property>
> <name>hbase.bucketcache.ioengine</name>
> <value>offheap</value>
> <source>hbase-site.xml</source>
> </property>
> <property>
> <name>hbase.bucketcache.percentage.in.combinedcache</name>
> <value>0.8</value>
> <source>hbase-site.xml</source>
> </property>
> <property>
> <name>hbase.bucketcache.size</name>
> <value>10000</value>
> <source>hbase-site.xml</source>
> </property>
>
> -XX:MaxDirectMemorySize=12g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70
> -XX:+CMSParallelRemarkEnabled
> Java Heap Size: 12G
>
>
> Thanks,
>
> Gaby
>

Re: Bucket cache configuration

[Ted Yu <yu...@gmail.com>]

Please read Procedure 9.1 under
http://hbase.apache.org/book.html#offheap.blockcache

Cheers


On Fri, Aug 29, 2014 at 2:04 PM, gabriela montiel <
gabriela.montiel@oracle.com> wrote:

> Hello,
>
> We are working on performance tunning on HBase 0.98 CDH 5.1 using bucket
> cache. We were able to set up the bucket cache configuration on all storage
> nodes but we wanted to know further details on how the MaxDirectMemorySize
> is related to the bucket cache size and the Java heap size. How does the
> MaxDirectMemorySize affects performance in HBase?
>
> Our current configuration is the following:
>
> <property>
> <name>hbase.bucketcache.ioengine</name>
> <value>offheap</value>
> <source>hbase-site.xml</source>
> </property>
> <property>
> <name>hbase.bucketcache.percentage.in.combinedcache</name>
> <value>0.8</value>
> <source>hbase-site.xml</source>
> </property>
> <property>
> <name>hbase.bucketcache.size</name>
> <value>10000</value>
> <source>hbase-site.xml</source>
> </property>
>
> -XX:MaxDirectMemorySize=12g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70
> -XX:+CMSParallelRemarkEnabled
> Java Heap Size: 12G
>
>
> Thanks,
>
> Gaby
>