HBase Region Server Memory Configuration

[Amit Shah <am...@gmail.com>]

Hello,

I am trying to understand hbase's memory configuration. I am using hbase
version 1.0 through cloudera (version 5.5.2). I need to benchmark query
performance (through phoenix 4.6) on hbase with and without bucket cache.

While reading about hbase memory configuration I have following doubts

1. I have configured java heap size on the region server as 3.5 GB on the
cloudera manager console though the region server web ui (server metrics ->
Memory tab) shows the max heap to be 1.7. hbase.block.cache.size is
configured to be 0.6, hbase.regionserver.global.memstore.lowerLimit is
configured to be 0.1 while hbase.regionserver.global.memstore.upperLimit is
configured to be 0.2 so that the blockcache + memstore combination does not
exceed 0.8.

What could be the reason behind the difference shown on the region server
web ui?

2. While configuring bucket cache, there are mainly two configs to set the
size of the bucket cache -XX:MaxDirectMemorySize
and hbase.bucketcache.size. It is recommended that XX:MaxDirectMemorySize
 should be of a higher value than hbase.bucketcache.size. What are the
hbase operations that use the offheap memory apart from bucket cache?

3. Is it worth benchmarking bucketcache in isolation i.e. without
LruBlockCache - hbase.block.cache.size set to 0?

Appreciate your inputs.

Thanks,
Amit

Re: HBase Region Server Memory Configuration

[Amit Shah <am...@gmail.com>]

Thanks Anoop for the quick replies.

 U may need to check the logs in CDH manager and the RS to see whether it
> is correctly set
>

I check the hbase logs but I couldn't find much. I could see some mentions
about the bucket cache but nothing about the heap config. I have pasted
some of my findings here <http://pastebin.com/RZqN5J3X>. Probably from your
response it looks like an issue with cloudera. I have posted a question on
their forum. I hope to get some insights.

In 1.0 version  there wont be any direct off heap space usage from HBase
> code other than BC code.  (Later versions have it).


Do you see any code changes between version 1.0 and the current stable
version (1.1.4) in terms of bucket cache? Would the benchmarks hold true
for hbase 1.1.4?

And you are testing for range scan or single row fetch queries?


I need to do tests with 3 types of queries - where clause, group by and
window function. So yes it would include range scan (the where clause).

A few more queries
1. I'm trying to monitor the bucket cache (when configured as
"CombinedBlockCache - offheap". On the region server UI I could see
buckets listed
<http://i.imgur.com/iNPd58Q.png>while the statistics of L2 cache are also
shown <http://i.imgur.com/3bbqnKq.png>. What does "Count of blocks"
indicate? Does it mean only those many blocks are loaded into the cache?
How does that relate to BucketCache Buckets?

2. Would it be worth to benchmark SlabCache (individually and/or with block
cache as "DoubleBlockCache")? From one of the readings
<http://www.n10k.com/blog/blockcache-showdown/>I realized the results were
not that great. Is it still an actively maintained implementation of block
cache?

I will follow up with my benchmark findings.

Thanks,
Amit.

On Wed, Apr 6, 2016 at 4:05 PM, Anoop John <an...@gmail.com> wrote:

> And you are testing for range scan or single row fetch queries?  As
> said in 1.0 the offheap L2 is not same performing as L1.  Optimization
> in trunk make it close and almost equal to L1. (Mainly when it is
> random single row reads).    If so u can even benchmark using the
> trunk also may be.  JFYI.
>
> Feel free to ask any Qs in the group or if not getting response send
> me directly also np.  Some times the group mails are unseen in midst
> of so many mails to group :-)
>
> -Anoop-
>
>
> ---------- Forwarded message ----------
> From: Anoop John <an...@gmail.com>
> Date: Wed, Apr 6, 2016 at 4:01 PM
> Subject: Re: HBase Region Server Memory Configuration
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
>
>
> So u have given the max heap size for the RS process via Cloudera
> manager and u fear that that much is not taken by the process.  U may
> need to check the logs in CDH manager and the RS to see whether it is
> correctly set
>
> So u r using bucket cache(BC) in offheap mode right.  In 1.0 version
> there wont be any direct off heap space usage from HBase code other
> than BC code.  (Later versions have it).  But the SocketChannel uses
> off heap space to move the data to network.  When written bytes to
> socket channel, it first copy it to off heap bytebuffers and then the
> OS level calls to do the network transfers.  So ya the process as such
> would need more offheap memory than the size of the bucket cache.
>
> At least in 1.0 the perf of off heap Bucket cache is not same as that
> of L1 cache.  When both are in place, by default, we will keep the
> index blocks etc (non data) in L1 and data blocks in L2.  That would
> be an ideal choice IMO.  The separation allows the L1 blocks to stay
> there with out getting possibly evicted. These are meta information
> for the data blocks.   The data blocks can stay in L2 cache.
>
> -Anoop-
>
>
> On Wed, Apr 6, 2016 at 3:24 PM, Amit Shah <am...@gmail.com> wrote:
> > Hello,
> >
> > I am trying to understand hbase's memory configuration. I am using hbase
> > version 1.0 through cloudera (version 5.5.2). I need to benchmark query
> > performance (through phoenix 4.6) on hbase with and without bucket cache.
> >
> > While reading about hbase memory configuration I have following doubts
> >
> > 1. I have configured java heap size on the region server as 3.5 GB on the
> > cloudera manager console though the region server web ui (server metrics
> ->
> > Memory tab) shows the max heap to be 1.7. hbase.block.cache.size is
> > configured to be 0.6, hbase.regionserver.global.memstore.lowerLimit is
> > configured to be 0.1 while hbase.regionserver.global.memstore.upperLimit
> is
> > configured to be 0.2 so that the blockcache + memstore combination does
> not
> > exceed 0.8.
> >
> > What could be the reason behind the difference shown on the region server
> > web ui?
> >
> > 2. While configuring bucket cache, there are mainly two configs to set
> the
> > size of the bucket cache -XX:MaxDirectMemorySize
> > and hbase.bucketcache.size. It is recommended that XX:MaxDirectMemorySize
> >  should be of a higher value than hbase.bucketcache.size. What are the
> > hbase operations that use the offheap memory apart from bucket cache?
> >
> > 3. Is it worth benchmarking bucketcache in isolation i.e. without
> > LruBlockCache - hbase.block.cache.size set to 0?
> >
> > Appreciate your inputs.
> >
> > Thanks,
> > Amit
>

Re: HBase Region Server Memory Configuration

[Anoop John <an...@gmail.com>]

So u have given the max heap size for the RS process via Cloudera
manager and u fear that that much is not taken by the process.  U may
need to check the logs in CDH manager and the RS to see whether it is
correctly set

So u r using bucket cache(BC) in offheap mode right.  In 1.0 version
there wont be any direct off heap space usage from HBase code other
than BC code.  (Later versions have it).  But the SocketChannel uses
off heap space to move the data to network.  When written bytes to
socket channel, it first copy it to off heap bytebuffers and then the
OS level calls to do the network transfers.  So ya the process as such
would need more offheap memory than the size of the bucket cache.

At least in 1.0 the perf of off heap Bucket cache is not same as that
of L1 cache.  When both are in place, by default, we will keep the
index blocks etc (non data) in L1 and data blocks in L2.  That would
be an ideal choice IMO.  The separation allows the L1 blocks to stay
there with out getting possibly evicted. These are meta information
for the data blocks.   The data blocks can stay in L2 cache.

-Anoop-


On Wed, Apr 6, 2016 at 3:24 PM, Amit Shah <am...@gmail.com> wrote:
> Hello,
>
> I am trying to understand hbase's memory configuration. I am using hbase
> version 1.0 through cloudera (version 5.5.2). I need to benchmark query
> performance (through phoenix 4.6) on hbase with and without bucket cache.
>
> While reading about hbase memory configuration I have following doubts
>
> 1. I have configured java heap size on the region server as 3.5 GB on the
> cloudera manager console though the region server web ui (server metrics ->
> Memory tab) shows the max heap to be 1.7. hbase.block.cache.size is
> configured to be 0.6, hbase.regionserver.global.memstore.lowerLimit is
> configured to be 0.1 while hbase.regionserver.global.memstore.upperLimit is
> configured to be 0.2 so that the blockcache + memstore combination does not
> exceed 0.8.
>
> What could be the reason behind the difference shown on the region server
> web ui?
>
> 2. While configuring bucket cache, there are mainly two configs to set the
> size of the bucket cache -XX:MaxDirectMemorySize
> and hbase.bucketcache.size. It is recommended that XX:MaxDirectMemorySize
>  should be of a higher value than hbase.bucketcache.size. What are the
> hbase operations that use the offheap memory apart from bucket cache?
>
> 3. Is it worth benchmarking bucketcache in isolation i.e. without
> LruBlockCache - hbase.block.cache.size set to 0?
>
> Appreciate your inputs.
>
> Thanks,
> Amit