You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dejan Menges <de...@gmail.com> on 2015/06/01 11:24:21 UTC

Monitor off heap Bucket Cache

Hi,

What's the best way to monitor / know how's bucket cache being used, how
much stuff is cached there, etc?

Our RegionServer can use 32G of heap size, so we exported HBASE_OFFHEAPSIZE
to 24G in hbase-env.sh, set hfile.block.cache.size to 0.05, and set couple
of block sizes that we know we are using knowing our usage patterns. And
this is where strange part starts - in web UI we see now, with turning this
off, with those values, that total BlockCache available is 1G - before it
was 10G. What we basically tried to achieve was to double it to 20G.

Documentation we were referring to was
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/admin_hbase_blockcache_configure.html#concept_cp3_fhy_dr_unique_1__section_m3r_2cz_dr_unique_1
as HBase book is not going into too much details how to properly configure
this and get what you want.

Btw. if we put hfile.block.cache.size to 0.2 we see in web UI that total
available cache is 24G, but then after some time we had region server
crashing. Host server have enough RAM, considering only data node and
region server running there (128G of RAM in total) so we thought we could
increase caching with turning on this functionality.

Do you maybe see what exactly we are doing wrong? How to exactly increase
offheap caching - and is it possible to monitor it anyhow, as in metrics we
don't see anything associating to it?

Thanks,
Dejan

Re: Monitor off heap Bucket Cache

Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Dejan,

On Mon, Jun 1, 2015 at 5:24 AM, Dejan Menges <de...@gmail.com> wrote:

> Hi,
>
> What's the best way to monitor / know how's bucket cache being used, how
> much stuff is cached there, etc?
>

Have a look at these:
https://apps.sematext.com/spm-reports/s/ivzcFizKhG
https://apps.sematext.com/spm-reports/s/UqSDOkO6AX

If this is what you are after, see http://sematext.com/spm

Note the above charts / metrics are for 0.98 HBase.  Not sure if these
metrics changed in 1.x.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/



> Our RegionServer can use 32G of heap size, so we exported HBASE_OFFHEAPSIZE
> to 24G in hbase-env.sh, set hfile.block.cache.size to 0.05, and set couple
> of block sizes that we know we are using knowing our usage patterns. And
> this is where strange part starts - in web UI we see now, with turning this
> off, with those values, that total BlockCache available is 1G - before it
> was 10G. What we basically tried to achieve was to double it to 20G.
>
> Documentation we were referring to was
>
> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/admin_hbase_blockcache_configure.html#concept_cp3_fhy_dr_unique_1__section_m3r_2cz_dr_unique_1
> as HBase book is not going into too much details how to properly configure
> this and get what you want.
>
> Btw. if we put hfile.block.cache.size to 0.2 we see in web UI that total
> available cache is 24G, but then after some time we had region server
> crashing. Host server have enough RAM, considering only data node and
> region server running there (128G of RAM in total) so we thought we could
> increase caching with turning on this functionality.
>
> Do you maybe see what exactly we are doing wrong? How to exactly increase
> offheap caching - and is it possible to monitor it anyhow, as in metrics we
> don't see anything associating to it?
>
> Thanks,
> Dejan
>

Re: Monitor off heap Bucket Cache

Posted by Nick Dimiduk <nd...@gmail.com>.
Also note that configuration is slightly changed between 0.98 and 1.0,
see HBASE-11520. From the release note:

> Remove "hbase.bucketcache.percentage.in.combinedcache". Simplifies config
of block cache. If you are using this config., after
> this patch goes in, it will be ignored. The L1 LruBlockCache will be
whatever hfile.block.cache.size is set to and the L2
> BucketCache will be whatever hbase.bucketcache.size is set to.

On Mon, Jun 1, 2015 at 8:10 AM, Stack <st...@duboce.net> wrote:

> On Mon, Jun 1, 2015 at 2:24 AM, Dejan Menges <de...@gmail.com>
> wrote:
>
> > Hi,
> >
> > What's the best way to monitor / know how's bucket cache being used, how
> > much stuff is cached there, etc?
> >
> >
> See the UI on a regionserver. Look down the page to the 'Block Cache'
> section. It has detail on both onheap and LRU offheap. See also the
> documentation on offheap cache:
> http://hbase.apache.org/book.html#offheap.blockcache Look also at metrics
> where we report block cache stats as well as offheap used by the JVM.
>
>
>
> > Our RegionServer can use 32G of heap size, so we exported
> HBASE_OFFHEAPSIZE
> > to 24G in hbase-env.sh, set hfile.block.cache.size to 0.05, and set
> couple
> > of block sizes that we know we are using knowing our usage patterns. And
> > this is where strange part starts - in web UI we see now, with turning
> this
> > off,
>
>
> Turning off what?
>
>
>
> > with those values, that total BlockCache available is 1G - before it
> > was 10G. What we basically tried to achieve was to double it to 20G.
> >
> >
> Are you sure this not the onheap portion of BlockCache?
>
>
>
> > Documentation we were referring to was
> >
> >
> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/admin_hbase_blockcache_configure.html#concept_cp3_fhy_dr_unique_1__section_m3r_2cz_dr_unique_1
> > as HBase book is not going into too much details how to properly
> configure
> > this and get what you want.
> >
> >
> Please let us know what is missing from here:
> http://hbase.apache.org/book.html#offheap.blockcache  We'd like to fix it.
>
>
>
> > Btw. if we put hfile.block.cache.size to 0.2 we see in web UI that total
> > available cache is 24G, but then after some time we had region server
> > crashing. Host server have enough RAM, considering only data node and
> > region server running there (128G of RAM in total) so we thought we could
> > increase caching with turning on this functionality.
> >
> >
> Make sure your hbase has HBASE-11678
>
>
>
> > Do you maybe see what exactly we are doing wrong? How to exactly increase
> > offheap caching - and is it possible to monitor it anyhow, as in metrics
> we
> > don't see anything associating to it?
> >
> >
> Please provide more detail. Your configurations and what you've changed in
> hbase-env.sh.
>
> St.Ack
>
>
>
> > Thanks,
> > Dejan
> >
>

Re: Monitor off heap Bucket Cache

Posted by Stack <st...@duboce.net>.
On Mon, Jun 1, 2015 at 2:24 AM, Dejan Menges <de...@gmail.com> wrote:

> Hi,
>
> What's the best way to monitor / know how's bucket cache being used, how
> much stuff is cached there, etc?
>
>
See the UI on a regionserver. Look down the page to the 'Block Cache'
section. It has detail on both onheap and LRU offheap. See also the
documentation on offheap cache:
http://hbase.apache.org/book.html#offheap.blockcache Look also at metrics
where we report block cache stats as well as offheap used by the JVM.



> Our RegionServer can use 32G of heap size, so we exported HBASE_OFFHEAPSIZE
> to 24G in hbase-env.sh, set hfile.block.cache.size to 0.05, and set couple
> of block sizes that we know we are using knowing our usage patterns. And
> this is where strange part starts - in web UI we see now, with turning this
> off,


Turning off what?



> with those values, that total BlockCache available is 1G - before it
> was 10G. What we basically tried to achieve was to double it to 20G.
>
>
Are you sure this not the onheap portion of BlockCache?



> Documentation we were referring to was
>
> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/admin_hbase_blockcache_configure.html#concept_cp3_fhy_dr_unique_1__section_m3r_2cz_dr_unique_1
> as HBase book is not going into too much details how to properly configure
> this and get what you want.
>
>
Please let us know what is missing from here:
http://hbase.apache.org/book.html#offheap.blockcache  We'd like to fix it.



> Btw. if we put hfile.block.cache.size to 0.2 we see in web UI that total
> available cache is 24G, but then after some time we had region server
> crashing. Host server have enough RAM, considering only data node and
> region server running there (128G of RAM in total) so we thought we could
> increase caching with turning on this functionality.
>
>
Make sure your hbase has HBASE-11678



> Do you maybe see what exactly we are doing wrong? How to exactly increase
> offheap caching - and is it possible to monitor it anyhow, as in metrics we
> don't see anything associating to it?
>
>
Please provide more detail. Your configurations and what you've changed in
hbase-env.sh.

St.Ack



> Thanks,
> Dejan
>