Block cache hit ratio question

[Vaibhav Puranik <vp...@gmail.com>]

Hi all,

Our HBase is slowing down because of lots of disk reads. We are trying to
analyze it. We have lots of real time random reads. And We found that HBase
is constantly reading from disk instead of from memory.

Looks like block cache hit ratio is a good place to start.

Is there any way we can get block cache hit ratio by table or by region? I
could only find it per region server in region server UI. I couldn't find
it in API either.

Feel free to suggest any other metrics we should be looking at too.

Thanks,
Vaibhav
GumGum

Re: Block cache hit ratio question

[Otis Gospodnetic <ot...@gmail.com>]

Hi,

Our SPM for HBase doesn't show per table or region cache stats, and looks
like that is not available yet, but it will let you slice by host and see
changes over time, e.g., as you tune. See http://sematext.com/spm . We use
it for monitoring our own clusters.

Otis
Solr & ElasticSearch Support
http://sematext.com/
On Oct 18, 2013 6:20 PM, "Vaibhav Puranik" <vp...@gmail.com> wrote:

> Hi all,
>
> Our HBase is slowing down because of lots of disk reads. We are trying to
> analyze it. We have lots of real time random reads. And We found that HBase
> is constantly reading from disk instead of from memory.
>
> Looks like block cache hit ratio is a good place to start.
>
> Is there any way we can get block cache hit ratio by table or by region? I
> could only find it per region server in region server UI. I couldn't find
> it in API either.
>
> Feel free to suggest any other metrics we should be looking at too.
>
> Thanks,
> Vaibhav
> GumGum
>

Re: Block cache hit ratio question

[Ted Yu <yu...@gmail.com>]

This is what I found
in src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java
(assuming you're using 0.94.x)

  public final MetricsLongValue blockCacheHitCount =

      new MetricsLongValue("blockCacheHitCount", registry);

  public final MetricsLongValue blockCacheMissCount =

      new MetricsLongValue("blockCacheMissCount", registry);

  public final MetricsLongValue blockCacheEvictedCount =

      new MetricsLongValue("blockCacheEvictedCount", registry);

  public final MetricsIntValue blockCacheHitRatio =

      new MetricsIntValue("blockCacheHitRatio", registry);

Cheers


On Fri, Oct 18, 2013 at 3:20 PM, Vaibhav Puranik <vp...@gmail.com> wrote:

> Hi all,
>
> Our HBase is slowing down because of lots of disk reads. We are trying to
> analyze it. We have lots of real time random reads. And We found that HBase
> is constantly reading from disk instead of from memory.
>
> Looks like block cache hit ratio is a good place to start.
>
> Is there any way we can get block cache hit ratio by table or by region? I
> could only find it per region server in region server UI. I couldn't find
> it in API either.
>
> Feel free to suggest any other metrics we should be looking at too.
>
> Thanks,
> Vaibhav
> GumGum
>

RE: Block cache hit ratio question

[Vladimir Rodionov <vr...@carrieriq.com>]

HBase block cache does not expose any statistics on a per-table basis. This looks like a nice to have feature in
a next releases. If your load is mostly random reads, than several factors affect the overall performance:

1. The avg read size / block size ratio. The large ratio - the better block cache hit ration you will have.
2. Hot data set size.
3 Available RAM for block cache

For example,

Block size = 64K
Avg read size = 1K
Hot data set size = 1GB

You will need 64GB of block cache to cache 100% your random reads.
Usually, for random reads the smaller block size - the better, but you can not decrease block size w/o additional overhead of bookkeeping
larger number of smaller blocks. Your actual limit is around 16KB.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: Vaibhav Puranik [vpuranik@gmail.com]
Sent: Friday, October 18, 2013 3:20 PM
To: user@hbase.apache.org
Subject: Block cache hit ratio question

Hi all,

Our HBase is slowing down because of lots of disk reads. We are trying to
analyze it. We have lots of real time random reads. And We found that HBase
is constantly reading from disk instead of from memory.

Looks like block cache hit ratio is a good place to start.

Is there any way we can get block cache hit ratio by table or by region? I
could only find it per region server in region server UI. I couldn't find
it in API either.

Feel free to suggest any other metrics we should be looking at too.

Thanks,
Vaibhav
GumGum

Confidentiality Notice:  The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited.  If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.