You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by yun peng <pe...@gmail.com> on 2012/11/10 16:03:59 UTC
How to monitor total disk access in HBase?
Hi, I want to profile the # of disk access (both random and sequential)
issued from HBase (into HDFS). For disk reads, I have tried use
blockCacheMissCount, which seems working. But is it the correct way for
reads (I can't confirmed it from HBase documents)?
For disk writes, I can't find any metrics in HBase. How should one get disk
writes in HBase?
Besides, it's best if I can obtain separate numbers regarding to random
access and sequential access. Thanks in advance.
Regards,
Yun
Re: How to monitor total disk access in HBase?
Posted by PG <pe...@gmail.com>.
Thanks, for the suggestions. Will do instrumentation based on JMX.
Regards,
Yun
On Nov 12, 2012, at 12:59 AM, Stack <st...@duboce.net> wrote:
> On Sat, Nov 10, 2012 at 7:03 AM, yun peng <pe...@gmail.com> wrote:
>> Hi, I want to profile the # of disk access (both random and sequential)
>> issued from HBase (into HDFS). For disk reads, I have tried use
>> blockCacheMissCount, which seems working. But is it the correct way for
>> reads (I can't confirmed it from HBase documents)?
>>
>
> That should give you a coarse measure. If you need better than that
> you may need to instrument the code some to dump more detailed metric
> on whether random or sequential access. Beware that hdfs may be
> reading from file system cache avoiding disk altogether on some reads.
>
>> For disk writes, I can't find any metrics in HBase. How should one get disk
>> writes in HBase?
>>
>
> We write when we append to the WAL and when we flush hfiles. If you
> need actual disk accesses, you'll probably need to add some extra
> emissions in the code; per WAL edit and then as we flush. Again, hdfs
> flush/sync usually means flush from hbase to hdfs; more specifically
> to datanode memory and not necessarily to disk (at least currently).
>
> St.Ack
Re: How to monitor total disk access in HBase?
Posted by Stack <st...@duboce.net>.
On Sat, Nov 10, 2012 at 7:03 AM, yun peng <pe...@gmail.com> wrote:
> Hi, I want to profile the # of disk access (both random and sequential)
> issued from HBase (into HDFS). For disk reads, I have tried use
> blockCacheMissCount, which seems working. But is it the correct way for
> reads (I can't confirmed it from HBase documents)?
>
That should give you a coarse measure. If you need better than that
you may need to instrument the code some to dump more detailed metric
on whether random or sequential access. Beware that hdfs may be
reading from file system cache avoiding disk altogether on some reads.
> For disk writes, I can't find any metrics in HBase. How should one get disk
> writes in HBase?
>
We write when we append to the WAL and when we flush hfiles. If you
need actual disk accesses, you'll probably need to add some extra
emissions in the code; per WAL edit and then as we flush. Again, hdfs
flush/sync usually means flush from hbase to hdfs; more specifically
to datanode memory and not necessarily to disk (at least currently).
St.Ack
Re: How to monitor total disk access in HBase?
Posted by Elliott Clark <ec...@apache.org>.
Thanks for bringing 3869 up. I'll try and get to it as I move the last few
parts of HBase to metrics2.
On Mon, Nov 12, 2012 at 6:01 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> I was just trawling HBase JIRA for this info after one happy user of SPM
> for HBase asked if we could show read requests separately from write
> requests.
>
> See
> https://issues.apache.org/jira/browse/HBASE-4989
> https://issues.apache.org/jira/browse/HBASE-3647
>
> There is also https://issues.apache.org/jira/browse/HBASE-3869, but it's
> still patchless.
>
> Otis
> --
> Performance Monitoring - http://sematext.com/spm/index.html
>
>
> On Sat, Nov 10, 2012 at 10:03 AM, yun peng <pe...@gmail.com> wrote:
>
> > Hi, I want to profile the # of disk access (both random and sequential)
> > issued from HBase (into HDFS). For disk reads, I have tried use
> > blockCacheMissCount, which seems working. But is it the correct way for
> > reads (I can't confirmed it from HBase documents)?
> >
> > For disk writes, I can't find any metrics in HBase. How should one get
> disk
> > writes in HBase?
> >
> > Besides, it's best if I can obtain separate numbers regarding to random
> > access and sequential access. Thanks in advance.
> > Regards,
> > Yun
> >
>
Re: How to monitor total disk access in HBase?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I was just trawling HBase JIRA for this info after one happy user of SPM
for HBase asked if we could show read requests separately from write
requests.
See
https://issues.apache.org/jira/browse/HBASE-4989
https://issues.apache.org/jira/browse/HBASE-3647
There is also https://issues.apache.org/jira/browse/HBASE-3869, but it's
still patchless.
Otis
--
Performance Monitoring - http://sematext.com/spm/index.html
On Sat, Nov 10, 2012 at 10:03 AM, yun peng <pe...@gmail.com> wrote:
> Hi, I want to profile the # of disk access (both random and sequential)
> issued from HBase (into HDFS). For disk reads, I have tried use
> blockCacheMissCount, which seems working. But is it the correct way for
> reads (I can't confirmed it from HBase documents)?
>
> For disk writes, I can't find any metrics in HBase. How should one get disk
> writes in HBase?
>
> Besides, it's best if I can obtain separate numbers regarding to random
> access and sequential access. Thanks in advance.
> Regards,
> Yun
>