You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Joarder KAMAL <jo...@gmail.com> on 2013/06/06 09:13:47 UTC
Extending HBase Metrics Framework
Two Questions:
1. Is it possible to extend HBase metrics/reporting framework to generate
custom metrics? I was looking at the api docs
http://hbase.apache.org/apidocs/
2. Is there any way to write my own client (instead of using Ganglia or
else) to retrieve the currently available metrics?
HBase version: 0.94.7
Any suggestions will be very helpful.
Many thanks ...
Regards,
Joarder Kamal
Re: Extending HBase Metrics Framework
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
Not 100% sure if I understand what you are after, but it sounds like you are OK/want to modify HBase source code and add tracking of some metric that is currently not being tracked.
Could you use Coda Hale Metric lib for that then? Any metric you capture with it can then go into JMX or Ganglia or SPM for HBase.
Otis
----
HBase / Hadoop Performance Monitoring - http://sematext.com/spm
----- Original Message -----
> From: Joarder KAMAL <jo...@gmail.com>
> To: dev@hbase.apache.org; "user@hbase.apache.org" <us...@hbase.apache.org>
> Cc:
> Sent: Thursday, June 6, 2013 3:13 AM
> Subject: Extending HBase Metrics Framework
>
>T wo Questions:
>
> 1. Is it possible to extend HBase metrics/reporting framework to generate
> custom metrics? I was looking at the api docs
> http://hbase.apache.org/apidocs/
> 2. Is there any way to write my own client (instead of using Ganglia or
> else) to retrieve the currently available metrics?
>
> HBase version: 0.94.7
>
> Any suggestions will be very helpful.
>
> Many thanks ...
>
> Regards,
> Joarder Kamal
>
Re: Extending HBase Metrics Framework
Posted by Jeff Kolesky <je...@opower.com>.
I have not written any code against the latest metrics, so I'm just
searching around. Take the advice with a grain of salt until a more
experienced developer pipes up.
BaseSourceImpl (implemented for hadoop1 and hadoop2) looks like the entry
point:
https://github.com/apache/hbase/blob/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseSourceImpl.java
Many implementations exist (also for both hadoop1 and hadoop2). One such
example is
https://github.com/apache/hbase/blob/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALSourceImpl.java
Jeff
On Thu, Jun 6, 2013 at 11:53 AM, Joarder KAMAL <jo...@gmail.com> wrote:
> Thanks for pointing out the way. I've looked at the Git code as well
> however looking back at the apidocs <http://hbase.apache.org/apidocs/> for
> HBase-0.97.0-Snapshot I can only see org.apache.hadoop.metrics2 package
> there.
>
> I am using HBase 0.94.7, could you kindly help directing to the right code
> repo and package?
>
> Thanks again...
>
>
> Regards,
> Joarder Kamal
>
>
> On 7 June 2013 03:50, Jeff Kolesky <je...@opower.com> wrote:
>
> > If you want to add metrics, you can implement a
> > org.apache.hadoop.metrics.Updater and register it with a MetricsContext.
> > There is an example at:
> >
> >
> >
> https://github.com/jeffkole/odoop/blob/master/hbase-filters/src/main/java/com/opower/hadoop/hbase/filter/DeployedFilterMetrics.java
> >
> > This exposes metrics for a filter that I wrote, but if you want to pull
> out
> > metrics for HBase internals, you likely need to do modify HBase source
> code
> > to get at the data or hook up to your metric updater.
> >
> > Jeff
> >
> >
> > On Thu, Jun 6, 2013 at 10:35 AM, Joarder KAMAL <jo...@gmail.com>
> wrote:
> >
> > > Hi Bing,
> > >
> > > Thanks for answering.
> > >
> > > 1. Probably my question was unclear. I was asking whether it is
> possible
> > to
> > > generate custom metrics from HBase by modifying the source code.
> > > 2. Thanks for letting me know that it is possible to log the metrics in
> > any
> > > kind of file. But I guess it would be still in XML format !! Can I save
> > the
> > > logs in raw text format?? I'll try to do this tomorrow.
> > >
> > > Thanks again.
> > >
> > >
> > > Regards,
> > > Joarder Kamal
> > >
> > >
> > >
> > > On 6 June 2013 17:42, Bing Jiang <ji...@gmail.com> wrote:
> > >
> > > > 1) Metrics infos have been collected from HMaster\RS, and covering
> rpc,
> > > > jvm, hbase functional module (like compaction\flush\request
> > > Count\DataBlock
> > > > read &&write etc).
> > > >
> > > > 2) You can set metrics info handler in conf/hadoop-metrics.properties
> > > > for example, you can set hbase.class =
> > > > org.apache.hadoop.metrics.ganglia.GangliaContext31, it will tell
> > Metrics
> > > > System where the metrics will be sent to. Of course, this is Ganglia.
> > > > if set
> > > >
> > hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext,
> > > > it will write the metrics to file.
> > > >
> > > > # hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> > > > # hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> > > > # hbase.period=10
> > > > # hbase.servers=10.100.4.151:8650
> > > >
> > #hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext
> > > > #hbase.period=10
> > > > #hbase.fileName=/opt/hbase/logs/metrics_hbase.log
> > > >
> > > >
> > > >
> > > >
> > > > 2013/6/6 Joarder KAMAL <jo...@gmail.com>
> > > >
> > > > > Two Questions:
> > > > >
> > > > > 1. Is it possible to extend HBase metrics/reporting framework to
> > > generate
> > > > > custom metrics? I was looking at the api docs
> > > > > http://hbase.apache.org/apidocs/
> > > > > 2. Is there any way to write my own client (instead of using
> Ganglia
> > or
> > > > > else) to retrieve the currently available metrics?
> > > > >
> > > > > HBase version: 0.94.7
> > > > >
> > > > > Any suggestions will be very helpful.
> > > > >
> > > > > Many thanks ...
> > > > >
> > > > > Regards,
> > > > > Joarder Kamal
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Bing Jiang
> > > > Tel：(86)134-2619-1361
> > > > weibo: http://weibo.com/jiangbinglover
> > > > BLOG: http://blog.sina.com.cn/jiangbinglover
> > > > National Research Center for Intelligent Computing Systems
> > > > Institute of Computing technology
> > > > Graduate University of Chinese Academy of Science
> > > >
> > >
> >
> >
> >
> > --
> > *Jeff Kolesky*
> > Chief Software Architect
> > *Opower*
> >
>
--
*Jeff Kolesky*
Chief Software Architect
*Opower*
Re: Extending HBase Metrics Framework
Posted by Joarder KAMAL <jo...@gmail.com>.
Thanks for pointing out the way. I've looked at the Git code as well
however looking back at the apidocs <http://hbase.apache.org/apidocs/> for
HBase-0.97.0-Snapshot I can only see org.apache.hadoop.metrics2 package
there.
I am using HBase 0.94.7, could you kindly help directing to the right code
repo and package?
Thanks again...
Regards,
Joarder Kamal
On 7 June 2013 03:50, Jeff Kolesky <je...@opower.com> wrote:
> If you want to add metrics, you can implement a
> org.apache.hadoop.metrics.Updater and register it with a MetricsContext.
> There is an example at:
>
>
> https://github.com/jeffkole/odoop/blob/master/hbase-filters/src/main/java/com/opower/hadoop/hbase/filter/DeployedFilterMetrics.java
>
> This exposes metrics for a filter that I wrote, but if you want to pull out
> metrics for HBase internals, you likely need to do modify HBase source code
> to get at the data or hook up to your metric updater.
>
> Jeff
>
>
> On Thu, Jun 6, 2013 at 10:35 AM, Joarder KAMAL <jo...@gmail.com> wrote:
>
> > Hi Bing,
> >
> > Thanks for answering.
> >
> > 1. Probably my question was unclear. I was asking whether it is possible
> to
> > generate custom metrics from HBase by modifying the source code.
> > 2. Thanks for letting me know that it is possible to log the metrics in
> any
> > kind of file. But I guess it would be still in XML format !! Can I save
> the
> > logs in raw text format?? I'll try to do this tomorrow.
> >
> > Thanks again.
> >
> >
> > Regards,
> > Joarder Kamal
> >
> >
> >
> > On 6 June 2013 17:42, Bing Jiang <ji...@gmail.com> wrote:
> >
> > > 1) Metrics infos have been collected from HMaster\RS, and covering rpc,
> > > jvm, hbase functional module (like compaction\flush\request
> > Count\DataBlock
> > > read &&write etc).
> > >
> > > 2) You can set metrics info handler in conf/hadoop-metrics.properties
> > > for example, you can set hbase.class =
> > > org.apache.hadoop.metrics.ganglia.GangliaContext31, it will tell
> Metrics
> > > System where the metrics will be sent to. Of course, this is Ganglia.
> > > if set
> > >
> hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext,
> > > it will write the metrics to file.
> > >
> > > # hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> > > # hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> > > # hbase.period=10
> > > # hbase.servers=10.100.4.151:8650
> > >
> #hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext
> > > #hbase.period=10
> > > #hbase.fileName=/opt/hbase/logs/metrics_hbase.log
> > >
> > >
> > >
> > >
> > > 2013/6/6 Joarder KAMAL <jo...@gmail.com>
> > >
> > > > Two Questions:
> > > >
> > > > 1. Is it possible to extend HBase metrics/reporting framework to
> > generate
> > > > custom metrics? I was looking at the api docs
> > > > http://hbase.apache.org/apidocs/
> > > > 2. Is there any way to write my own client (instead of using Ganglia
> or
> > > > else) to retrieve the currently available metrics?
> > > >
> > > > HBase version: 0.94.7
> > > >
> > > > Any suggestions will be very helpful.
> > > >
> > > > Many thanks ...
> > > >
> > > > Regards,
> > > > Joarder Kamal
> > > >
> > >
> > >
> > >
> > > --
> > > Bing Jiang
> > > Tel：(86)134-2619-1361
> > > weibo: http://weibo.com/jiangbinglover
> > > BLOG: http://blog.sina.com.cn/jiangbinglover
> > > National Research Center for Intelligent Computing Systems
> > > Institute of Computing technology
> > > Graduate University of Chinese Academy of Science
> > >
> >
>
>
>
> --
> *Jeff Kolesky*
> Chief Software Architect
> *Opower*
>
Re: Extending HBase Metrics Framework
Posted by Jeff Kolesky <je...@opower.com>.
If you want to add metrics, you can implement a
org.apache.hadoop.metrics.Updater and register it with a MetricsContext.
There is an example at:
https://github.com/jeffkole/odoop/blob/master/hbase-filters/src/main/java/com/opower/hadoop/hbase/filter/DeployedFilterMetrics.java
This exposes metrics for a filter that I wrote, but if you want to pull out
metrics for HBase internals, you likely need to do modify HBase source code
to get at the data or hook up to your metric updater.
Jeff
On Thu, Jun 6, 2013 at 10:35 AM, Joarder KAMAL <jo...@gmail.com> wrote:
> Hi Bing,
>
> Thanks for answering.
>
> 1. Probably my question was unclear. I was asking whether it is possible to
> generate custom metrics from HBase by modifying the source code.
> 2. Thanks for letting me know that it is possible to log the metrics in any
> kind of file. But I guess it would be still in XML format !! Can I save the
> logs in raw text format?? I'll try to do this tomorrow.
>
> Thanks again.
>
>
> Regards,
> Joarder Kamal
>
>
>
> On 6 June 2013 17:42, Bing Jiang <ji...@gmail.com> wrote:
>
> > 1) Metrics infos have been collected from HMaster\RS, and covering rpc,
> > jvm, hbase functional module (like compaction\flush\request
> Count\DataBlock
> > read &&write etc).
> >
> > 2) You can set metrics info handler in conf/hadoop-metrics.properties
> > for example, you can set hbase.class =
> > org.apache.hadoop.metrics.ganglia.GangliaContext31, it will tell Metrics
> > System where the metrics will be sent to. Of course, this is Ganglia.
> > if set
> > hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext,
> > it will write the metrics to file.
> >
> > # hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> > # hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> > # hbase.period=10
> > # hbase.servers=10.100.4.151:8650
> > #hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext
> > #hbase.period=10
> > #hbase.fileName=/opt/hbase/logs/metrics_hbase.log
> >
> >
> >
> >
> > 2013/6/6 Joarder KAMAL <jo...@gmail.com>
> >
> > > Two Questions:
> > >
> > > 1. Is it possible to extend HBase metrics/reporting framework to
> generate
> > > custom metrics? I was looking at the api docs
> > > http://hbase.apache.org/apidocs/
> > > 2. Is there any way to write my own client (instead of using Ganglia or
> > > else) to retrieve the currently available metrics?
> > >
> > > HBase version: 0.94.7
> > >
> > > Any suggestions will be very helpful.
> > >
> > > Many thanks ...
> > >
> > > Regards,
> > > Joarder Kamal
> > >
> >
> >
> >
> > --
> > Bing Jiang
> > Tel：(86)134-2619-1361
> > weibo: http://weibo.com/jiangbinglover
> > BLOG: http://blog.sina.com.cn/jiangbinglover
> > National Research Center for Intelligent Computing Systems
> > Institute of Computing technology
> > Graduate University of Chinese Academy of Science
> >
>
--
*Jeff Kolesky*
Chief Software Architect
*Opower*
Re: Extending HBase Metrics Framework
Posted by Joarder KAMAL <jo...@gmail.com>.
Hi Bing,
Thanks for answering.
1. Probably my question was unclear. I was asking whether it is possible to
generate custom metrics from HBase by modifying the source code.
2. Thanks for letting me know that it is possible to log the metrics in any
kind of file. But I guess it would be still in XML format !! Can I save the
logs in raw text format?? I'll try to do this tomorrow.
Thanks again.
Regards,
Joarder Kamal
On 6 June 2013 17:42, Bing Jiang <ji...@gmail.com> wrote:
> 1) Metrics infos have been collected from HMaster\RS, and covering rpc,
> jvm, hbase functional module (like compaction\flush\request Count\DataBlock
> read &&write etc).
>
> 2) You can set metrics info handler in conf/hadoop-metrics.properties
> for example, you can set hbase.class =
> org.apache.hadoop.metrics.ganglia.GangliaContext31, it will tell Metrics
> System where the metrics will be sent to. Of course, this is Ganglia.
> if set
> hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext,
> it will write the metrics to file.
>
> # hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> # hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
> # hbase.period=10
> # hbase.servers=10.100.4.151:8650
> #hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext
> #hbase.period=10
> #hbase.fileName=/opt/hbase/logs/metrics_hbase.log
>
>
>
>
> 2013/6/6 Joarder KAMAL <jo...@gmail.com>
>
> > Two Questions:
> >
> > 1. Is it possible to extend HBase metrics/reporting framework to generate
> > custom metrics? I was looking at the api docs
> > http://hbase.apache.org/apidocs/
> > 2. Is there any way to write my own client (instead of using Ganglia or
> > else) to retrieve the currently available metrics?
> >
> > HBase version: 0.94.7
> >
> > Any suggestions will be very helpful.
> >
> > Many thanks ...
> >
> > Regards,
> > Joarder Kamal
> >
>
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
Re: Extending HBase Metrics Framework
Posted by Bing Jiang <ji...@gmail.com>.
1) Metrics infos have been collected from HMaster\RS, and covering rpc,
jvm, hbase functional module (like compaction\flush\request Count\DataBlock
read &&write etc).
2) You can set metrics info handler in conf/hadoop-metrics.properties
for example, you can set hbase.class =
org.apache.hadoop.metrics.ganglia.GangliaContext31, it will tell Metrics
System where the metrics will be sent to. Of course, this is Ganglia.
if set
hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext,
it will write the metrics to file.
# hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
# hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
# hbase.period=10
# hbase.servers=10.100.4.151:8650
#hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext
#hbase.period=10
#hbase.fileName=/opt/hbase/logs/metrics_hbase.log
2013/6/6 Joarder KAMAL <jo...@gmail.com>
> Two Questions:
>
> 1. Is it possible to extend HBase metrics/reporting framework to generate
> custom metrics? I was looking at the api docs
> http://hbase.apache.org/apidocs/
> 2. Is there any way to write my own client (instead of using Ganglia or
> else) to retrieve the currently available metrics?
>
> HBase version: 0.94.7
>
> Any suggestions will be very helpful.
>
> Many thanks ...
>
> Regards,
> Joarder Kamal
>
--
Bing Jiang
Tel：(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
Re: Extending HBase Metrics Framework
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
Not 100% sure if I understand what you are after, but it sounds like you are OK/want to modify HBase source code and add tracking of some metric that is currently not being tracked.
Could you use Coda Hale Metric lib for that then? Any metric you capture with it can then go into JMX or Ganglia or SPM for HBase.
Otis
----
HBase / Hadoop Performance Monitoring - http://sematext.com/spm
----- Original Message -----
> From: Joarder KAMAL <jo...@gmail.com>
> To: dev@hbase.apache.org; "user@hbase.apache.org" <us...@hbase.apache.org>
> Cc:
> Sent: Thursday, June 6, 2013 3:13 AM
> Subject: Extending HBase Metrics Framework
>
>T wo Questions:
>
> 1. Is it possible to extend HBase metrics/reporting framework to generate
> custom metrics? I was looking at the api docs
> http://hbase.apache.org/apidocs/
> 2. Is there any way to write my own client (instead of using Ganglia or
> else) to retrieve the currently available metrics?
>
> HBase version: 0.94.7
>
> Any suggestions will be very helpful.
>
> Many thanks ...
>
> Regards,
> Joarder Kamal
>