You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Kannan Muthukkaruppan <Ka...@facebook.com> on 2010/07/29 04:29:31 UTC
RPC statistics from region servers...
I was using jconsole to check the stats published by region servers..
Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
and,
RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
I no longer see attributes like:
"incrementColumnValueNumOps "
"incrementColumnValueAvgTime "
"putNumOps "
"putAvgTime "
"getNumOps "
"getAvgTime "
"deleteNumOps "
"deleteAvgTime "
"openScannerNumOps "
"openScannerAvtTime "
"nextNumOps "
"nextAvgTime "
"closeNumOps "
"closeAvgTime "
Anyone aware of what change happened in this area?
regards,
Kannan
RE: RPC statistics from region servers...
Posted by Kannan Muthukkaruppan <Ka...@facebook.com>.
Correct. This is in trunk. (We had previously tried this in 0.20.x).
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Wednesday, July 28, 2010 7:32 PM
To: dev@hbase.apache.org
Subject: Re: RPC statistics from region servers...
You are comparing trunk with 0.20.x I guess.
On Wed, Jul 28, 2010 at 7:29 PM, Kannan Muthukkaruppan
<Ka...@facebook.com>wrote:
> I was using jconsole to check the stats published by region servers..
>
> Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
>
> RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> and,
> RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
>
> I no longer see attributes like:
> "incrementColumnValueNumOps "
> "incrementColumnValueAvgTime "
> "putNumOps "
> "putAvgTime "
> "getNumOps "
> "getAvgTime "
> "deleteNumOps "
> "deleteAvgTime "
> "openScannerNumOps "
> "openScannerAvtTime "
> "nextNumOps "
> "nextAvgTime "
> "closeNumOps "
> "closeAvgTime "
>
> Anyone aware of what change happened in this area?
>
> regards,
> Kannan
>
>
>
Re: RPC statistics from region servers...
Posted by Ted Yu <yu...@gmail.com>.
You are comparing trunk with 0.20.x I guess.
On Wed, Jul 28, 2010 at 7:29 PM, Kannan Muthukkaruppan
<Ka...@facebook.com>wrote:
> I was using jconsole to check the stats published by region servers..
>
> Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
>
> RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> and,
> RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
>
> I no longer see attributes like:
> "incrementColumnValueNumOps "
> "incrementColumnValueAvgTime "
> "putNumOps "
> "putAvgTime "
> "getNumOps "
> "getAvgTime "
> "deleteNumOps "
> "deleteAvgTime "
> "openScannerNumOps "
> "openScannerAvtTime "
> "nextNumOps "
> "nextAvgTime "
> "closeNumOps "
> "closeAvgTime "
>
> Anyone aware of what change happened in this area?
>
> regards,
> Kannan
>
>
>
Re: RPC statistics from region servers...
Posted by Gary Helmling <gh...@gmail.com>.
Fixed in HBASE-2890
On Jul 31, 2010, at 4:38 AM, "KannanM@gmail.com (Kannan
Muthukkaruppan)" <ka...@gmail.com> wrote:
> Gary: Got it. Thanks for the clarification.
>
> On Thu, Jul 29, 2010 at 4:39 PM, Gary Helmling <gh...@gmail.com>
> wrote:
>
>> Ryan:
>> Yep, agreed on removal of the code to method name mappings. Makes
>> send to
>> pull the HBASE-2146 change at the same time as all it did was wrap
>> the same
>> mappings.
>>
>>
>> Kannan:
>>
>>>> We could do a similar initialization of the JMX MBean by just
>>> introspecting
>>>> the RPC interfaces though (HMasterInterface,
>>>> HMasterRegionInterface,
>>>> HRegionInterface).
>>>
>>> What would be the net effect of this change compared to the state
>>> trunk
>> is
>>> currently in? Could you clarify?
>>>
>>>
>> This would just setup the RPC stats MBean on server startup so that
>> it
>> contained an attribute for each defined RPC method. Otherwise, as
>> you saw,
>> the attributes don't even show up in JMX until you have some cluster
>> activity with those methods and have configured hadoop-
>> metrics.properties
>> to
>> do periodic updates.
>>
>> It's not really a functional change, just makes the stats a little
>> more
>> intuitive. Instead of sometimes displaying, sometimes not, they
>> would show
>> up immediately with zeroed values. You would still need the
>> hadoop-metrics.properties configuration using a context with periodic
>> updates in order for actual values to show, though.
>>
>> It's a trivial change, I'll throw up a patch.
>>
>>
>> Gary
>>
Re: RPC statistics from region servers...
Posted by "KannanM@gmail.com (Kannan Muthukkaruppan)" <ka...@gmail.com>.
Gary: Got it. Thanks for the clarification.
On Thu, Jul 29, 2010 at 4:39 PM, Gary Helmling <gh...@gmail.com> wrote:
> Ryan:
> Yep, agreed on removal of the code to method name mappings. Makes send to
> pull the HBASE-2146 change at the same time as all it did was wrap the same
> mappings.
>
>
> Kannan:
>
> > > We could do a similar initialization of the JMX MBean by just
> > introspecting
> > > the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
> > > HRegionInterface).
> >
> > What would be the net effect of this change compared to the state trunk
> is
> > currently in? Could you clarify?
> >
> >
> This would just setup the RPC stats MBean on server startup so that it
> contained an attribute for each defined RPC method. Otherwise, as you saw,
> the attributes don't even show up in JMX until you have some cluster
> activity with those methods and have configured hadoop-metrics.properties
> to
> do periodic updates.
>
> It's not really a functional change, just makes the stats a little more
> intuitive. Instead of sometimes displaying, sometimes not, they would show
> up immediately with zeroed values. You would still need the
> hadoop-metrics.properties configuration using a context with periodic
> updates in order for actual values to show, though.
>
> It's a trivial change, I'll throw up a patch.
>
>
> Gary
>
Re: RPC statistics from region servers...
Posted by Gary Helmling <gh...@gmail.com>.
Ryan:
Yep, agreed on removal of the code to method name mappings. Makes send to
pull the HBASE-2146 change at the same time as all it did was wrap the same
mappings.
Kannan:
> > We could do a similar initialization of the JMX MBean by just
> introspecting
> > the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
> > HRegionInterface).
>
> What would be the net effect of this change compared to the state trunk is
> currently in? Could you clarify?
>
>
This would just setup the RPC stats MBean on server startup so that it
contained an attribute for each defined RPC method. Otherwise, as you saw,
the attributes don't even show up in JMX until you have some cluster
activity with those methods and have configured hadoop-metrics.properties to
do periodic updates.
It's not really a functional change, just makes the stats a little more
intuitive. Instead of sometimes displaying, sometimes not, they would show
up immediately with zeroed values. You would still need the
hadoop-metrics.properties configuration using a context with periodic
updates in order for actual values to show, though.
It's a trivial change, I'll throw up a patch.
Gary
RE: RPC statistics from region servers...
Posted by Kannan Muthukkaruppan <Ka...@facebook.com>.
I do seem to be getting data in multiPut* and get* now from all the Region Servers. So things look good now.
Gary:
> We could do a similar initialization of the JMX MBean by just introspecting
> the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
> HRegionInterface).
What would be the net effect of this change compared to the state trunk is currently in? Could you clarify?
Regards,
Kannan
-----Original Message-----
From: Gary Helmling [mailto:ghelmling@gmail.com]
Sent: Thursday, July 29, 2010 3:50 PM
To: dev@hbase.apache.org
Subject: Re: RPC statistics from region servers...
Kannan,
It looks like that code went away as part of HBASE-2219, which removed the
mapping of rpc method names to codes that was previously used in
serialization.
We could do a similar initialization of the JMX MBean by just introspecting
the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
HRegionInterface). Want to open a JIRA?
Gary
On Thu, Jul 29, 2010 at 3:16 PM, Kannan Muthukkaruppan
<Ka...@facebook.com>wrote:
> Pushed the hadoop-metrics.properties changes, and it seems better but not
> quite. [It doesn't look like all the Region Servers are reporting similar
> set of counters (on some I am seeing "multiput" stuff, although we aren' t),
> lot of the average times are close to 0.]
>
> So next I looked into the HBASE-2146 patch, and that doesn't appear to be
> in trunk although at some point it was in trunk earlier.
>
>
> http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402
>
> Will try applying the patch locally to see if that helps.
>
> Regards,
> Kannan
> -----Original Message-----
> From: Kannan Muthukkaruppan [mailto:Kannan@facebook.com]
> Sent: Thursday, July 29, 2010 2:41 PM
> To: dev@hbase.apache.org
> Subject: RE: RPC statistics from region servers...
>
> Gary,
>
> With some recent restructuring of our conf/* and cluster deploy scripts, we
> did lose our hadoop-metrics.properties changes, and reverted back to the
> default one. That could very well report the issue I ran into. I'll report
> back when I try it out.
>
> Thanks again for your help on this.
>
> Q: Why don't we make the suggested settings in
> http://hbase.apache.org/docs/current/metrics.html be part of the default
> conf/hadoop-metrics.properties?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Gary Helmling [mailto:ghelmling@gmail.com]
> Sent: Thursday, July 29, 2010 1:08 AM
> To: dev@hbase.apache.org
> Subject: Re: RPC statistics from region servers...
>
> Hmm, this sounds like HBASE-2146. That patch had added code to initialize
> the RPC metrics MBean with a know list of RPC method names maintained in
> HBaseRPC.Invocation. That way all per-method metrics would show up on
> server startup. I believe that that method name mapping was later removed
> though (it can make the RPC serialization brittle, complicating things like
> rolling restarts), so maybe the patch that was applied to trunk was later
> removed?
>
> The MBean attributes for individual RPC methods _should_ be there after
> there has been activity involving those requests against the cluster. If
> you start jconsole prior to the cluster activity however, I don't think the
> MBean definition will update, so you'll need to restart jconsole for them
> to
> show up.
>
> In addition, you should make sure you're running with a metrics context
> with
> an updater configured in
> conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
> GangliaContext, etc). And configure rpc.period with the update interval.
> These metrics don't show up as live counters in JMX, they're only pushed
> out
> to JMX during the periodic updates.
>
> If you've already configured all of those conditions and the metrics
> _still_
> aren't showing up, then let me know and I'll try to do some testing on
> trunk
> tomorrow.
>
> --gh
>
>
> On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
> <Ka...@facebook.com>wrote:
>
> > > they don't exist until used?
> >
> > I was doing a whole bunch of puts & gets against the cluster during that
> > time. That's what you mean, correct?
> >
> > Regards,
> > Kannan
> > -----Original Message-----
> > From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> > Sent: Wednesday, July 28, 2010 8:21 PM
> > To: dev@hbase.apache.org
> > Cc: hbase-dev@hadoop.apache.org
> > Subject: Re: RPC statistics from region servers...
> >
> > I think those are dynamic metrics... they don't exist until used?
> >
> > On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
> > wrote:
> > > I was using jconsole to check the stats published by region servers..
> > >
> > > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
> > >
> > > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> > > and,
> > > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
> > >
> > > I no longer see attributes like:
> > > "incrementColumnValueNumOps "
> > > "incrementColumnValueAvgTime "
> > > "putNumOps "
> > > "putAvgTime "
> > > "getNumOps "
> > > "getAvgTime "
> > > "deleteNumOps "
> > > "deleteAvgTime "
> > > "openScannerNumOps "
> > > "openScannerAvtTime "
> > > "nextNumOps "
> > > "nextAvgTime "
> > > "closeNumOps "
> > > "closeAvgTime "
> > >
> > > Anyone aware of what change happened in this area?
> > >
> > > regards,
> > > Kannan
> > >
> > >
> >
>
Re: RPC statistics from region servers...
Posted by Ryan Rawson <ry...@gmail.com>.
Hi,
I did HBASE-2219, it was a pretty bad hack, since it prevented even
doing a uber-trivial RPC change without ruining background
compatibility, and in my micro-benchmarks, it wasnt even faster to
boot.
I was talking to Kannan earlier on IRC, and he was noting about
multi-put vs put... I noted that nearly all Puts, even of Size=1 go
through the multi-put code path now, and that seemed to be part of the
problem as well.
-ryan
On Thu, Jul 29, 2010 at 3:50 PM, Gary Helmling <gh...@gmail.com> wrote:
> Kannan,
>
> It looks like that code went away as part of HBASE-2219, which removed the
> mapping of rpc method names to codes that was previously used in
> serialization.
>
> We could do a similar initialization of the JMX MBean by just introspecting
> the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
> HRegionInterface). Want to open a JIRA?
>
> Gary
>
>
> On Thu, Jul 29, 2010 at 3:16 PM, Kannan Muthukkaruppan
> <Ka...@facebook.com>wrote:
>
>> Pushed the hadoop-metrics.properties changes, and it seems better but not
>> quite. [It doesn't look like all the Region Servers are reporting similar
>> set of counters (on some I am seeing "multiput" stuff, although we aren' t),
>> lot of the average times are close to 0.]
>>
>> So next I looked into the HBASE-2146 patch, and that doesn't appear to be
>> in trunk although at some point it was in trunk earlier.
>>
>>
>> http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402
>>
>> Will try applying the patch locally to see if that helps.
>>
>> Regards,
>> Kannan
>> -----Original Message-----
>> From: Kannan Muthukkaruppan [mailto:Kannan@facebook.com]
>> Sent: Thursday, July 29, 2010 2:41 PM
>> To: dev@hbase.apache.org
>> Subject: RE: RPC statistics from region servers...
>>
>> Gary,
>>
>> With some recent restructuring of our conf/* and cluster deploy scripts, we
>> did lose our hadoop-metrics.properties changes, and reverted back to the
>> default one. That could very well report the issue I ran into. I'll report
>> back when I try it out.
>>
>> Thanks again for your help on this.
>>
>> Q: Why don't we make the suggested settings in
>> http://hbase.apache.org/docs/current/metrics.html be part of the default
>> conf/hadoop-metrics.properties?
>>
>> Regards,
>> Kannan
>> -----Original Message-----
>> From: Gary Helmling [mailto:ghelmling@gmail.com]
>> Sent: Thursday, July 29, 2010 1:08 AM
>> To: dev@hbase.apache.org
>> Subject: Re: RPC statistics from region servers...
>>
>> Hmm, this sounds like HBASE-2146. That patch had added code to initialize
>> the RPC metrics MBean with a know list of RPC method names maintained in
>> HBaseRPC.Invocation. That way all per-method metrics would show up on
>> server startup. I believe that that method name mapping was later removed
>> though (it can make the RPC serialization brittle, complicating things like
>> rolling restarts), so maybe the patch that was applied to trunk was later
>> removed?
>>
>> The MBean attributes for individual RPC methods _should_ be there after
>> there has been activity involving those requests against the cluster. If
>> you start jconsole prior to the cluster activity however, I don't think the
>> MBean definition will update, so you'll need to restart jconsole for them
>> to
>> show up.
>>
>> In addition, you should make sure you're running with a metrics context
>> with
>> an updater configured in
>> conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
>> GangliaContext, etc). And configure rpc.period with the update interval.
>> These metrics don't show up as live counters in JMX, they're only pushed
>> out
>> to JMX during the periodic updates.
>>
>> If you've already configured all of those conditions and the metrics
>> _still_
>> aren't showing up, then let me know and I'll try to do some testing on
>> trunk
>> tomorrow.
>>
>> --gh
>>
>>
>> On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
>> <Ka...@facebook.com>wrote:
>>
>> > > they don't exist until used?
>> >
>> > I was doing a whole bunch of puts & gets against the cluster during that
>> > time. That's what you mean, correct?
>> >
>> > Regards,
>> > Kannan
>> > -----Original Message-----
>> > From: Ryan Rawson [mailto:ryanobjc@gmail.com]
>> > Sent: Wednesday, July 28, 2010 8:21 PM
>> > To: dev@hbase.apache.org
>> > Cc: hbase-dev@hadoop.apache.org
>> > Subject: Re: RPC statistics from region servers...
>> >
>> > I think those are dynamic metrics... they don't exist until used?
>> >
>> > On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
>> > wrote:
>> > > I was using jconsole to check the stats published by region servers..
>> > >
>> > > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
>> > >
>> > > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
>> > > and,
>> > > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
>> > >
>> > > I no longer see attributes like:
>> > > "incrementColumnValueNumOps "
>> > > "incrementColumnValueAvgTime "
>> > > "putNumOps "
>> > > "putAvgTime "
>> > > "getNumOps "
>> > > "getAvgTime "
>> > > "deleteNumOps "
>> > > "deleteAvgTime "
>> > > "openScannerNumOps "
>> > > "openScannerAvtTime "
>> > > "nextNumOps "
>> > > "nextAvgTime "
>> > > "closeNumOps "
>> > > "closeAvgTime "
>> > >
>> > > Anyone aware of what change happened in this area?
>> > >
>> > > regards,
>> > > Kannan
>> > >
>> > >
>> >
>>
>
Re: RPC statistics from region servers...
Posted by Gary Helmling <gh...@gmail.com>.
Kannan,
It looks like that code went away as part of HBASE-2219, which removed the
mapping of rpc method names to codes that was previously used in
serialization.
We could do a similar initialization of the JMX MBean by just introspecting
the RPC interfaces though (HMasterInterface, HMasterRegionInterface,
HRegionInterface). Want to open a JIRA?
Gary
On Thu, Jul 29, 2010 at 3:16 PM, Kannan Muthukkaruppan
<Ka...@facebook.com>wrote:
> Pushed the hadoop-metrics.properties changes, and it seems better but not
> quite. [It doesn't look like all the Region Servers are reporting similar
> set of counters (on some I am seeing "multiput" stuff, although we aren' t),
> lot of the average times are close to 0.]
>
> So next I looked into the HBASE-2146 patch, and that doesn't appear to be
> in trunk although at some point it was in trunk earlier.
>
>
> http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402
>
> Will try applying the patch locally to see if that helps.
>
> Regards,
> Kannan
> -----Original Message-----
> From: Kannan Muthukkaruppan [mailto:Kannan@facebook.com]
> Sent: Thursday, July 29, 2010 2:41 PM
> To: dev@hbase.apache.org
> Subject: RE: RPC statistics from region servers...
>
> Gary,
>
> With some recent restructuring of our conf/* and cluster deploy scripts, we
> did lose our hadoop-metrics.properties changes, and reverted back to the
> default one. That could very well report the issue I ran into. I'll report
> back when I try it out.
>
> Thanks again for your help on this.
>
> Q: Why don't we make the suggested settings in
> http://hbase.apache.org/docs/current/metrics.html be part of the default
> conf/hadoop-metrics.properties?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Gary Helmling [mailto:ghelmling@gmail.com]
> Sent: Thursday, July 29, 2010 1:08 AM
> To: dev@hbase.apache.org
> Subject: Re: RPC statistics from region servers...
>
> Hmm, this sounds like HBASE-2146. That patch had added code to initialize
> the RPC metrics MBean with a know list of RPC method names maintained in
> HBaseRPC.Invocation. That way all per-method metrics would show up on
> server startup. I believe that that method name mapping was later removed
> though (it can make the RPC serialization brittle, complicating things like
> rolling restarts), so maybe the patch that was applied to trunk was later
> removed?
>
> The MBean attributes for individual RPC methods _should_ be there after
> there has been activity involving those requests against the cluster. If
> you start jconsole prior to the cluster activity however, I don't think the
> MBean definition will update, so you'll need to restart jconsole for them
> to
> show up.
>
> In addition, you should make sure you're running with a metrics context
> with
> an updater configured in
> conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
> GangliaContext, etc). And configure rpc.period with the update interval.
> These metrics don't show up as live counters in JMX, they're only pushed
> out
> to JMX during the periodic updates.
>
> If you've already configured all of those conditions and the metrics
> _still_
> aren't showing up, then let me know and I'll try to do some testing on
> trunk
> tomorrow.
>
> --gh
>
>
> On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
> <Ka...@facebook.com>wrote:
>
> > > they don't exist until used?
> >
> > I was doing a whole bunch of puts & gets against the cluster during that
> > time. That's what you mean, correct?
> >
> > Regards,
> > Kannan
> > -----Original Message-----
> > From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> > Sent: Wednesday, July 28, 2010 8:21 PM
> > To: dev@hbase.apache.org
> > Cc: hbase-dev@hadoop.apache.org
> > Subject: Re: RPC statistics from region servers...
> >
> > I think those are dynamic metrics... they don't exist until used?
> >
> > On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
> > wrote:
> > > I was using jconsole to check the stats published by region servers..
> > >
> > > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
> > >
> > > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> > > and,
> > > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
> > >
> > > I no longer see attributes like:
> > > "incrementColumnValueNumOps "
> > > "incrementColumnValueAvgTime "
> > > "putNumOps "
> > > "putAvgTime "
> > > "getNumOps "
> > > "getAvgTime "
> > > "deleteNumOps "
> > > "deleteAvgTime "
> > > "openScannerNumOps "
> > > "openScannerAvtTime "
> > > "nextNumOps "
> > > "nextAvgTime "
> > > "closeNumOps "
> > > "closeAvgTime "
> > >
> > > Anyone aware of what change happened in this area?
> > >
> > > regards,
> > > Kannan
> > >
> > >
> >
>
RE: RPC statistics from region servers...
Posted by Kannan Muthukkaruppan <Ka...@facebook.com>.
Pushed the hadoop-metrics.properties changes, and it seems better but not quite. [It doesn't look like all the Region Servers are reporting similar set of counters (on some I am seeing "multiput" stuff, although we aren' t), lot of the average times are close to 0.]
So next I looked into the HBASE-2146 patch, and that doesn't appear to be in trunk although at some point it was in trunk earlier.
http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java?p2=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&p1=%2Fhadoop%2Fhbase%2Ftrunk%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhbase%2Fipc%2FHBaseRpcMetrics.java&r1=901402&r2=901401&view=diff&pathrev=901402
Will try applying the patch locally to see if that helps.
Regards,
Kannan
-----Original Message-----
From: Kannan Muthukkaruppan [mailto:Kannan@facebook.com]
Sent: Thursday, July 29, 2010 2:41 PM
To: dev@hbase.apache.org
Subject: RE: RPC statistics from region servers...
Gary,
With some recent restructuring of our conf/* and cluster deploy scripts, we did lose our hadoop-metrics.properties changes, and reverted back to the default one. That could very well report the issue I ran into. I'll report back when I try it out.
Thanks again for your help on this.
Q: Why don't we make the suggested settings in http://hbase.apache.org/docs/current/metrics.html be part of the default conf/hadoop-metrics.properties?
Regards,
Kannan
-----Original Message-----
From: Gary Helmling [mailto:ghelmling@gmail.com]
Sent: Thursday, July 29, 2010 1:08 AM
To: dev@hbase.apache.org
Subject: Re: RPC statistics from region servers...
Hmm, this sounds like HBASE-2146. That patch had added code to initialize
the RPC metrics MBean with a know list of RPC method names maintained in
HBaseRPC.Invocation. That way all per-method metrics would show up on
server startup. I believe that that method name mapping was later removed
though (it can make the RPC serialization brittle, complicating things like
rolling restarts), so maybe the patch that was applied to trunk was later
removed?
The MBean attributes for individual RPC methods _should_ be there after
there has been activity involving those requests against the cluster. If
you start jconsole prior to the cluster activity however, I don't think the
MBean definition will update, so you'll need to restart jconsole for them to
show up.
In addition, you should make sure you're running with a metrics context with
an updater configured in
conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
GangliaContext, etc). And configure rpc.period with the update interval.
These metrics don't show up as live counters in JMX, they're only pushed out
to JMX during the periodic updates.
If you've already configured all of those conditions and the metrics _still_
aren't showing up, then let me know and I'll try to do some testing on trunk
tomorrow.
--gh
On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
<Ka...@facebook.com>wrote:
> > they don't exist until used?
>
> I was doing a whole bunch of puts & gets against the cluster during that
> time. That's what you mean, correct?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Wednesday, July 28, 2010 8:21 PM
> To: dev@hbase.apache.org
> Cc: hbase-dev@hadoop.apache.org
> Subject: Re: RPC statistics from region servers...
>
> I think those are dynamic metrics... they don't exist until used?
>
> On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
> wrote:
> > I was using jconsole to check the stats published by region servers..
> >
> > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
> >
> > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> > and,
> > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
> >
> > I no longer see attributes like:
> > "incrementColumnValueNumOps "
> > "incrementColumnValueAvgTime "
> > "putNumOps "
> > "putAvgTime "
> > "getNumOps "
> > "getAvgTime "
> > "deleteNumOps "
> > "deleteAvgTime "
> > "openScannerNumOps "
> > "openScannerAvtTime "
> > "nextNumOps "
> > "nextAvgTime "
> > "closeNumOps "
> > "closeAvgTime "
> >
> > Anyone aware of what change happened in this area?
> >
> > regards,
> > Kannan
> >
> >
>
RE: RPC statistics from region servers...
Posted by Kannan Muthukkaruppan <Ka...@facebook.com>.
<<With same recent restructuring>>
s/same/some
-----Original Message-----
From: Kannan Muthukkaruppan [mailto:Kannan@facebook.com]
Sent: Thursday, July 29, 2010 2:41 PM
To: dev@hbase.apache.org
Subject: RE: RPC statistics from region servers...
Gary,
With same recent restructuring of our conf/* and cluster deploy scripts, we did lose our hadoop-metrics.properties changes, and reverted back to the default one. That could very well report the issue I ran into. I'll report back when I try it out.
Thanks again for your help on this.
Q: Why don't we make the suggested settings in http://hbase.apache.org/docs/current/metrics.html be part of the default conf/hadoop-metrics.properties?
Regards,
Kannan
-----Original Message-----
From: Gary Helmling [mailto:ghelmling@gmail.com]
Sent: Thursday, July 29, 2010 1:08 AM
To: dev@hbase.apache.org
Subject: Re: RPC statistics from region servers...
Hmm, this sounds like HBASE-2146. That patch had added code to initialize
the RPC metrics MBean with a know list of RPC method names maintained in
HBaseRPC.Invocation. That way all per-method metrics would show up on
server startup. I believe that that method name mapping was later removed
though (it can make the RPC serialization brittle, complicating things like
rolling restarts), so maybe the patch that was applied to trunk was later
removed?
The MBean attributes for individual RPC methods _should_ be there after
there has been activity involving those requests against the cluster. If
you start jconsole prior to the cluster activity however, I don't think the
MBean definition will update, so you'll need to restart jconsole for them to
show up.
In addition, you should make sure you're running with a metrics context with
an updater configured in
conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
GangliaContext, etc). And configure rpc.period with the update interval.
These metrics don't show up as live counters in JMX, they're only pushed out
to JMX during the periodic updates.
If you've already configured all of those conditions and the metrics _still_
aren't showing up, then let me know and I'll try to do some testing on trunk
tomorrow.
--gh
On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
<Ka...@facebook.com>wrote:
> > they don't exist until used?
>
> I was doing a whole bunch of puts & gets against the cluster during that
> time. That's what you mean, correct?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Wednesday, July 28, 2010 8:21 PM
> To: dev@hbase.apache.org
> Cc: hbase-dev@hadoop.apache.org
> Subject: Re: RPC statistics from region servers...
>
> I think those are dynamic metrics... they don't exist until used?
>
> On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
> wrote:
> > I was using jconsole to check the stats published by region servers..
> >
> > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
> >
> > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> > and,
> > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
> >
> > I no longer see attributes like:
> > "incrementColumnValueNumOps "
> > "incrementColumnValueAvgTime "
> > "putNumOps "
> > "putAvgTime "
> > "getNumOps "
> > "getAvgTime "
> > "deleteNumOps "
> > "deleteAvgTime "
> > "openScannerNumOps "
> > "openScannerAvtTime "
> > "nextNumOps "
> > "nextAvgTime "
> > "closeNumOps "
> > "closeAvgTime "
> >
> > Anyone aware of what change happened in this area?
> >
> > regards,
> > Kannan
> >
> >
>
RE: RPC statistics from region servers...
Posted by Kannan Muthukkaruppan <Ka...@facebook.com>.
Gary,
With same recent restructuring of our conf/* and cluster deploy scripts, we did lose our hadoop-metrics.properties changes, and reverted back to the default one. That could very well report the issue I ran into. I'll report back when I try it out.
Thanks again for your help on this.
Q: Why don't we make the suggested settings in http://hbase.apache.org/docs/current/metrics.html be part of the default conf/hadoop-metrics.properties?
Regards,
Kannan
-----Original Message-----
From: Gary Helmling [mailto:ghelmling@gmail.com]
Sent: Thursday, July 29, 2010 1:08 AM
To: dev@hbase.apache.org
Subject: Re: RPC statistics from region servers...
Hmm, this sounds like HBASE-2146. That patch had added code to initialize
the RPC metrics MBean with a know list of RPC method names maintained in
HBaseRPC.Invocation. That way all per-method metrics would show up on
server startup. I believe that that method name mapping was later removed
though (it can make the RPC serialization brittle, complicating things like
rolling restarts), so maybe the patch that was applied to trunk was later
removed?
The MBean attributes for individual RPC methods _should_ be there after
there has been activity involving those requests against the cluster. If
you start jconsole prior to the cluster activity however, I don't think the
MBean definition will update, so you'll need to restart jconsole for them to
show up.
In addition, you should make sure you're running with a metrics context with
an updater configured in
conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
GangliaContext, etc). And configure rpc.period with the update interval.
These metrics don't show up as live counters in JMX, they're only pushed out
to JMX during the periodic updates.
If you've already configured all of those conditions and the metrics _still_
aren't showing up, then let me know and I'll try to do some testing on trunk
tomorrow.
--gh
On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
<Ka...@facebook.com>wrote:
> > they don't exist until used?
>
> I was doing a whole bunch of puts & gets against the cluster during that
> time. That's what you mean, correct?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Wednesday, July 28, 2010 8:21 PM
> To: dev@hbase.apache.org
> Cc: hbase-dev@hadoop.apache.org
> Subject: Re: RPC statistics from region servers...
>
> I think those are dynamic metrics... they don't exist until used?
>
> On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
> wrote:
> > I was using jconsole to check the stats published by region servers..
> >
> > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
> >
> > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> > and,
> > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
> >
> > I no longer see attributes like:
> > "incrementColumnValueNumOps "
> > "incrementColumnValueAvgTime "
> > "putNumOps "
> > "putAvgTime "
> > "getNumOps "
> > "getAvgTime "
> > "deleteNumOps "
> > "deleteAvgTime "
> > "openScannerNumOps "
> > "openScannerAvtTime "
> > "nextNumOps "
> > "nextAvgTime "
> > "closeNumOps "
> > "closeAvgTime "
> >
> > Anyone aware of what change happened in this area?
> >
> > regards,
> > Kannan
> >
> >
>
RE: RPC statistics from region servers...
Posted by Kannan Muthukkaruppan <Ka...@facebook.com>.
Gary,
Thanks for the detailed response. I'll check into all of these and report back.
Regards,
Kannan
-----Original Message-----
From: Gary Helmling [mailto:ghelmling@gmail.com]
Sent: Thursday, July 29, 2010 1:08 AM
To: dev@hbase.apache.org
Subject: Re: RPC statistics from region servers...
Hmm, this sounds like HBASE-2146. That patch had added code to initialize
the RPC metrics MBean with a know list of RPC method names maintained in
HBaseRPC.Invocation. That way all per-method metrics would show up on
server startup. I believe that that method name mapping was later removed
though (it can make the RPC serialization brittle, complicating things like
rolling restarts), so maybe the patch that was applied to trunk was later
removed?
The MBean attributes for individual RPC methods _should_ be there after
there has been activity involving those requests against the cluster. If
you start jconsole prior to the cluster activity however, I don't think the
MBean definition will update, so you'll need to restart jconsole for them to
show up.
In addition, you should make sure you're running with a metrics context with
an updater configured in
conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
GangliaContext, etc). And configure rpc.period with the update interval.
These metrics don't show up as live counters in JMX, they're only pushed out
to JMX during the periodic updates.
If you've already configured all of those conditions and the metrics _still_
aren't showing up, then let me know and I'll try to do some testing on trunk
tomorrow.
--gh
On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
<Ka...@facebook.com>wrote:
> > they don't exist until used?
>
> I was doing a whole bunch of puts & gets against the cluster during that
> time. That's what you mean, correct?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Wednesday, July 28, 2010 8:21 PM
> To: dev@hbase.apache.org
> Cc: hbase-dev@hadoop.apache.org
> Subject: Re: RPC statistics from region servers...
>
> I think those are dynamic metrics... they don't exist until used?
>
> On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
> wrote:
> > I was using jconsole to check the stats published by region servers..
> >
> > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
> >
> > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> > and,
> > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
> >
> > I no longer see attributes like:
> > "incrementColumnValueNumOps "
> > "incrementColumnValueAvgTime "
> > "putNumOps "
> > "putAvgTime "
> > "getNumOps "
> > "getAvgTime "
> > "deleteNumOps "
> > "deleteAvgTime "
> > "openScannerNumOps "
> > "openScannerAvtTime "
> > "nextNumOps "
> > "nextAvgTime "
> > "closeNumOps "
> > "closeAvgTime "
> >
> > Anyone aware of what change happened in this area?
> >
> > regards,
> > Kannan
> >
> >
>
Re: RPC statistics from region servers...
Posted by Gary Helmling <gh...@gmail.com>.
Hmm, this sounds like HBASE-2146. That patch had added code to initialize
the RPC metrics MBean with a know list of RPC method names maintained in
HBaseRPC.Invocation. That way all per-method metrics would show up on
server startup. I believe that that method name mapping was later removed
though (it can make the RPC serialization brittle, complicating things like
rolling restarts), so maybe the patch that was applied to trunk was later
removed?
The MBean attributes for individual RPC methods _should_ be there after
there has been activity involving those requests against the cluster. If
you start jconsole prior to the cluster activity however, I don't think the
MBean definition will update, so you'll need to restart jconsole for them to
show up.
In addition, you should make sure you're running with a metrics context with
an updater configured in
conf/hadoop-metrics.properties (NullContextWithUpdateThread, FileContext,
GangliaContext, etc). And configure rpc.period with the update interval.
These metrics don't show up as live counters in JMX, they're only pushed out
to JMX during the periodic updates.
If you've already configured all of those conditions and the metrics _still_
aren't showing up, then let me know and I'll try to do some testing on trunk
tomorrow.
--gh
On Wed, Jul 28, 2010 at 8:32 PM, Kannan Muthukkaruppan
<Ka...@facebook.com>wrote:
> > they don't exist until used?
>
> I was doing a whole bunch of puts & gets against the cluster during that
> time. That's what you mean, correct?
>
> Regards,
> Kannan
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Wednesday, July 28, 2010 8:21 PM
> To: dev@hbase.apache.org
> Cc: hbase-dev@hadoop.apache.org
> Subject: Re: RPC statistics from region servers...
>
> I think those are dynamic metrics... they don't exist until used?
>
> On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
> wrote:
> > I was using jconsole to check the stats published by region servers..
> >
> > Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
> >
> > RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> > and,
> > RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
> >
> > I no longer see attributes like:
> > "incrementColumnValueNumOps "
> > "incrementColumnValueAvgTime "
> > "putNumOps "
> > "putAvgTime "
> > "getNumOps "
> > "getAvgTime "
> > "deleteNumOps "
> > "deleteAvgTime "
> > "openScannerNumOps "
> > "openScannerAvtTime "
> > "nextNumOps "
> > "nextAvgTime "
> > "closeNumOps "
> > "closeAvgTime "
> >
> > Anyone aware of what change happened in this area?
> >
> > regards,
> > Kannan
> >
> >
>
RE: RPC statistics from region servers...
Posted by Kannan Muthukkaruppan <Ka...@facebook.com>.
> they don't exist until used?
I was doing a whole bunch of puts & gets against the cluster during that time. That's what you mean, correct?
Regards,
Kannan
-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com]
Sent: Wednesday, July 28, 2010 8:21 PM
To: dev@hbase.apache.org
Cc: hbase-dev@hadoop.apache.org
Subject: Re: RPC statistics from region servers...
I think those are dynamic metrics... they don't exist until used?
On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
wrote:
> I was using jconsole to check the stats published by region servers..
>
> Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
>
> RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> and,
> RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
>
> I no longer see attributes like:
> "incrementColumnValueNumOps "
> "incrementColumnValueAvgTime "
> "putNumOps "
> "putAvgTime "
> "getNumOps "
> "getAvgTime "
> "deleteNumOps "
> "deleteAvgTime "
> "openScannerNumOps "
> "openScannerAvtTime "
> "nextNumOps "
> "nextAvgTime "
> "closeNumOps "
> "closeAvgTime "
>
> Anyone aware of what change happened in this area?
>
> regards,
> Kannan
>
>
Re: RPC statistics from region servers...
Posted by Ryan Rawson <ry...@gmail.com>.
I think those are dynamic metrics... they don't exist until used?
On Jul 28, 2010 7:30 PM, "Kannan Muthukkaruppan" <Ka...@facebook.com>
wrote:
> I was using jconsole to check the stats published by region servers..
>
> Under hadoop.HBase.RPCStatistics-60020.Attributes, I only see:
>
> RpcQueueTime* (NumOps,AvgTime,MinTime,MaxTime)
> and,
> RPCProcessingTime* (NumOps,AvgTime,MinTime,MaxTime).
>
> I no longer see attributes like:
> "incrementColumnValueNumOps "
> "incrementColumnValueAvgTime "
> "putNumOps "
> "putAvgTime "
> "getNumOps "
> "getAvgTime "
> "deleteNumOps "
> "deleteAvgTime "
> "openScannerNumOps "
> "openScannerAvtTime "
> "nextNumOps "
> "nextAvgTime "
> "closeNumOps "
> "closeAvgTime "
>
> Anyone aware of what change happened in this area?
>
> regards,
> Kannan
>
>