You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by sunweiwei <su...@asiainfo-linkage.com> on 2014/05/26 03:42:31 UTC
答复: 答复: what in regionserver's Used Heap
Hi
This is jmap -histo result.
num #instances #bytes class name
----------------------------------------------
1: 14917882 3396492464 [C
2: 1996994 2118021808 [B
num #instances #bytes class name
----------------------------------------------
1: 14917882 3396492464 [C
2: 1996994 2118021808 [B
3: 43341650 1733666000 java.util.LinkedHashMap$Entry
4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
5: 14446577 924580928 org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
6: 14816939 474142048 java.lang.String
7: 14463510 347124240 org.apache.hadoop.metrics2.lib.MetricsInfoImpl
8: 6380066 255202640 org.apache.hadoop.hbase.KeyValue
9: 7274264 174582336 java.util.concurrent.ConcurrentSkipListMap$Node
10: 3412039 81888936 java.util.concurrent.ConcurrentSkipListMap$Index
11: 9208 28516704 [I
12: 259293 26673528 [Ljava.lang.Object;
13: 424072 20355456 java.nio.HeapByteBuffer
14: 418655 20095440 java.nio.HeapCharBuffer
15: 657703 15784872 org.apache.hadoop.metrics2.impl.MetricGaugeDouble
16: 589926 14158224 org.apache.hadoop.metrics2.impl.MetricCounterLong
17: 522366 12536784 org.apache.hadoop.metrics2.impl.MetricGaugeLong
18: 504238 12101712 java.lang.Double
19: 436872 10484928 com.google.protobuf.LiteralByteString
20: 72414 9990968 <constMethodKlass>
21: 72414 9857616 <methodKlass>
22: 55752 9354400 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
23: 103154 7427088 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto
24: 145924 7004352 org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl
25: 5322 6835120 <constantPoolKlass>
26: 251260 6030240 java.util.ArrayList
27: 94355 5979040 <symbolKlass>
28: 103170 5777520 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue
29: 238244 5717856 java.util.Collections$UnmodifiableRandomAccessList
30: 130150 5206000 java.util.TreeMap$Entry
31: 103160 4951680 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue
Thanks
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2014年5月23日 22:34
收件人: user@hbase.apache.org
主题: Re: 答复: what in regionserver's Used Heap
You can use jmap with -histo option to find out heap consumption:
http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html
Please pastebin jmap output.
Cheers
On Thu, May 22, 2014 at 11:30 PM, sunweiwei <su...@asiainfo-linkage.com>wrote:
> Hi Ted
>
> I have checked regionsever:60030/rs-status#regionMemstoreStats. Memstore
> and Cache Size is very small.
> But Used Heap is very large.
> Is there Any other thing in the Used Heap, and use lots of heaps?
>
> Thanks
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2014年5月23日 11:59
> 收件人: user@hbase.apache.org
> 主题: Re: what in regionserver's Used Heap
>
> On regionsever:60030/rs-status#regionMemstoreStats, you should be able to
> see how much memstore each region uses.
>
> You can also use ganglia, etc to view the metrics.
> See http://hbase.apache.org/book.html#hbase_metrics (15.4.4.3 and
> 15.4.4.7)
>
> Cheers
>
>
> On Thu, May 22, 2014 at 8:39 PM, sunweiwei <sunww@asiainfo-linkage.com
> >wrote:
>
> > Hi
> >
> >
> >
> > I'm using hbase0.96.
> >
> >
> >
> > In hbase web, I see :
> >
> > Max Heap is 16.0 G
> >
> > Memstore Size is 208.6 M
> >
> > Cache Size is 138M
> >
> > Cache Free is 6.2G
> >
> >
> >
> > And Used Heap is 10.2 G.
> >
> >
> >
> > Used Heap is large. I want to know what could be in then Used
> Heap.
> >
> >
> >
> >
> > Thanks.
> >
> >
>
>
Re: 答复: 答复: 答复: what in regionserver's Used Heap
Posted by Ted Yu <yu...@gmail.com>.
Patch v1 has been attached to HBASE-11267
Cheers
On Wed, May 28, 2014 at 7:34 PM, sunweiwei <su...@asiainfo-linkage.com>
wrote:
> Thanks you
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2014年5月29日 6:06
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: 答复: what in regionserver's Used Heap
>
> Weiwei:
> I logged HBASE-11267 for the heap consumption issue you reported.
>
> Cheers
>
>
> On Sun, May 25, 2014 at 6:42 PM, sunweiwei <sunww@asiainfo-linkage.com
> >wrote:
>
> > Hi
> >
> > This is jmap -histo result.
> > num #instances #bytes class name
> > ----------------------------------------------
> > 1: 14917882 3396492464 [C
> > 2: 1996994 2118021808 [B
> >
> > num #instances #bytes class name
> > ----------------------------------------------
> > 1: 14917882 3396492464 [C
> > 2: 1996994 2118021808 [B
> > 3: 43341650 1733666000 java.util.LinkedHashMap$Entry
> > 4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
> > 5: 14446577 924580928
> > org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
> > 6: 14816939 474142048 java.lang.String
> > 7: 14463510 347124240
> > org.apache.hadoop.metrics2.lib.MetricsInfoImpl
> > 8: 6380066 255202640 org.apache.hadoop.hbase.KeyValue
> > 9: 7274264 174582336
> > java.util.concurrent.ConcurrentSkipListMap$Node
> > 10: 3412039 81888936
> > java.util.concurrent.ConcurrentSkipListMap$Index
> > 11: 9208 28516704 [I
> > 12: 259293 26673528 [Ljava.lang.Object;
> > 13: 424072 20355456 java.nio.HeapByteBuffer
> > 14: 418655 20095440 java.nio.HeapCharBuffer
> > 15: 657703 15784872
> > org.apache.hadoop.metrics2.impl.MetricGaugeDouble
> > 16: 589926 14158224
> > org.apache.hadoop.metrics2.impl.MetricCounterLong
> > 17: 522366 12536784
> > org.apache.hadoop.metrics2.impl.MetricGaugeLong
> > 18: 504238 12101712 java.lang.Double
> > 19: 436872 10484928 com.google.protobuf.LiteralByteString
> > 20: 72414 9990968 <constMethodKlass>
> > 21: 72414 9857616 <methodKlass>
> > 22: 55752 9354400
> > [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
> > 23: 103154 7427088
> > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto
> > 24: 145924 7004352
> > org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl
> > 25: 5322 6835120 <constantPoolKlass>
> > 26: 251260 6030240 java.util.ArrayList
> > 27: 94355 5979040 <symbolKlass>
> > 28: 103170 5777520
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue
> > 29: 238244 5717856
> > java.util.Collections$UnmodifiableRandomAccessList
> > 30: 130150 5206000 java.util.TreeMap$Entry
> > 31: 103160 4951680
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue
> >
> >
> > Thanks
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2014年5月23日 22:34
> > 收件人: user@hbase.apache.org
> > 主题: Re: 答复: what in regionserver's Used Heap
> >
> > You can use jmap with -histo option to find out heap consumption:
> > http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html
> >
> > Please pastebin jmap output.
> >
> > Cheers
> >
> >
> > On Thu, May 22, 2014 at 11:30 PM, sunweiwei <sunww@asiainfo-linkage.com
> > >wrote:
> >
> > > Hi Ted
> > >
> > > I have checked regionsever:60030/rs-status#regionMemstoreStats.
> Memstore
> > > and Cache Size is very small.
> > > But Used Heap is very large.
> > > Is there Any other thing in the Used Heap, and use lots of heaps?
> > >
> > > Thanks
> > >
> > > -----邮件原件-----
> > > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > > 发送时间: 2014年5月23日 11:59
> > > 收件人: user@hbase.apache.org
> > > 主题: Re: what in regionserver's Used Heap
> > >
> > > On regionsever:60030/rs-status#regionMemstoreStats, you should be able
> to
> > > see how much memstore each region uses.
> > >
> > > You can also use ganglia, etc to view the metrics.
> > > See http://hbase.apache.org/book.html#hbase_metrics (15.4.4.3 and
> > > 15.4.4.7)
> > >
> > > Cheers
> > >
> > >
> > > On Thu, May 22, 2014 at 8:39 PM, sunweiwei <sunww@asiainfo-linkage.com
> > > >wrote:
> > >
> > > > Hi
> > > >
> > > >
> > > >
> > > > I'm using hbase0.96.
> > > >
> > > >
> > > >
> > > > In hbase web, I see :
> > > >
> > > > Max Heap is 16.0 G
> > > >
> > > > Memstore Size is 208.6 M
> > > >
> > > > Cache Size is 138M
> > > >
> > > > Cache Free is 6.2G
> > > >
> > > >
> > > >
> > > > And Used Heap is 10.2 G.
> > > >
> > > >
> > > >
> > > > Used Heap is large. I want to know what could be in then Used
> > > Heap.
> > > >
> > > >
> > > >
> > > >
> > > > Thanks.
> > > >
> > > >
> > >
> > >
> >
> >
>
>
答复: 答复: 答复: what in regionserver's Used Heap
Posted by sunweiwei <su...@asiainfo-linkage.com>.
Thanks you
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2014年5月29日 6:06
收件人: user@hbase.apache.org
主题: Re: 答复: 答复: what in regionserver's Used Heap
Weiwei:
I logged HBASE-11267 for the heap consumption issue you reported.
Cheers
On Sun, May 25, 2014 at 6:42 PM, sunweiwei <su...@asiainfo-linkage.com>wrote:
> Hi
>
> This is jmap -histo result.
> num #instances #bytes class name
> ----------------------------------------------
> 1: 14917882 3396492464 [C
> 2: 1996994 2118021808 [B
>
> num #instances #bytes class name
> ----------------------------------------------
> 1: 14917882 3396492464 [C
> 2: 1996994 2118021808 [B
> 3: 43341650 1733666000 java.util.LinkedHashMap$Entry
> 4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
> 5: 14446577 924580928
> org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
> 6: 14816939 474142048 java.lang.String
> 7: 14463510 347124240
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl
> 8: 6380066 255202640 org.apache.hadoop.hbase.KeyValue
> 9: 7274264 174582336
> java.util.concurrent.ConcurrentSkipListMap$Node
> 10: 3412039 81888936
> java.util.concurrent.ConcurrentSkipListMap$Index
> 11: 9208 28516704 [I
> 12: 259293 26673528 [Ljava.lang.Object;
> 13: 424072 20355456 java.nio.HeapByteBuffer
> 14: 418655 20095440 java.nio.HeapCharBuffer
> 15: 657703 15784872
> org.apache.hadoop.metrics2.impl.MetricGaugeDouble
> 16: 589926 14158224
> org.apache.hadoop.metrics2.impl.MetricCounterLong
> 17: 522366 12536784
> org.apache.hadoop.metrics2.impl.MetricGaugeLong
> 18: 504238 12101712 java.lang.Double
> 19: 436872 10484928 com.google.protobuf.LiteralByteString
> 20: 72414 9990968 <constMethodKlass>
> 21: 72414 9857616 <methodKlass>
> 22: 55752 9354400
> [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
> 23: 103154 7427088
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto
> 24: 145924 7004352
> org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl
> 25: 5322 6835120 <constantPoolKlass>
> 26: 251260 6030240 java.util.ArrayList
> 27: 94355 5979040 <symbolKlass>
> 28: 103170 5777520
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue
> 29: 238244 5717856
> java.util.Collections$UnmodifiableRandomAccessList
> 30: 130150 5206000 java.util.TreeMap$Entry
> 31: 103160 4951680
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue
>
>
> Thanks
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2014年5月23日 22:34
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: what in regionserver's Used Heap
>
> You can use jmap with -histo option to find out heap consumption:
> http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html
>
> Please pastebin jmap output.
>
> Cheers
>
>
> On Thu, May 22, 2014 at 11:30 PM, sunweiwei <sunww@asiainfo-linkage.com
> >wrote:
>
> > Hi Ted
> >
> > I have checked regionsever:60030/rs-status#regionMemstoreStats. Memstore
> > and Cache Size is very small.
> > But Used Heap is very large.
> > Is there Any other thing in the Used Heap, and use lots of heaps?
> >
> > Thanks
> >
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2014年5月23日 11:59
> > 收件人: user@hbase.apache.org
> > 主题: Re: what in regionserver's Used Heap
> >
> > On regionsever:60030/rs-status#regionMemstoreStats, you should be able to
> > see how much memstore each region uses.
> >
> > You can also use ganglia, etc to view the metrics.
> > See http://hbase.apache.org/book.html#hbase_metrics (15.4.4.3 and
> > 15.4.4.7)
> >
> > Cheers
> >
> >
> > On Thu, May 22, 2014 at 8:39 PM, sunweiwei <sunww@asiainfo-linkage.com
> > >wrote:
> >
> > > Hi
> > >
> > >
> > >
> > > I'm using hbase0.96.
> > >
> > >
> > >
> > > In hbase web, I see :
> > >
> > > Max Heap is 16.0 G
> > >
> > > Memstore Size is 208.6 M
> > >
> > > Cache Size is 138M
> > >
> > > Cache Free is 6.2G
> > >
> > >
> > >
> > > And Used Heap is 10.2 G.
> > >
> > >
> > >
> > > Used Heap is large. I want to know what could be in then Used
> > Heap.
> > >
> > >
> > >
> > >
> > > Thanks.
> > >
> > >
> >
> >
>
>
Re: 答复: 答复: what in regionserver's Used Heap
Posted by Ted Yu <yu...@gmail.com>.
Weiwei:
I logged HBASE-11267 for the heap consumption issue you reported.
Cheers
On Sun, May 25, 2014 at 6:42 PM, sunweiwei <su...@asiainfo-linkage.com>wrote:
> Hi
>
> This is jmap -histo result.
> num #instances #bytes class name
> ----------------------------------------------
> 1: 14917882 3396492464 [C
> 2: 1996994 2118021808 [B
>
> num #instances #bytes class name
> ----------------------------------------------
> 1: 14917882 3396492464 [C
> 2: 1996994 2118021808 [B
> 3: 43341650 1733666000 java.util.LinkedHashMap$Entry
> 4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
> 5: 14446577 924580928
> org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
> 6: 14816939 474142048 java.lang.String
> 7: 14463510 347124240
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl
> 8: 6380066 255202640 org.apache.hadoop.hbase.KeyValue
> 9: 7274264 174582336
> java.util.concurrent.ConcurrentSkipListMap$Node
> 10: 3412039 81888936
> java.util.concurrent.ConcurrentSkipListMap$Index
> 11: 9208 28516704 [I
> 12: 259293 26673528 [Ljava.lang.Object;
> 13: 424072 20355456 java.nio.HeapByteBuffer
> 14: 418655 20095440 java.nio.HeapCharBuffer
> 15: 657703 15784872
> org.apache.hadoop.metrics2.impl.MetricGaugeDouble
> 16: 589926 14158224
> org.apache.hadoop.metrics2.impl.MetricCounterLong
> 17: 522366 12536784
> org.apache.hadoop.metrics2.impl.MetricGaugeLong
> 18: 504238 12101712 java.lang.Double
> 19: 436872 10484928 com.google.protobuf.LiteralByteString
> 20: 72414 9990968 <constMethodKlass>
> 21: 72414 9857616 <methodKlass>
> 22: 55752 9354400
> [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
> 23: 103154 7427088
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto
> 24: 145924 7004352
> org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl
> 25: 5322 6835120 <constantPoolKlass>
> 26: 251260 6030240 java.util.ArrayList
> 27: 94355 5979040 <symbolKlass>
> 28: 103170 5777520
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue
> 29: 238244 5717856
> java.util.Collections$UnmodifiableRandomAccessList
> 30: 130150 5206000 java.util.TreeMap$Entry
> 31: 103160 4951680
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue
>
>
> Thanks
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2014年5月23日 22:34
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: what in regionserver's Used Heap
>
> You can use jmap with -histo option to find out heap consumption:
> http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html
>
> Please pastebin jmap output.
>
> Cheers
>
>
> On Thu, May 22, 2014 at 11:30 PM, sunweiwei <sunww@asiainfo-linkage.com
> >wrote:
>
> > Hi Ted
> >
> > I have checked regionsever:60030/rs-status#regionMemstoreStats. Memstore
> > and Cache Size is very small.
> > But Used Heap is very large.
> > Is there Any other thing in the Used Heap, and use lots of heaps?
> >
> > Thanks
> >
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2014年5月23日 11:59
> > 收件人: user@hbase.apache.org
> > 主题: Re: what in regionserver's Used Heap
> >
> > On regionsever:60030/rs-status#regionMemstoreStats, you should be able to
> > see how much memstore each region uses.
> >
> > You can also use ganglia, etc to view the metrics.
> > See http://hbase.apache.org/book.html#hbase_metrics (15.4.4.3 and
> > 15.4.4.7)
> >
> > Cheers
> >
> >
> > On Thu, May 22, 2014 at 8:39 PM, sunweiwei <sunww@asiainfo-linkage.com
> > >wrote:
> >
> > > Hi
> > >
> > >
> > >
> > > I'm using hbase0.96.
> > >
> > >
> > >
> > > In hbase web, I see :
> > >
> > > Max Heap is 16.0 G
> > >
> > > Memstore Size is 208.6 M
> > >
> > > Cache Size is 138M
> > >
> > > Cache Free is 6.2G
> > >
> > >
> > >
> > > And Used Heap is 10.2 G.
> > >
> > >
> > >
> > > Used Heap is large. I want to know what could be in then Used
> > Heap.
> > >
> > >
> > >
> > >
> > > Thanks.
> > >
> > >
> >
> >
>
>
答复: 答复: 答复: 答复: what in regionserver's Used Heap
Posted by sunweiwei <su...@asiainfo-linkage.com>.
This is hadoop-metrics2-hbase.properties.
hbase.extendedperiod = 3600
# Configuration of the "hbase" context for ganglia
# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)
# hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext
hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
hbase.period=10
hbase.servers=hadoop01:8663
# Configuration of the "jvm" context for ganglia
# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)
# jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
jvm.period=10
jvm.servers=hadoop01:8663
# Configuration of the "rpc" context for ganglia
# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter)
# rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
rpc.period=10
rpc.servers=hadoop01:8663
#Ganglia following hadoop example
hbase.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31
hbase.sink.ganglia.period=10
# default for supportsparse is false
*.sink.ganglia.supportsparse=true
.sink.ganglia.slope=jvm.metrics.gcCount=zero,jvm.metrics.memHeapUsedM=both
.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40
hbase.sink.ganglia.servers=hadoop01:8663
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2014年5月26日 22:57
收件人: user@hbase.apache.org
主题: Re: 答复: 答复: 答复: what in regionserver's Used Heap
Can you pastebin the contents of conf/hadoop-metrics2-hbase.properties ?
Thanks
On Sun, May 25, 2014 at 11:37 PM, sunweiwei <su...@asiainfo-linkage.com>wrote:
> Yes, I'm using metrics2 with ganglia.
> And I have closed ganglia, because there is too many region metrics.
> But I don't delete metrics2 configuration in
> $HBASE_HOME/conf/hadoop-metrics2-hbase.properties. Is this OK?
>
>
> Thanks
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2014年5月26日 13:10
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: 答复: what in regionserver's Used Heap
>
> Some metrics2 classes show up in the histogram.
>
> I am digging into these classes and polling developers who are more
> familiar with metrics2 system.
>
> FYI
>
>
> On Sun, May 25, 2014 at 6:42 PM, sunweiwei <sunww@asiainfo-linkage.com
> >wrote:
>
> > Hi
> >
> > This is jmap -histo result.
> > num #instances #bytes class name
> > ----------------------------------------------
> > 1: 14917882 3396492464 [C
> > 2: 1996994 2118021808 [B
> >
> > num #instances #bytes class name
> > ----------------------------------------------
> > 1: 14917882 3396492464 [C
> > 2: 1996994 2118021808 [B
> > 3: 43341650 1733666000 java.util.LinkedHashMap$Entry
> > 4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
> > 5: 14446577 924580928
> > org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
> > 6: 14816939 474142048 java.lang.String
> > 7: 14463510 347124240
> > org.apache.hadoop.metrics2.lib.MetricsInfoImpl
> > 8: 6380066 255202640 org.apache.hadoop.hbase.KeyValue
> > 9: 7274264 174582336
> > java.util.concurrent.ConcurrentSkipListMap$Node
> > 10: 3412039 81888936
> > java.util.concurrent.ConcurrentSkipListMap$Index
> > 11: 9208 28516704 [I
> > 12: 259293 26673528 [Ljava.lang.Object;
> > 13: 424072 20355456 java.nio.HeapByteBuffer
> > 14: 418655 20095440 java.nio.HeapCharBuffer
> > 15: 657703 15784872
> > org.apache.hadoop.metrics2.impl.MetricGaugeDouble
> > 16: 589926 14158224
> > org.apache.hadoop.metrics2.impl.MetricCounterLong
> > 17: 522366 12536784
> > org.apache.hadoop.metrics2.impl.MetricGaugeLong
> > 18: 504238 12101712 java.lang.Double
> > 19: 436872 10484928 com.google.protobuf.LiteralByteString
> > 20: 72414 9990968 <constMethodKlass>
> > 21: 72414 9857616 <methodKlass>
> > 22: 55752 9354400
> > [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
> > 23: 103154 7427088
> > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto
> > 24: 145924 7004352
> > org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl
> > 25: 5322 6835120 <constantPoolKlass>
> > 26: 251260 6030240 java.util.ArrayList
> > 27: 94355 5979040 <symbolKlass>
> > 28: 103170 5777520
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue
> > 29: 238244 5717856
> > java.util.Collections$UnmodifiableRandomAccessList
> > 30: 130150 5206000 java.util.TreeMap$Entry
> > 31: 103160 4951680
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue
> >
> >
> > Thanks
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2014年5月23日 22:34
> > 收件人: user@hbase.apache.org
> > 主题: Re: 答复: what in regionserver's Used Heap
> >
> > You can use jmap with -histo option to find out heap consumption:
> > http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html
> >
> > Please pastebin jmap output.
> >
> > Cheers
> >
> >
> > On Thu, May 22, 2014 at 11:30 PM, sunweiwei <sunww@asiainfo-linkage.com
> > >wrote:
> >
> > > Hi Ted
> > >
> > > I have checked regionsever:60030/rs-status#regionMemstoreStats.
> Memstore
> > > and Cache Size is very small.
> > > But Used Heap is very large.
> > > Is there Any other thing in the Used Heap, and use lots of heaps?
> > >
> > > Thanks
> > >
> > > -----邮件原件-----
> > > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > > 发送时间: 2014年5月23日 11:59
> > > 收件人: user@hbase.apache.org
> > > 主题: Re: what in regionserver's Used Heap
> > >
> > > On regionsever:60030/rs-status#regionMemstoreStats, you should be able
> to
> > > see how much memstore each region uses.
> > >
> > > You can also use ganglia, etc to view the metrics.
> > > See http://hbase.apache.org/book.html#hbase_metrics (15.4.4.3 and
> > > 15.4.4.7)
> > >
> > > Cheers
> > >
> > >
> > > On Thu, May 22, 2014 at 8:39 PM, sunweiwei <sunww@asiainfo-linkage.com
> > > >wrote:
> > >
> > > > Hi
> > > >
> > > >
> > > >
> > > > I'm using hbase0.96.
> > > >
> > > >
> > > >
> > > > In hbase web, I see :
> > > >
> > > > Max Heap is 16.0 G
> > > >
> > > > Memstore Size is 208.6 M
> > > >
> > > > Cache Size is 138M
> > > >
> > > > Cache Free is 6.2G
> > > >
> > > >
> > > >
> > > > And Used Heap is 10.2 G.
> > > >
> > > >
> > > >
> > > > Used Heap is large. I want to know what could be in then Used
> > > Heap.
> > > >
> > > >
> > > >
> > > >
> > > > Thanks.
> > > >
> > > >
> > >
> > >
> >
> >
>
>
Re: 答复: 答复: 答复: what in regionserver's Used Heap
Posted by Ted Yu <yu...@gmail.com>.
Can you pastebin the contents of conf/hadoop-metrics2-hbase.properties ?
Thanks
On Sun, May 25, 2014 at 11:37 PM, sunweiwei <su...@asiainfo-linkage.com>wrote:
> Yes, I'm using metrics2 with ganglia.
> And I have closed ganglia, because there is too many region metrics.
> But I don't delete metrics2 configuration in
> $HBASE_HOME/conf/hadoop-metrics2-hbase.properties. Is this OK?
>
>
> Thanks
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2014年5月26日 13:10
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: 答复: what in regionserver's Used Heap
>
> Some metrics2 classes show up in the histogram.
>
> I am digging into these classes and polling developers who are more
> familiar with metrics2 system.
>
> FYI
>
>
> On Sun, May 25, 2014 at 6:42 PM, sunweiwei <sunww@asiainfo-linkage.com
> >wrote:
>
> > Hi
> >
> > This is jmap -histo result.
> > num #instances #bytes class name
> > ----------------------------------------------
> > 1: 14917882 3396492464 [C
> > 2: 1996994 2118021808 [B
> >
> > num #instances #bytes class name
> > ----------------------------------------------
> > 1: 14917882 3396492464 [C
> > 2: 1996994 2118021808 [B
> > 3: 43341650 1733666000 java.util.LinkedHashMap$Entry
> > 4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
> > 5: 14446577 924580928
> > org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
> > 6: 14816939 474142048 java.lang.String
> > 7: 14463510 347124240
> > org.apache.hadoop.metrics2.lib.MetricsInfoImpl
> > 8: 6380066 255202640 org.apache.hadoop.hbase.KeyValue
> > 9: 7274264 174582336
> > java.util.concurrent.ConcurrentSkipListMap$Node
> > 10: 3412039 81888936
> > java.util.concurrent.ConcurrentSkipListMap$Index
> > 11: 9208 28516704 [I
> > 12: 259293 26673528 [Ljava.lang.Object;
> > 13: 424072 20355456 java.nio.HeapByteBuffer
> > 14: 418655 20095440 java.nio.HeapCharBuffer
> > 15: 657703 15784872
> > org.apache.hadoop.metrics2.impl.MetricGaugeDouble
> > 16: 589926 14158224
> > org.apache.hadoop.metrics2.impl.MetricCounterLong
> > 17: 522366 12536784
> > org.apache.hadoop.metrics2.impl.MetricGaugeLong
> > 18: 504238 12101712 java.lang.Double
> > 19: 436872 10484928 com.google.protobuf.LiteralByteString
> > 20: 72414 9990968 <constMethodKlass>
> > 21: 72414 9857616 <methodKlass>
> > 22: 55752 9354400
> > [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
> > 23: 103154 7427088
> > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto
> > 24: 145924 7004352
> > org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl
> > 25: 5322 6835120 <constantPoolKlass>
> > 26: 251260 6030240 java.util.ArrayList
> > 27: 94355 5979040 <symbolKlass>
> > 28: 103170 5777520
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue
> > 29: 238244 5717856
> > java.util.Collections$UnmodifiableRandomAccessList
> > 30: 130150 5206000 java.util.TreeMap$Entry
> > 31: 103160 4951680
> >
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue
> >
> >
> > Thanks
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2014年5月23日 22:34
> > 收件人: user@hbase.apache.org
> > 主题: Re: 答复: what in regionserver's Used Heap
> >
> > You can use jmap with -histo option to find out heap consumption:
> > http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html
> >
> > Please pastebin jmap output.
> >
> > Cheers
> >
> >
> > On Thu, May 22, 2014 at 11:30 PM, sunweiwei <sunww@asiainfo-linkage.com
> > >wrote:
> >
> > > Hi Ted
> > >
> > > I have checked regionsever:60030/rs-status#regionMemstoreStats.
> Memstore
> > > and Cache Size is very small.
> > > But Used Heap is very large.
> > > Is there Any other thing in the Used Heap, and use lots of heaps?
> > >
> > > Thanks
> > >
> > > -----邮件原件-----
> > > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > > 发送时间: 2014年5月23日 11:59
> > > 收件人: user@hbase.apache.org
> > > 主题: Re: what in regionserver's Used Heap
> > >
> > > On regionsever:60030/rs-status#regionMemstoreStats, you should be able
> to
> > > see how much memstore each region uses.
> > >
> > > You can also use ganglia, etc to view the metrics.
> > > See http://hbase.apache.org/book.html#hbase_metrics (15.4.4.3 and
> > > 15.4.4.7)
> > >
> > > Cheers
> > >
> > >
> > > On Thu, May 22, 2014 at 8:39 PM, sunweiwei <sunww@asiainfo-linkage.com
> > > >wrote:
> > >
> > > > Hi
> > > >
> > > >
> > > >
> > > > I'm using hbase0.96.
> > > >
> > > >
> > > >
> > > > In hbase web, I see :
> > > >
> > > > Max Heap is 16.0 G
> > > >
> > > > Memstore Size is 208.6 M
> > > >
> > > > Cache Size is 138M
> > > >
> > > > Cache Free is 6.2G
> > > >
> > > >
> > > >
> > > > And Used Heap is 10.2 G.
> > > >
> > > >
> > > >
> > > > Used Heap is large. I want to know what could be in then Used
> > > Heap.
> > > >
> > > >
> > > >
> > > >
> > > > Thanks.
> > > >
> > > >
> > >
> > >
> >
> >
>
>
答复: 答复: 答复: what in regionserver's Used Heap
Posted by sunweiwei <su...@asiainfo-linkage.com>.
Yes, I'm using metrics2 with ganglia.
And I have closed ganglia, because there is too many region metrics.
But I don't delete metrics2 configuration in $HBASE_HOME/conf/hadoop-metrics2-hbase.properties. Is this OK?
Thanks
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2014年5月26日 13:10
收件人: user@hbase.apache.org
主题: Re: 答复: 答复: what in regionserver's Used Heap
Some metrics2 classes show up in the histogram.
I am digging into these classes and polling developers who are more
familiar with metrics2 system.
FYI
On Sun, May 25, 2014 at 6:42 PM, sunweiwei <su...@asiainfo-linkage.com>wrote:
> Hi
>
> This is jmap -histo result.
> num #instances #bytes class name
> ----------------------------------------------
> 1: 14917882 3396492464 [C
> 2: 1996994 2118021808 [B
>
> num #instances #bytes class name
> ----------------------------------------------
> 1: 14917882 3396492464 [C
> 2: 1996994 2118021808 [B
> 3: 43341650 1733666000 java.util.LinkedHashMap$Entry
> 4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
> 5: 14446577 924580928
> org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
> 6: 14816939 474142048 java.lang.String
> 7: 14463510 347124240
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl
> 8: 6380066 255202640 org.apache.hadoop.hbase.KeyValue
> 9: 7274264 174582336
> java.util.concurrent.ConcurrentSkipListMap$Node
> 10: 3412039 81888936
> java.util.concurrent.ConcurrentSkipListMap$Index
> 11: 9208 28516704 [I
> 12: 259293 26673528 [Ljava.lang.Object;
> 13: 424072 20355456 java.nio.HeapByteBuffer
> 14: 418655 20095440 java.nio.HeapCharBuffer
> 15: 657703 15784872
> org.apache.hadoop.metrics2.impl.MetricGaugeDouble
> 16: 589926 14158224
> org.apache.hadoop.metrics2.impl.MetricCounterLong
> 17: 522366 12536784
> org.apache.hadoop.metrics2.impl.MetricGaugeLong
> 18: 504238 12101712 java.lang.Double
> 19: 436872 10484928 com.google.protobuf.LiteralByteString
> 20: 72414 9990968 <constMethodKlass>
> 21: 72414 9857616 <methodKlass>
> 22: 55752 9354400
> [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
> 23: 103154 7427088
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto
> 24: 145924 7004352
> org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl
> 25: 5322 6835120 <constantPoolKlass>
> 26: 251260 6030240 java.util.ArrayList
> 27: 94355 5979040 <symbolKlass>
> 28: 103170 5777520
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue
> 29: 238244 5717856
> java.util.Collections$UnmodifiableRandomAccessList
> 30: 130150 5206000 java.util.TreeMap$Entry
> 31: 103160 4951680
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue
>
>
> Thanks
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2014年5月23日 22:34
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: what in regionserver's Used Heap
>
> You can use jmap with -histo option to find out heap consumption:
> http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html
>
> Please pastebin jmap output.
>
> Cheers
>
>
> On Thu, May 22, 2014 at 11:30 PM, sunweiwei <sunww@asiainfo-linkage.com
> >wrote:
>
> > Hi Ted
> >
> > I have checked regionsever:60030/rs-status#regionMemstoreStats. Memstore
> > and Cache Size is very small.
> > But Used Heap is very large.
> > Is there Any other thing in the Used Heap, and use lots of heaps?
> >
> > Thanks
> >
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2014年5月23日 11:59
> > 收件人: user@hbase.apache.org
> > 主题: Re: what in regionserver's Used Heap
> >
> > On regionsever:60030/rs-status#regionMemstoreStats, you should be able to
> > see how much memstore each region uses.
> >
> > You can also use ganglia, etc to view the metrics.
> > See http://hbase.apache.org/book.html#hbase_metrics (15.4.4.3 and
> > 15.4.4.7)
> >
> > Cheers
> >
> >
> > On Thu, May 22, 2014 at 8:39 PM, sunweiwei <sunww@asiainfo-linkage.com
> > >wrote:
> >
> > > Hi
> > >
> > >
> > >
> > > I'm using hbase0.96.
> > >
> > >
> > >
> > > In hbase web, I see :
> > >
> > > Max Heap is 16.0 G
> > >
> > > Memstore Size is 208.6 M
> > >
> > > Cache Size is 138M
> > >
> > > Cache Free is 6.2G
> > >
> > >
> > >
> > > And Used Heap is 10.2 G.
> > >
> > >
> > >
> > > Used Heap is large. I want to know what could be in then Used
> > Heap.
> > >
> > >
> > >
> > >
> > > Thanks.
> > >
> > >
> >
> >
>
>
Re: 答复: 答复: what in regionserver's Used Heap
Posted by Ted Yu <yu...@gmail.com>.
Some metrics2 classes show up in the histogram.
I am digging into these classes and polling developers who are more
familiar with metrics2 system.
FYI
On Sun, May 25, 2014 at 6:42 PM, sunweiwei <su...@asiainfo-linkage.com>wrote:
> Hi
>
> This is jmap -histo result.
> num #instances #bytes class name
> ----------------------------------------------
> 1: 14917882 3396492464 [C
> 2: 1996994 2118021808 [B
>
> num #instances #bytes class name
> ----------------------------------------------
> 1: 14917882 3396492464 [C
> 2: 1996994 2118021808 [B
> 3: 43341650 1733666000 java.util.LinkedHashMap$Entry
> 4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
> 5: 14446577 924580928
> org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
> 6: 14816939 474142048 java.lang.String
> 7: 14463510 347124240
> org.apache.hadoop.metrics2.lib.MetricsInfoImpl
> 8: 6380066 255202640 org.apache.hadoop.hbase.KeyValue
> 9: 7274264 174582336
> java.util.concurrent.ConcurrentSkipListMap$Node
> 10: 3412039 81888936
> java.util.concurrent.ConcurrentSkipListMap$Index
> 11: 9208 28516704 [I
> 12: 259293 26673528 [Ljava.lang.Object;
> 13: 424072 20355456 java.nio.HeapByteBuffer
> 14: 418655 20095440 java.nio.HeapCharBuffer
> 15: 657703 15784872
> org.apache.hadoop.metrics2.impl.MetricGaugeDouble
> 16: 589926 14158224
> org.apache.hadoop.metrics2.impl.MetricCounterLong
> 17: 522366 12536784
> org.apache.hadoop.metrics2.impl.MetricGaugeLong
> 18: 504238 12101712 java.lang.Double
> 19: 436872 10484928 com.google.protobuf.LiteralByteString
> 20: 72414 9990968 <constMethodKlass>
> 21: 72414 9857616 <methodKlass>
> 22: 55752 9354400
> [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
> 23: 103154 7427088
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto
> 24: 145924 7004352
> org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl
> 25: 5322 6835120 <constantPoolKlass>
> 26: 251260 6030240 java.util.ArrayList
> 27: 94355 5979040 <symbolKlass>
> 28: 103170 5777520
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue$QualifierValue
> 29: 238244 5717856
> java.util.Collections$UnmodifiableRandomAccessList
> 30: 130150 5206000 java.util.TreeMap$Entry
> 31: 103160 4951680
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MutationProto$ColumnValue
>
>
> Thanks
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2014年5月23日 22:34
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: what in regionserver's Used Heap
>
> You can use jmap with -histo option to find out heap consumption:
> http://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html
>
> Please pastebin jmap output.
>
> Cheers
>
>
> On Thu, May 22, 2014 at 11:30 PM, sunweiwei <sunww@asiainfo-linkage.com
> >wrote:
>
> > Hi Ted
> >
> > I have checked regionsever:60030/rs-status#regionMemstoreStats. Memstore
> > and Cache Size is very small.
> > But Used Heap is very large.
> > Is there Any other thing in the Used Heap, and use lots of heaps?
> >
> > Thanks
> >
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2014年5月23日 11:59
> > 收件人: user@hbase.apache.org
> > 主题: Re: what in regionserver's Used Heap
> >
> > On regionsever:60030/rs-status#regionMemstoreStats, you should be able to
> > see how much memstore each region uses.
> >
> > You can also use ganglia, etc to view the metrics.
> > See http://hbase.apache.org/book.html#hbase_metrics (15.4.4.3 and
> > 15.4.4.7)
> >
> > Cheers
> >
> >
> > On Thu, May 22, 2014 at 8:39 PM, sunweiwei <sunww@asiainfo-linkage.com
> > >wrote:
> >
> > > Hi
> > >
> > >
> > >
> > > I'm using hbase0.96.
> > >
> > >
> > >
> > > In hbase web, I see :
> > >
> > > Max Heap is 16.0 G
> > >
> > > Memstore Size is 208.6 M
> > >
> > > Cache Size is 138M
> > >
> > > Cache Free is 6.2G
> > >
> > >
> > >
> > > And Used Heap is 10.2 G.
> > >
> > >
> > >
> > > Used Heap is large. I want to know what could be in then Used
> > Heap.
> > >
> > >
> > >
> > >
> > > Thanks.
> > >
> > >
> >
> >
>
>