You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 陈加俊 <cj...@gmail.com> on 2011/04/15 06:50:07 UTC
heap memory allocation
How does heap memory allocated in regionserver ?
Metrics:
request=0.0, regions=1283, stores=2304, storefiles=1968,
storefileIndexSize=246, memstoreSize=791, compactionQueueSize=0,
usedHeap=5669, maxHeap=11991, blockCacheSize=2095157424,
blockCacheFree=419681872, blockCacheCount=24410, blockCacheHitRatio=92,
fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
maxHeap=11991
usedHeap=5669
blockCacheSize=2095157424
memstoreSize=791
????
Free=maxHeap-usedHeap?
--
Thanks & Best regards
jiajun
Re: heap memory allocation
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Currently bloom filters and hfile indexes are in storefileIndexSize
Why do you want bigger blocks? Just for less index size? Be aware it
may lower random read performance.
J-D
On Fri, Apr 15, 2011 at 8:23 AM, Matt Corgan <mc...@hotpads.com> wrote:
> Some of our servers have 5.2gb hbase heaps with the standard 40% (2gb)
> memstore and 20% (1gb) blockcache. I'm wondering where the indexes and
> bloom filters get counted. Is it in that remaining 2gb, and are the bloom
> filters counted in the storefileIndexSize=566 metric?
>
> The overall data in each server isn't too large on disk (~50gb) after gzip
> compression of 25x, but it's made of long keys with short values so there is
> a lot of metadata. I'm thinking of upping the block size to 256k but
> thought i'd ask how it worked first.
>
> some metrics:
>
> regions=401, stores=401, storefiles=758, storefileIndexSize=566,
> memstoreSize=1793, usedHeap=3658, maxHeap=5183, blockCacheSize=816306600
>
> Thanks for the help,
> Matt
>
> On Fri, Apr 15, 2011 at 10:39 AM, Doug Meil
> <do...@explorysmedical.com>wrote:
>
>> The HBase book has come through some updates recently on these metrics.
>>
>> http://hbase.apache.org/book.html#hbase_metrics
>>
>>
>> -----Original Message-----
>> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
>> Sent: Friday, April 15, 2011 5:04 AM
>> To: user@hbase.apache.org
>> Cc: 陈加俊; hbase-user@hadoop.apache.org
>> Subject: Re: heap memory allocation
>>
>> On Thu, Apr 14, 2011 at 9:50 PM, 陈加俊 <cj...@gmail.com> wrote:
>> > How does heap memory allocated in regionserver ?
>> >
>> > Metrics:
>> >
>> > request=0.0, regions=1283, stores=2304, storefiles=1968,
>> > storefileIndexSize=246, memstoreSize=791, compactionQueueSize=0,
>> > usedHeap=5669, maxHeap=11991, blockCacheSize=2095157424,
>> > blockCacheFree=419681872, blockCacheCount=24410,
>> > blockCacheHitRatio=92, fsReadLatency=0, fsWriteLatency=0,
>> > fsSyncLatency=0
>> >
>> >
>> >
>> > maxHeap=11991
>> > usedHeap=5669
>> >
>>
>> This just the state of the JVM memory bean at the last update time.
>> Looks like you have given hbase 12G. We're using about 5.5 Gig. See
>> here for where we read it from the bean:
>>
>> http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/HRegionServer.html#751
>> These look like they report in MB.
>>
>> > blockCacheSize=2095157424
>> > memstoreSize=791
>> > ????
>>
>> These look like they report in bytes and MB. Should make them all same
>> units. HBase has a cache of hdfs blocks. And memstore is where we write
>> edits too before they get flushed to the filesystem.
>>
>> >
>> > Free=maxHeap-usedHeap?
>> >
>>
>> Read the javadoc on the bean returned by
>> ManagementFactory.getMemoryMXBean() call.
>>
>> St.Ack
>>
>> >
>> > --
>> > Thanks & Best regards
>> > jiajun
>> >
>>
>
Re: heap memory allocation
Posted by Matt Corgan <mc...@hotpads.com>.
J-D - yes, I think that increasing block size by 4x would cut the index size
by 4x, assuming 1 index entry per block. Random reads are not as important
for this table, and tend to cache well. I guess that on the flip-side,
scans might be faster. If I start to see problems, I guess my other option
is to reduce the memstore and blockcache percentages.
Stack - thanks, that's the exact issue.
A lot of the data in this table is very cold archived data, so storing the
indexes and blooms in the block cache where they could get evicted would
make sense. Lars left a comment about compressing the indexes... i could
see using something like an array based binary trie to represent the index
more compactly, while also adding performance. You could even bundle bytes
into int or long "words" for faster traversal.
On Fri, Apr 15, 2011 at 12:47 PM, Stack <st...@duboce.net> wrote:
> On Fri, Apr 15, 2011 at 8:23 AM, Matt Corgan <mc...@hotpads.com> wrote:
> > Some of our servers have 5.2gb hbase heaps with the standard 40% (2gb)
> > memstore and 20% (1gb) blockcache. I'm wondering where the indexes and
> > bloom filters get counted. Is it in that remaining 2gb, and are the
> bloom
> > filters counted in the storefileIndexSize=566 metric?
> >
>
>
> Indexes are the storefileIndexSize metric.
>
> I don't think we count bloom space or rather, I believe they are kept
> in the cache (someone correct me if I'm off).
>
>
> > The overall data in each server isn't too large on disk (~50gb) after
> gzip
> > compression of 25x, but it's made of long keys with short values so there
> is
> > a lot of metadata. I'm thinking of upping the block size to 256k but
> > thought i'd ask how it worked first.
> >
>
> This skew -- long keys and short values -- makes for bigger indices
> for sure. Marc Limotte in an earlier thread takes a look at this
> indexing sizing. At first it seemed like the math was off and then he
> looked at his storefiles and figured that it starts to add up . See
> http://hbase.apache.org/book/keysize.html
>
> St.Ack
>
Re: heap memory allocation
Posted by Stack <st...@duboce.net>.
On Fri, Apr 15, 2011 at 8:23 AM, Matt Corgan <mc...@hotpads.com> wrote:
> Some of our servers have 5.2gb hbase heaps with the standard 40% (2gb)
> memstore and 20% (1gb) blockcache. I'm wondering where the indexes and
> bloom filters get counted. Is it in that remaining 2gb, and are the bloom
> filters counted in the storefileIndexSize=566 metric?
>
Indexes are the storefileIndexSize metric.
I don't think we count bloom space or rather, I believe they are kept
in the cache (someone correct me if I'm off).
> The overall data in each server isn't too large on disk (~50gb) after gzip
> compression of 25x, but it's made of long keys with short values so there is
> a lot of metadata. I'm thinking of upping the block size to 256k but
> thought i'd ask how it worked first.
>
This skew -- long keys and short values -- makes for bigger indices
for sure. Marc Limotte in an earlier thread takes a look at this
indexing sizing. At first it seemed like the math was off and then he
looked at his storefiles and figured that it starts to add up . See
http://hbase.apache.org/book/keysize.html
St.Ack
Re: heap memory allocation
Posted by Matt Corgan <mc...@hotpads.com>.
Some of our servers have 5.2gb hbase heaps with the standard 40% (2gb)
memstore and 20% (1gb) blockcache. I'm wondering where the indexes and
bloom filters get counted. Is it in that remaining 2gb, and are the bloom
filters counted in the storefileIndexSize=566 metric?
The overall data in each server isn't too large on disk (~50gb) after gzip
compression of 25x, but it's made of long keys with short values so there is
a lot of metadata. I'm thinking of upping the block size to 256k but
thought i'd ask how it worked first.
some metrics:
regions=401, stores=401, storefiles=758, storefileIndexSize=566,
memstoreSize=1793, usedHeap=3658, maxHeap=5183, blockCacheSize=816306600
Thanks for the help,
Matt
On Fri, Apr 15, 2011 at 10:39 AM, Doug Meil
<do...@explorysmedical.com>wrote:
> The HBase book has come through some updates recently on these metrics.
>
> http://hbase.apache.org/book.html#hbase_metrics
>
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Friday, April 15, 2011 5:04 AM
> To: user@hbase.apache.org
> Cc: 陈加俊; hbase-user@hadoop.apache.org
> Subject: Re: heap memory allocation
>
> On Thu, Apr 14, 2011 at 9:50 PM, 陈加俊 <cj...@gmail.com> wrote:
> > How does heap memory allocated in regionserver ?
> >
> > Metrics:
> >
> > request=0.0, regions=1283, stores=2304, storefiles=1968,
> > storefileIndexSize=246, memstoreSize=791, compactionQueueSize=0,
> > usedHeap=5669, maxHeap=11991, blockCacheSize=2095157424,
> > blockCacheFree=419681872, blockCacheCount=24410,
> > blockCacheHitRatio=92, fsReadLatency=0, fsWriteLatency=0,
> > fsSyncLatency=0
> >
> >
> >
> > maxHeap=11991
> > usedHeap=5669
> >
>
> This just the state of the JVM memory bean at the last update time.
> Looks like you have given hbase 12G. We're using about 5.5 Gig. See
> here for where we read it from the bean:
>
> http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/HRegionServer.html#751
> These look like they report in MB.
>
> > blockCacheSize=2095157424
> > memstoreSize=791
> > ????
>
> These look like they report in bytes and MB. Should make them all same
> units. HBase has a cache of hdfs blocks. And memstore is where we write
> edits too before they get flushed to the filesystem.
>
> >
> > Free=maxHeap-usedHeap?
> >
>
> Read the javadoc on the bean returned by
> ManagementFactory.getMemoryMXBean() call.
>
> St.Ack
>
> >
> > --
> > Thanks & Best regards
> > jiajun
> >
>
RE: heap memory allocation
Posted by Doug Meil <do...@explorysmedical.com>.
The HBase book has come through some updates recently on these metrics.
http://hbase.apache.org/book.html#hbase_metrics
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Friday, April 15, 2011 5:04 AM
To: user@hbase.apache.org
Cc: 陈加俊; hbase-user@hadoop.apache.org
Subject: Re: heap memory allocation
On Thu, Apr 14, 2011 at 9:50 PM, 陈加俊 <cj...@gmail.com> wrote:
> How does heap memory allocated in regionserver ?
>
> Metrics:
>
> request=0.0, regions=1283, stores=2304, storefiles=1968,
> storefileIndexSize=246, memstoreSize=791, compactionQueueSize=0,
> usedHeap=5669, maxHeap=11991, blockCacheSize=2095157424,
> blockCacheFree=419681872, blockCacheCount=24410,
> blockCacheHitRatio=92, fsReadLatency=0, fsWriteLatency=0,
> fsSyncLatency=0
>
>
>
> maxHeap=11991
> usedHeap=5669
>
This just the state of the JVM memory bean at the last update time.
Looks like you have given hbase 12G. We're using about 5.5 Gig. See
here for where we read it from the bean:
http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/HRegionServer.html#751
These look like they report in MB.
> blockCacheSize=2095157424
> memstoreSize=791
> ????
These look like they report in bytes and MB. Should make them all same units. HBase has a cache of hdfs blocks. And memstore is where we write edits too before they get flushed to the filesystem.
>
> Free=maxHeap-usedHeap?
>
Read the javadoc on the bean returned by
ManagementFactory.getMemoryMXBean() call.
St.Ack
>
> --
> Thanks & Best regards
> jiajun
>
Re: heap memory allocation
Posted by Stack <st...@duboce.net>.
On Thu, Apr 14, 2011 at 9:50 PM, 陈加俊 <cj...@gmail.com> wrote:
> How does heap memory allocated in regionserver ?
>
> Metrics:
>
> request=0.0, regions=1283, stores=2304, storefiles=1968,
> storefileIndexSize=246, memstoreSize=791, compactionQueueSize=0,
> usedHeap=5669, maxHeap=11991, blockCacheSize=2095157424,
> blockCacheFree=419681872, blockCacheCount=24410, blockCacheHitRatio=92,
> fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
>
>
>
> maxHeap=11991
> usedHeap=5669
>
This just the state of the JVM memory bean at the last update time.
Looks like you have given hbase 12G. We're using about 5.5 Gig. See
here for where we read it from the bean:
http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/HRegionServer.html#751
These look like they report in MB.
> blockCacheSize=2095157424
> memstoreSize=791
> ????
These look like they report in bytes and MB. Should make them all
same units. HBase has a cache of hdfs blocks. And memstore is where
we write edits too before they get flushed to the filesystem.
>
> Free=maxHeap-usedHeap?
>
Read the javadoc on the bean returned by
ManagementFactory.getMemoryMXBean() call.
St.Ack
>
> --
> Thanks & Best regards
> jiajun
>