[jira] Commented: (HBASE-3551) Loaded hfile indexes occupy a good chunk of heap; look into shrinking the amount used and/or evicting unused indices

["Marc Limotte (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005272#comment-13005272 ] 

Marc Limotte commented on HBASE-3551:
-------------------------------------

I understand this better now.  I did some poking around with the HFile tool.  Average key length does seem to be around 150 bytes, as I estimated.
 
For one hfile /hbase/foo/fb820ae7002fc96f78165802a0b05e63/metrics/14129209576094096, metadata is:

avgKeyLen=159, avgValueLen=7, entries=49285512, length=615516343
fileinfoOffset=592314718, dataIndexOffset=592315104, dataIndexCount=131869, metaIndexOffset=0, metaIndexCount=0, totalBytes=8653853680, entryCount=49285512, version=1

Size of index = length - dataIndexOffset = 615516343 - 592315104 = 22mb

Index data per Region Server = 22mb * 180 regions = almost 4gb.  Plus the other column family, so this does seem to add up to the 5 to 6gb of HEAP we are seeing.

# of entries per dataindex entry = 49285512 / 131869 = 374
Times the key size (avg 157 bytes for this file) = 59k (close to the block size of 64k).  So, seems to make sense.

I also looked at the keyvalue pairs using the HFile tool (a section of output is below).

We have a few billion rows (2 - 4 billion).  I haven't done a full row count.

What I didn't understand previously is that it's not 374 rows, but 374 "entries".  An entry means a single column entry and the key is repeated for each column value.  Given our fairly large key, that would add up quickly.

Solutions
1) Increase the hbase block size (I did this and it resolved our situation for now)  
2) Modifying our schema to use smaller keys - perhaps IDs instead of string names.
3) Modifying our schema to have fewer columns - we could combine several related columns into one compound value.
4) An LRU cache for storefile indexes

Given the other options, #4 may not be warranted, so I think we can close this issue.


> Loaded hfile indexes occupy a good chunk of heap; look into shrinking the amount used and/or evicting unused indices
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-3551
>                 URL: https://issues.apache.org/jira/browse/HBASE-3551
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: stack
>
> I hung with a user Marc and we were looking over configs and his cluster profile up on ec2.  One thing we noticed was that his 100+ 1G regions of two families had ~2.5G of heap resident.  We did a bit of math and couldn't get to 2.5G so that needs looking into.  Even still, 2.5G is a bunch of heap to give over to indices (He actually OOME'd when he had his RS heap set to just 3G; we shouldn't OOME, we should just run slower).  It sounds like he needs the indices loaded but still, for some cases we should drop indices for unaccessed files.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira