increasing block cache size

[Ted Tuttle <te...@mentacapital.com>]

Hello All-

 

We are running 5 node v0.94 cluster on machines w/ 32G of RAM.  I'd like
to utilize more of our memory for block cache.  So I have a couple of
questions.

 

Currently, we are running RSs w/ 16G heap and the following memory
settings:

 

  hfile.block.cache.size=0.25

  hbase.regionserver.global.memstore.upperLimit=0.2

  hbase.regionserver.global.memstore.lowerLimit=0.15

 

Typically overall stats for RS are:

 

  numberOfOnlineRegions=228, usedHeapMB=7870, maxHeapMB=16358

 

Used heap typically goes to ~10G then we get GC back down to about 6-8G.
So by my reckoning we have about 25% of heap unutilized.

 

More detailed, typical RS info:

 

numberOfOnlineRegions=245, numberOfStores=245, numberOfStorefiles=249,
storefileIndexSizeMB=17, rootIndexSizeKB=17688,
totalStaticIndexSizeKB=11910, totalStaticBloomSizeKB=36107,
memstoreSizeMB=361, readRequestsCount=-1829306476,
writeRequestsCount=16153099, compactionQueueSize=0, flushQueueSize=0,
usedHeapMB=11407, maxHeapMB=16358, blockCacheSizeMB=3201.95,
blockCacheFreeMB=887.66, blockCacheCount=3243,
blockCacheHitCount=2932634895, blockCacheMissCount=5223350,
blockCacheEvictedCount=678037, blockCacheHitRatio=99%,
blockCacheHitCachingRatio=99%, hdfsBlocksLocalityIndex=95

 

>From these stats I see my memstores are always <1G or <6% of heap.

 

So here are my questions:

 

1) Are memstore settings above relevant in v0.94 w/ off heap memstores
the default?

2) We have good locality and cache hits.  I'd like to take block cache
size to 50% and increase overall heap to 24G.   Any concerns about this
given stats above?

3) Any other memory recommendations?

 

Thanks,

Ted