Cache in hbase for already persisted data and memory limits

[Michał Podsiadłowski <po...@gmail.com>]

Hi All,

again I have few questions about setting hbase on production. Now my
concerns are about memory requirements for dataNode and region server.  How
large should be the limits and what is ( if any ) the "rule of thumb" for
those estimates.

Second question is about hbase architecture - is there any cache for data
that have been already persisted to hdfs?
MemCache which can be found quite easily is only for data waiting to be
flushed to hdfs. If there isn't any in hbase layer are you aware of any in
hdfs layer? This is quite important questions since existence of such a
cache affects estimations for memory limits and performance.

Thanks,
Michael

Re: Cache in hbase for already persisted data and memory limits

[Jean-Daniel Cryans <jd...@apache.org>]

2GB for Datanode, 4 for regionserver or more if you can.

We have a LRU cache inside HBase and you can specify that a family be kept
in memory.

J-D

On Dec 11, 2009 5:39 AM, "Michał Podsiadłowski" <po...@gmail.com>
wrote:

Hi All,

again I have few questions about setting hbase on production. Now my
concerns are about memory requirements for dataNode and region server.  How
large should be the limits and what is ( if any ) the "rule of thumb" for
those estimates.

Second question is about hbase architecture - is there any cache for data
that have been already persisted to hdfs?
MemCache which can be found quite easily is only for data waiting to be
flushed to hdfs. If there isn't any in hbase layer are you aware of any in
hdfs layer? This is quite important questions since existence of such a
cache affects estimations for memory limits and performance.

Thanks,
Michael