hbase memstore size

[Ozhan Gulen <oz...@gmail.com>]

Hello,
In our hbase cluster memstore flush size is 128 mb. And to insert data to
tables, we only use bulk load tool. Since bulk loading bypasses memstores,
they are never used, so we want to minimize memstore flush size. But
memstore flush size is used in many important calculations in hbase such
that;

region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
“hbase.hregion.max.filesize”)

So setting memstore value smaller or "0" for example,  results in some
other problems.
What do you suggest us in that case. Setting memstore size to 128 holds
some memory for tens of regions in region server and we want to get rid of
it.
Thanks a lot.

ozhan

Re: hbase memstore size

[lars hofhansl <la...@apache.org>]

Also...

If you want to use the ram for the block cache (as you should)

Reduce:
hbase.regionserver.global.memstore.upperLimit from 0.4 to 0.1
and
hbase.regionserver.global.memstore.lowerLimit from 0.35 to 0.1
(you can reduce these two even low if you want)

And then increase:
hfile.block.cache.size from 0.25 (0.94), or 0.4 (0.98) to 0.8

-- Lars



________________________________
 From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org> 
Sent: Sunday, August 3, 2014 5:42 PM
Subject: Re: hbase memstore size
 

If you never use the memstore, you do not need to reduce its size. Memory for the memstore isn't allocated unless you use it.

Also the formula is Min( R^3 * 2 * "hbase.hregion.memstore.flush.size”, “hbase.hregion.max.filesize”) now :)



________________________________

From: Ozhan Gulen <oz...@gmail.com>
To: dev@hbase.apache.org 
Sent: Sunday, August 3, 2014 2:18 AM
Subject: hbase memstore size


Hello,
In our hbase cluster memstore flush size is 128 mb. And to insert data to
tables, we only use bulk load tool. Since bulk loading bypasses memstores,
they are never used, so we want to minimize memstore flush size. But
memstore flush size is used in many important calculations in hbase such
that;

region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
“hbase.hregion.max.filesize”)

So setting memstore value smaller or "0" for example,  results in some
other problems.
What do you suggest us in that case. Setting memstore size to 128 holds
some memory for tens of regions in region server and we want to get rid of
it.
Thanks a lot.

ozhan

Re: hbase memstore size

[lars hofhansl <la...@apache.org>]

If you never use the memstore, you do not need to reduce its size. Memory for the memstore isn't allocated unless you use it.

Also the formula is Min( R^3 * 2 * "hbase.hregion.memstore.flush.size”, “hbase.hregion.max.filesize”) now :)



________________________________
 From: Ozhan Gulen <oz...@gmail.com>
To: dev@hbase.apache.org 
Sent: Sunday, August 3, 2014 2:18 AM
Subject: hbase memstore size
 

Hello,
In our hbase cluster memstore flush size is 128 mb. And to insert data to
tables, we only use bulk load tool. Since bulk loading bypasses memstores,
they are never used, so we want to minimize memstore flush size. But
memstore flush size is used in many important calculations in hbase such
that;

region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
“hbase.hregion.max.filesize”)

So setting memstore value smaller or "0" for example,  results in some
other problems.
What do you suggest us in that case. Setting memstore size to 128 holds
some memory for tens of regions in region server and we want to get rid of
it.
Thanks a lot.

ozhan