You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Subash Kunjupillai <su...@ericsson.com> on 2017/10/04 16:42:22 UTC
Hbase regionserver heap occupancy because of MSLAB
As sited in HBase Doc
<http://archive-primary.cloudera.com/cdh5/cdh/5/hbase-0.98.1-cdh5.1.3/book/regions.arch.html#too_many_regions>
, I understand that MSLAB of each memstore will occupy 2MB(default) of the
region server heap memory. So if there are 2000 regions in my region server,
3.9GB of my heap would be occupied.
My question is, memory for MSLAB will be allocated as soon as the regions
come online or only if data is written to memstore? Because I've a cluster
where 1000 of empty regions in a table, since data are removed after
configured TTL. Would like to know would that cause unnecessary heap usage?
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
Re: Hbase regionserver heap occupancy because of MSLAB
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Hi
If there is MSLAB enabled - one is with pool and another is without Pool.
I hope you are using MSLAB without pool in this case. So only when ever
there are writes that happen then we create 2MB chunks and write data to
them. As and when the chunk is full we keep creating new chunks.
So when a memstore snapshot happens then we flush out all those chunks and
again we keep creating chunks to continue with writes.
So in your case when there are empty regions - ya there won't be any chunk
being created.
>>since data are removed after
configured TTL. Would like to know would that cause unnecessary heap usage?
If there is an active write that is happening then the last chunk would be
around till that is flushed and cleared off. How long is the TTL? Is it few
seconds or minutes where TTL expiry can even happen on the memstore data
itself?
Regards
Ram
On Thu, Oct 5, 2017 at 9:25 AM, Stack <st...@duboce.net> wrote:
> On Wed, Oct 4, 2017 at 9:42 AM, Subash Kunjupillai <su...@ericsson.com>
> wrote:
>
> > As sited in HBase Doc
> > <http://archive-primary.cloudera.com/cdh5/cdh/5/hbase-0.98.1
> > -cdh5.1.3/book/regions.arch.html#too_many_regions>
> > , I understand that MSLAB of each memstore will occupy 2MB(default) of
> the
> > region server heap memory. So if there are 2000 regions in my region
> > server,
> > 3.9GB of my heap would be occupied.
> >
> > My question is, memory for MSLAB will be allocated as soon as the regions
> > come online or only if data is written to memstore?
>
>
> It looks like the latter, lazy allocation on first Cell add.
> See HeapMemStoreLAB alloc called out of DefaultMemStore.
>
> If up for experimenting, you could try running without MSLAB with G1GC
> enabled; various report G1GC mitigates need for MSLAB (I've not tried this
> myself).
>
>
>
> > Because I've a cluster
> > where 1000 of empty regions in a table, since data are removed after
> > configured TTL. Would like to know would that cause unnecessary heap
> usage?
> >
> >
> >
> As I read it, it would; the last Chunk allocated in DefaultMemStore would
> stick around.
>
> St.Ack
>
>
>
>
> >
> >
> > --
> > Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416
> > .html
> >
>
Re: Hbase regionserver heap occupancy because of MSLAB
Posted by Stack <st...@duboce.net>.
On Wed, Oct 4, 2017 at 9:42 AM, Subash Kunjupillai <su...@ericsson.com>
wrote:
> As sited in HBase Doc
> <http://archive-primary.cloudera.com/cdh5/cdh/5/hbase-0.98.1
> -cdh5.1.3/book/regions.arch.html#too_many_regions>
> , I understand that MSLAB of each memstore will occupy 2MB(default) of the
> region server heap memory. So if there are 2000 regions in my region
> server,
> 3.9GB of my heap would be occupied.
>
> My question is, memory for MSLAB will be allocated as soon as the regions
> come online or only if data is written to memstore?
It looks like the latter, lazy allocation on first Cell add.
See HeapMemStoreLAB alloc called out of DefaultMemStore.
If up for experimenting, you could try running without MSLAB with G1GC
enabled; various report G1GC mitigates need for MSLAB (I've not tried this
myself).
> Because I've a cluster
> where 1000 of empty regions in a table, since data are removed after
> configured TTL. Would like to know would that cause unnecessary heap usage?
>
>
>
As I read it, it would; the last Chunk allocated in DefaultMemStore would
stick around.
St.Ack
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416
> .html
>
Re: Hbase regionserver heap occupancy because of MSLAB
Posted by Subash Kunjupillai <su...@ericsson.com>.
Hi Ted,I would love to explore that option, I see this feature is available
from HBase 1.2 version, but right now we are planning for release of our
product with HBase 1.1.8 :(
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
Re: Hbase regionserver heap occupancy because of MSLAB
Posted by Ted Yu <yu...@gmail.com>.
For the last question, have you considered using SimpleRegionNormalizer ?
Take a look at its javadoc for how it works.
FYI
On Wed, Oct 4, 2017 at 9:42 AM, Subash Kunjupillai <su...@ericsson.com>
wrote:
> As sited in HBase Doc
> <http://archive-primary.cloudera.com/cdh5/cdh/5/hbase-
> 0.98.1-cdh5.1.3/book/regions.arch.html#too_many_regions>
> , I understand that MSLAB of each memstore will occupy 2MB(default) of the
> region server heap memory. So if there are 2000 regions in my region
> server,
> 3.9GB of my heap would be occupied.
>
> My question is, memory for MSLAB will be allocated as soon as the regions
> come online or only if data is written to memstore? Because I've a cluster
> where 1000 of empty regions in a table, since data are removed after
> configured TTL. Would like to know would that cause unnecessary heap usage?
>
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html
>