How to avoid update blocking

[Manjeet Singh <ma...@gmail.com>]

How to avoid read/write update blocking in hbase and on what condition
Hbase block read/write?



what should the frequency of Major compaction ( I have daily data insert on
average is 18 GB and 48 GB max, its with SNAPPY and FAST_DIFF)

 a. should it be on daily basis?

 b. should it be in weekends?

 c. does Major compaction on daily basis will avoid blocking?


What is the calculation? Right now its on weekend when load is low.

we have :
total 7 (3+4) data nodes
3 80GB Ram 1.8TB HDD
4 125 GB Ram HDD 6 TB

below are few important configurations:

hbase.regionserver.global.memstore.lowerLimit = .38
hbase.regionserver.global.memstore.upperLimit = .4
hbase.hregion.memstore.block.multiplier=2
hbase.hregion.majorcompaction=0 ( scheduled on weekend)
hbase.hregion.memstore.flush.size 128 MB
hbase.hregion.preclose.flush.size 5 MB
hbase.hstore.compactionThreshold 3

-- 
luv all

Re: How to avoid update blocking

[Anoop John <an...@gmail.com>]

The blocking happens in write path only.  While reading there is no
blocking as such.  Are you speaking abt throttling?

In write path the blocking comes in because of 2 reasons

1.  Per region u have 128 MB of flush size. Means when this region
reaches this size, a flush is initiated. But that happens in
background. Still cells getting written to this region. But there is a
blocking multiplier which is 2 in ur case.  Means before actual flush
completion, if the region size reaches  2 * 128 MB,  that region will
not allow to take any writes.  But here we will throw Exception back
to client so as to make it retry.
2. Global memstore size reaches the max.   U have this as 40%.  Means
40% of ur xmx for RS process.  How much is this?  And how many regions
u have?  U allow every region to be 256 MB max.  256 * No of regions
is what u ideally try to allow.  But ur global memstore size may be
lesser than this.  So once this happens, the write are blocked.  Not
throwing back any exceptions but the writer handlers at RS side are
blocked.. U can see these in logs also.. U referring to this  only?

The IO on ur cluster decides the flush speed.  Also there is config to
specify the number of flusher threads.  Same way u can control the
number of threads for compactions (Minor).   When compaction process i
lagging so behind, there might reach a situation that there are so
many HFiles under a store and in such cases, the flushers has to wait
for the compactions to get completed..  Pls see all these thread
counts and notice how big ur flush Q and compaction Q grows..

-Anoop-



On Sun, Feb 12, 2017 at 2:38 AM, Ted Yu <yu...@gmail.com> wrote:
> Which hbase release are you using ?
>
> You have given estimate on write load. How about read load ?
> Do you observe blocking / slow down on write path ?
>
> Consider increasing memstore, decreasing block cache if read load is light.
> I assume the 4 nodes with 125 GB Ram have HDD (instead of SSD). If you have
> SSD, you can use SSD for WAL - see HBASE-12848
>
> bq. its on weekend when load is low.
>
> This should have low impact on write load and is good practice.
>
> Cheers
>
> On Mon, Feb 6, 2017 at 9:15 PM, Manjeet Singh <ma...@gmail.com>
> wrote:
>
>> How to avoid read/write update blocking in hbase and on what condition
>> Hbase block read/write?
>>
>>
>>
>> what should the frequency of Major compaction ( I have daily data insert on
>> average is 18 GB and 48 GB max, its with SNAPPY and FAST_DIFF)
>>
>>  a. should it be on daily basis?
>>
>>  b. should it be in weekends?
>>
>>  c. does Major compaction on daily basis will avoid blocking?
>>
>>
>> What is the calculation? Right now its on weekend when load is low.
>>
>> we have :
>> total 7 (3+4) data nodes
>> 3 80GB Ram 1.8TB HDD
>> 4 125 GB Ram HDD 6 TB
>>
>> below are few important configurations:
>>
>> hbase.regionserver.global.memstore.lowerLimit = .38
>> hbase.regionserver.global.memstore.upperLimit = .4
>> hbase.hregion.memstore.block.multiplier=2
>> hbase.hregion.majorcompaction=0 ( scheduled on weekend)
>> hbase.hregion.memstore.flush.size 128 MB
>> hbase.hregion.preclose.flush.size 5 MB
>> hbase.hstore.compactionThreshold 3
>>
>> --
>> luv all
>>

Re: How to avoid update blocking

[Ted Yu <yu...@gmail.com>]

Which hbase release are you using ?

You have given estimate on write load. How about read load ?
Do you observe blocking / slow down on write path ?

Consider increasing memstore, decreasing block cache if read load is light.
I assume the 4 nodes with 125 GB Ram have HDD (instead of SSD). If you have
SSD, you can use SSD for WAL - see HBASE-12848

bq. its on weekend when load is low.

This should have low impact on write load and is good practice.

Cheers

On Mon, Feb 6, 2017 at 9:15 PM, Manjeet Singh <ma...@gmail.com>
wrote:

> How to avoid read/write update blocking in hbase and on what condition
> Hbase block read/write?
>
>
>
> what should the frequency of Major compaction ( I have daily data insert on
> average is 18 GB and 48 GB max, its with SNAPPY and FAST_DIFF)
>
>  a. should it be on daily basis?
>
>  b. should it be in weekends?
>
>  c. does Major compaction on daily basis will avoid blocking?
>
>
> What is the calculation? Right now its on weekend when load is low.
>
> we have :
> total 7 (3+4) data nodes
> 3 80GB Ram 1.8TB HDD
> 4 125 GB Ram HDD 6 TB
>
> below are few important configurations:
>
> hbase.regionserver.global.memstore.lowerLimit = .38
> hbase.regionserver.global.memstore.upperLimit = .4
> hbase.hregion.memstore.block.multiplier=2
> hbase.hregion.majorcompaction=0 ( scheduled on weekend)
> hbase.hregion.memstore.flush.size 128 MB
> hbase.hregion.preclose.flush.size 5 MB
> hbase.hstore.compactionThreshold 3
>
> --
> luv all
>

Re: How to avoid update blocking

[Manjeet Singh <ma...@gmail.com>]

Hi all

I am still looking if anyone can help?

Thanks
Manjeet
On 7 Feb 2017 10:45, "Manjeet Singh" <ma...@gmail.com> wrote:

>
>
> How to avoid read/write update blocking in hbase and on what condition
> Hbase block read/write?
>
>
>
> what should the frequency of Major compaction ( I have daily data insert
> on average is 18 GB and 48 GB max, its with SNAPPY and FAST_DIFF)
>
>  a. should it be on daily basis?
>
>  b. should it be in weekends?
>
>  c. does Major compaction on daily basis will avoid blocking?
>
>
> What is the calculation? Right now its on weekend when load is low.
>
> we have :
> total 7 (3+4) data nodes
> 3 80GB Ram 1.8TB HDD
> 4 125 GB Ram HDD 6 TB
>
> below are few important configurations:
>
> hbase.regionserver.global.memstore.lowerLimit = .38
> hbase.regionserver.global.memstore.upperLimit = .4
> hbase.hregion.memstore.block.multiplier=2
> hbase.hregion.majorcompaction=0 ( scheduled on weekend)
> hbase.hregion.memstore.flush.size 128 MB
> hbase.hregion.preclose.flush.size 5 MB
> hbase.hstore.compactionThreshold 3
>
> --
> luv all
>